I am currently in process of self hosting n8n using an Enterprise License.
Have setup different k8s nodes for Main, Worker and Webhook. Have cerated and tested few workflows in test and production mode.
I can see workflow history also. But when I try to see Execution history, I get an empty screen with 403. I am logged in as a Owner user. What is going on here.
It is Kubernetes based self hosted env using multimain + queue mode.
I was able to run few small workflows using webhooks and they ran fine.
here are all the configs setup in n8n main container
DB_TYPE: 'postgresdb'
EXECUTIONS_MODE: 'queue'
N8N_LOG_LEVEL: info
QUEUE_BULL_REDIS_PORT: '6379'
DB_POSTGRESDB_SSL_REJECT_UNAUTHORIZED: 'false'
DB_POSTGRESDB_DATABASE: 'n8n-dev'
N8N_MULTI_MAIN_SETUP_ENABLED: 'true'
N8N_DISABLE_PRODUCTION_MAIN_PROCESS: true
N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS: true
N8N_RUNNERS_ENABLED: true
OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS: true
N8N_DIAGNOSTICS_ENABLED: false
N8N_VERSION_NOTIFICATIONS_ENABLED: false
N8N_TEMPLATES_ENABLED: false
N8N_HOST: 'n8n.mydomain.com'
N8N_EDITOR_BASE_URL: 'https://n8n.mydomain.com'
WEBHOOK_URL: 'https://n8n-webhook.mydomain.com'
DB_POSTGRESDB_HOST: a
DB_POSTGRESDB_USER: a
DB_POSTGRESDB_PASSWORD: a
N8N_ENCRYPTION_KEY: a
QUEUE_BULL_REDIS_HOST: a
Here are the startup logs for Leader instance
`Invalid number value for N8N_PORT: tcp://10.100.7.27:443`
`Initializing n8n process`
`n8n ready on 0.0.0.0, port 5678`
`n8n Task Broker ready on 127.0.0.1, port 5679`
`[license SDK] Skipping renewal on init because renewal is not due yet or cert is not initialized`
`Task runner connection attempt failed with status code 403`
`Task runner connection attempt failed with status code 403`
`Cannot set LDAP login enabled state when an authentication method other than email or ldap is active (current: saml)`
`Version: 1.93.0`
` - "S-Test" (ID: wcHEPzTpb7ZNdsda))`
` => Started`
` - "My workflow 2" (ID: NvvFhzwc6rfweOhw))`
` => Started`
` - "My workflow" (ID: loVRNwerfbucmehc))`
` => Started`
``
`Editor is now accessible via:`
`https://n8n.mydomain.com`
`Registered runner "JS Task Runner" (9OEaxuESReT0MNZGGj2Yu)`
`[Instance ID main-Scm0i66ORu01NgcU] Leader is now this instance`
`[license SDK] License auto-renewals enabled`
` ================================`
` Start Active Workflows:`
` ================================`
`Execution 90 (job 48) finished successfully`
`Execution 91 (job 49) finished successfully`
`Pruning old insights data`
`[license SDK] attempting license renewal`
`[license SDK] license successfully renewed`
Am looking at setting up k8s, but osmething that I noticed was this on the runner docs:
N8N_RUNNERS_BROKER_LISTEN_ADDRESS=0.0.0.0 By default, the task broker only listens to localhost. When using multiple containers (for example, with Docker Compose), it needs to be able to accept external connections.
How is it going? Did you manage to fix it?
You could try disable task runner, see if it’s stable then, and maybe try disable multi-main, for testing.
It’s strange you mentioned it worked, which makes me think, maybe you was on leader, and maybe leader switch happen and that node has issue :? just guessing atm
Our setup is Queue enabled setup.
I have configured a Redis instance which will I belive act as a broker. Host is set as part of QUEUE_BULL_REDIS_HOST env var.
Will internal n8n broker be active if a redis is provisioned ??
Also, I tried swithcing from multi main to single main node and also tried disabling task runner.
Still I am not able to see the execution history.
I can confirm entries are getting added to execution_entity table in Database, but those are not reflecting in UI
Am wondering if setting Execution data | n8n Docs will help, my assumption atm are, execution history maybe too large and timing out? Hense no display. So it could be execution data is too large, so if we enable the purge, it may fix it.
You could also try spin up a latest version of n8n, and copy db over with enc key, and see if you see the same issue. (keep the original running, make another env for testing)
But it doesn’t explain why you get the 403 forbidden :?
I have a feeling this is probably a bug that may have been fixed already, but i can’t see any PR or issue logged for this, I would try on newer version, there is clearly a permission issue here, and likely due to multi-main.
@King_Samuel_David
It’s a newly deployed instance with only around 100-120 test executions in data base. We still haven’t rolled it out for full scale usage.
I have also updated to latest ver : 1.95.3
But still same behaviour…getting 403 when I try to see any of the execution history: Personal or Project wise
Hey @Sahil_Wadhwa,
could you try deploying without N8N_EDITOR_BASE_URL. You are already setting the N8N_HOST and n8n isn’t handling TSL certs (https) from what I can see.
Best to test with a single container first. Once that works, the working config ideally is shared between main, worker and webhook, with a few exceptions like N8N_DISABLE_PRODUCTION_MAIN_PROCESS.
@marcus thanks for this info, how does multimain communcate? Leader / follower like Kafka?
@Sahil_Wadhwa I guess instead of deploying 1.93, maybe the issue is from there, but try deploy latest again a fresh deployment, and test executions quickly?
I deployed using k8s, but didn’t face the same issue.
Leader selection is implemented with Redis, using a special leader key that is locked to the current leader and has a short lifetime. Both the leader and others try to grab the key lock in short intervals. As long as the leader is alive, it will refresh the lock. If the leader dies, another instance will grab the lock for itself and become the leader. We have some configuration around that mechanism here.
Invalid number value for N8N_PORT: tcp://10.100.7.27:443
Initializing n8n process
n8n ready on 0.0.0.0, port 5678
There is a deprecation related to your environment variables. Please take the recommended actions to update your configuration:
N8N_RUNNERS_ENABLED → Running n8n without task runners is deprecated. Task runners will be turned on by default in a future version. Please set N8N_RUNNERS_ENABLED=true to enable task runners now and avoid potential issues in the future. Learn more: Task runners | n8n Docs
[license SDK] Skipping renewal on init because renewal is not due yet or cert is not initialized
Cannot set LDAP login enabled state when an authentication method other than email or ldap is active (current: saml)
Set local git repository to track remote
Version: 1.95.3
Start Active Workflows:
“SW-Test” (ID: wcHEPzTpb7ZNdqJa))
=> Started
“My workflow” (ID: loVRNbdxdbucmehc))
=> Started
“DataDogWebhook” (ID: iP3PE1qSO7d5ANhR))
=> Started
I have a feel it’s going to be hard to detect, and is likely due to multi-main in my opinion, where the role switch casued it and then caused an invaid state in the DB / config.
It may also not happen again unless them exact conditions happen again, it could have been a number of potentials things happening in N8N when it make this invalid state, it maybe in logs, if you have still. Can help review logs if needed.