Bad Gateway

Integra_Ninja · February 1, 2022, 6:08pm

Hello, I’m having difficulties starting n8n after the page’s certificate expired and was not automatically renewed.
I tried simply updating the n8n version to see if the issue is resolved, but now it’s just giving “bad gateway”.

I have enough disk space on linux.

Stopping n8n...
Initializing n8n process


INFO: Started with migration for wait functionality.
      Depending on the number of saved executions, that may take a little bit.


query is slow: INSERT INTO "temporary_execution_entity"("id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId") SELECT "id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId" FROM "execution_entity"
execution time: 54272
query is slow: DROP TABLE "execution_entity"
execution time: 35084
query is slow: CREATE INDEX "IDX_cefb067df2402f6aed0638a6c1" ON "execution_entity" ("stoppedAt")
execution time: 129961
query is slow: CREATE INDEX "IDX_ca4a71b47f28ac6ea88293a8e2" ON "execution_entity" ("waitTill")
execution time: 8020
query is slow: VACUUM;
execution time: 256646
There was an error initializing DB: "SQLITE_FULL: database or disk is full"

Stopping n8n...
Initializing n8n process
query is slow: SELECT id FROM migrations where name = "MakeStoppedAtNullable1607431743769"
execution time: 13049
query is slow: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'migrations'
execution time: 1003
There was an error initializing DB: "SQLITE_BUSY: database is locked"

Stopping n8n...
Initializing n8n process
query is slow: SELECT id FROM migrations where name = "MakeStoppedAtNullable1607431743769"
execution time: 13047
query is slow: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'migrations'
execution time: 1009
There was an error initializing DB: "SQLITE_BUSY: database is locked"

Stopping n8n...
Initializing n8n process
query is slow: SELECT id FROM migrations where name = "MakeStoppedAtNullable1607431743769"
execution time: 10948
query is slow: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'migrations'
execution time: 1004
There was an error initializing DB: "SQLITE_BUSY: database is locked"

Stopping n8n...
Initializing n8n process
query is slow: SELECT id FROM migrations where name = "MakeStoppedAtNullable1607431743769"
execution time: 10418
query is slow: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'migrations'
execution time: 1005
There was an error initializing DB: "SQLITE_BUSY: database is locked"

Stopping n8n...
Initializing n8n process
query is slow: SELECT id FROM migrations where name = "MakeStoppedAtNullable1607431743769"
execution time: 9377
query is slow: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'migrations'
execution time: 10103


INFO: Started with migration for wait functionality.
      Depending on the number of saved executions, that may take a little bit.


query is slow: INSERT INTO "temporary_execution_entity"("id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId") SELECT "id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId" FROM "execution_entity"
execution time: 190494

Stopping n8n...
There was an error shutting down n8n. Error: InternalHooks not initialized
    at Function.getInstance (/usr/local/lib/node_modules/n8n/dist/src/InternalHooksManager.js:11:15)
    at process.stopProcess (/usr/local/lib/node_modules/n8n/dist/commands/start.js:31:46)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
Initializing n8n process


INFO: Started with migration for wait functionality.
      Depending on the number of saved executions, that may take a little bit.


root@vps-8667f52b:~# docker logs -ft  n8n_n8n_1
2022-02-01T17:37:22.951544806Z Initializing n8n process
2022-02-01T17:37:23.352861170Z
2022-02-01T17:37:23.352926541Z
2022-02-01T17:37:23.352936637Z INFO: Started with migration for wait functionality.
2022-02-01T17:37:23.352942862Z       Depending on the number of saved executions, that may take a little bit.
2022-02-01T17:37:23.352947879Z
2022-02-01T17:37:23.352951259Z
2022-02-01T17:39:48.466395981Z query is slow: INSERT INTO "temporary_execution_entity"("id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId") SELECT "id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId" FROM "execution_entity"
2022-02-01T17:39:48.479992497Z execution time: 145021
2022-02-01T17:41:45.739306233Z query is slow: DROP TABLE "execution_entity"
2022-02-01T17:41:45.743485576Z execution time: 117256
2022-02-01T17:42:03.145129770Z query is slow: CREATE INDEX "IDX_cefb067df2402f6aed0638a6c1" ON "execution_entity" ("stoppedAt")
2022-02-01T17:42:03.147720051Z execution time: 17381
2022-02-01T17:42:12.447046119Z query is slow: CREATE INDEX "IDX_ca4a71b47f28ac6ea88293a8e2" ON "execution_entity" ("waitTill")
2022-02-01T17:42:12.449149769Z execution time: 9293
2022-02-01T17:43:42.702579766Z query is slow: VACUUM;
2022-02-01T17:43:42.716827060Z execution time: 90238
2022-02-01T17:43:42.721046170Z There was an error initializing DB: "SQLITE_FULL: database or disk is full"
2022-02-01T17:43:42.725387352Z
2022-02-01T17:43:42.725477928Z Stopping n8n...
2022-02-01T17:43:48.591392843Z Initializing n8n process
2022-02-01T17:43:49.051533963Z
2022-02-01T17:43:49.051598454Z
2022-02-01T17:43:49.051609077Z INFO: Started with migration for wait functionality.
2022-02-01T17:43:49.051615771Z       Depending on the number of saved executions, that may take a little bit.
2022-02-01T17:43:49.051622132Z
2022-02-01T17:43:49.051628580Z
2022-02-01T17:44:19.488552450Z query is slow: INSERT INTO "temporary_execution_entity"("id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId") SELECT "id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId" FROM "execution_entity"
2022-02-01T17:44:19.492805918Z execution time: 30377
2022-02-01T17:44:34.723427749Z query is slow: DROP TABLE "execution_entity"
2022-02-01T17:44:34.724631404Z execution time: 15219
2022-02-01T17:45:13.254292454Z query is slow: CREATE INDEX "IDX_cefb067df2402f6aed0638a6c1" ON "execution_entity" ("stoppedAt")
2022-02-01T17:45:13.255227841Z execution time: 38507
2022-02-01T17:45:23.239616069Z query is slow: CREATE INDEX "IDX_ca4a71b47f28ac6ea88293a8e2" ON "execution_entity" ("waitTill")
2022-02-01T17:45:23.240625916Z execution time: 9979
2022-02-01T17:47:24.236088791Z query is slow: VACUUM;
2022-02-01T17:47:24.237931464Z execution time: 120985
2022-02-01T17:47:24.246084702Z There was an error initializing DB: "SQLITE_FULL: database or disk is full"
2022-02-01T17:47:24.248659646Z
2022-02-01T17:47:24.248726687Z Stopping n8n...
2022-02-01T17:47:30.741600913Z Initializing n8n process
2022-02-01T17:47:31.078111916Z
2022-02-01T17:47:31.078232186Z
2022-02-01T17:47:31.078253113Z INFO: Started with migration for wait functionality.
2022-02-01T17:47:31.078258452Z       Depending on the number of saved executions, that may take a little bit.
2022-02-01T17:47:31.078262817Z
2022-02-01T17:47:31.078266576Z
2022-02-01T17:48:25.418791715Z query is slow: INSERT INTO "temporary_execution_entity"("id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId") SELECT "id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId" FROM "execution_entity"
2022-02-01T17:48:25.424072331Z execution time: 54272
2022-02-01T17:49:00.538824328Z query is slow: DROP TABLE "execution_entity"
2022-02-01T17:49:00.540129350Z execution time: 35084
2022-02-01T17:51:10.529489806Z query is slow: CREATE INDEX "IDX_cefb067df2402f6aed0638a6c1" ON "execution_entity" ("stoppedAt")
2022-02-01T17:51:10.531298685Z execution time: 129961
2022-02-01T17:51:18.556735757Z query is slow: CREATE INDEX "IDX_ca4a71b47f28ac6ea88293a8e2" ON "execution_entity" ("waitTill")
2022-02-01T17:51:18.557367701Z execution time: 8020
2022-02-01T17:55:35.215554050Z query is slow: VACUUM;
2022-02-01T17:55:35.219569428Z execution time: 256646
2022-02-01T17:55:35.228650420Z There was an error initializing DB: "SQLITE_FULL: database or disk is full"
2022-02-01T17:55:35.231145052Z
2022-02-01T17:55:35.231176659Z Stopping n8n...
2022-02-01T17:55:41.231781957Z Initializing n8n process
2022-02-01T17:55:54.836292335Z query is slow: SELECT id FROM migrations where name = "MakeStoppedAtNullable1607431743769"
2022-02-01T17:55:54.837082423Z execution time: 13049
2022-02-01T17:55:55.850173404Z query is slow: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'migrations'
2022-02-01T17:55:55.850697731Z execution time: 1003
2022-02-01T17:55:55.852384577Z There was an error initializing DB: "SQLITE_BUSY: database is locked"
2022-02-01T17:55:55.853761211Z
2022-02-01T17:55:55.853775042Z Stopping n8n...
2022-02-01T17:56:05.172190034Z Initializing n8n process
2022-02-01T17:56:19.070053734Z query is slow: SELECT id FROM migrations where name = "MakeStoppedAtNullable1607431743769"
2022-02-01T17:56:19.071726790Z execution time: 13047
2022-02-01T17:56:20.101287568Z query is slow: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'migrations'
2022-02-01T17:56:20.102303948Z execution time: 1009
2022-02-01T17:56:20.106014911Z There was an error initializing DB: "SQLITE_BUSY: database is locked"
2022-02-01T17:56:20.108428978Z
2022-02-01T17:56:20.108453796Z Stopping n8n...
2022-02-01T17:56:29.780478349Z Initializing n8n process
2022-02-01T17:56:41.331073997Z query is slow: SELECT id FROM migrations where name = "MakeStoppedAtNullable1607431743769"
2022-02-01T17:56:41.332019357Z execution time: 10948
2022-02-01T17:56:42.344115603Z query is slow: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'migrations'
2022-02-01T17:56:42.344765514Z execution time: 1004
2022-02-01T17:56:42.346889046Z There was an error initializing DB: "SQLITE_BUSY: database is locked"
2022-02-01T17:56:42.348088942Z
2022-02-01T17:56:42.348103257Z Stopping n8n...
2022-02-01T17:56:50.085733014Z Initializing n8n process
2022-02-01T17:57:01.001433487Z query is slow: SELECT id FROM migrations where name = "MakeStoppedAtNullable1607431743769"
2022-02-01T17:57:01.003044191Z execution time: 10418
2022-02-01T17:57:02.026155091Z query is slow: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'migrations'
2022-02-01T17:57:02.026927086Z execution time: 1005
2022-02-01T17:57:02.029741636Z There was an error initializing DB: "SQLITE_BUSY: database is locked"
2022-02-01T17:57:02.031838573Z
2022-02-01T17:57:02.031870478Z Stopping n8n...
2022-02-01T17:57:11.683156732Z Initializing n8n process
2022-02-01T17:57:21.617706225Z query is slow: SELECT id FROM migrations where name = "MakeStoppedAtNullable1607431743769"
2022-02-01T17:57:21.618810327Z execution time: 9377
2022-02-01T17:57:31.733810934Z query is slow: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'migrations'
2022-02-01T17:57:31.735717514Z execution time: 10103
2022-02-01T17:57:31.759615456Z
2022-02-01T17:57:31.759663732Z
2022-02-01T17:57:31.759674563Z INFO: Started with migration for wait functionality.
2022-02-01T17:57:31.759683357Z       Depending on the number of saved executions, that may take a little bit.
2022-02-01T17:57:31.759690467Z
2022-02-01T17:57:31.759696538Z
2022-02-01T18:00:42.292005763Z query is slow: INSERT INTO "temporary_execution_entity"("id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId") SELECT "id", "data", "finished", "mode", "retryOf", "retrySuccessId", "startedAt", "stoppedAt", "workflowData", "workflowId" FROM "execution_entity"
2022-02-01T18:00:42.296573841Z execution time: 190494
2022-02-01T18:02:42.029110262Z
2022-02-01T18:02:42.029399143Z Stopping n8n...
2022-02-01T18:02:42.057481437Z There was an error shutting down n8n. Error: InternalHooks not initialized
2022-02-01T18:02:42.057521888Z     at Function.getInstance (/usr/local/lib/node_modules/n8n/dist/src/InternalHooksManager.js:11:15)
2022-02-01T18:02:42.057526988Z     at process.stopProcess (/usr/local/lib/node_modules/n8n/dist/commands/start.js:31:46)
2022-02-01T18:02:42.057543357Z     at processTicksAndRejections (internal/process/task_queues.js:93:5)
2022-02-01T18:03:27.019316222Z Initializing n8n process
2022-02-01T18:03:27.469873715Z
2022-02-01T18:03:27.469919805Z
2022-02-01T18:03:27.469929662Z INFO: Started with migration for wait functionality.
2022-02-01T18:03:27.469936087Z       Depending on the number of saved executions, that may take a little bit.
2022-02-01T18:03:27.469942479Z
2022-02-01T18:03:27.469948065Z
2022-02-01T18:05:33.080350165Z query is slow: DROP TABLE IF EXISTS "temporary_execution_entity"
2022-02-01T18:05:33.085300218Z execution time: 125546

Integra_Ninja · February 1, 2022, 6:34pm

update… i tried to back to version 0.134.0, but now, without a certificate again, how can i activate?

Integra_Ninja · February 1, 2022, 6:37pm

now i get this error on logs…

2022-02-01T18:35:50.134314228Z Deprecation warning: value provided is not in a recognized RFC2822 or ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions. Non RFC2822/ISO date formats are discouraged. Please refer to http://momentjs.com/guides/#/warnings/js-date/ for more info.
2022-02-01T18:35:50.134473918Z Arguments:
2022-02-01T18:35:50.134482695Z [0] _isAMomentObject: true, _isUTC: false, _useUTC: false, _l: undefined, _i: 2/1/2022, _f: undefined, _strict: undefined, _locale: [object Object]
2022-02-01T18:35:50.134487070Z Error
2022-02-01T18:35:50.134490863Z     at Function.createFromInputFallback (/usr/local/lib/node_modules/n8n/node_modules/moment/moment.js:319:25)
2022-02-01T18:35:50.134494790Z     at configFromString (/usr/local/lib/node_modules/n8n/node_modules/moment/moment.js:2536:19)
2022-02-01T18:35:50.134498548Z     at configFromInput (/usr/local/lib/node_modules/n8n/node_modules/moment/moment.js:2977:13)
2022-02-01T18:35:50.134502318Z     at prepareConfig (/usr/local/lib/node_modules/n8n/node_modules/moment/moment.js:2960:13)
2022-02-01T18:35:50.134506010Z     at createFromConfig (/usr/local/lib/node_modules/n8n/node_modules/moment/moment.js:2927:44)
2022-02-01T18:35:50.134509755Z     at createLocalOrUTC (/usr/local/lib/node_modules/n8n/node_modules/moment/moment.js:3021:16)
2022-02-01T18:35:50.134513400Z     at createLocal (/usr/local/lib/node_modules/n8n/node_modules/moment/moment.js:3025:16)
2022-02-01T18:35:50.134517148Z     at hooks (/usr/local/lib/node_modules/n8n/node_modules/moment/moment.js:16:29)
2022-02-01T18:35:50.134521070Z     at Object.execute (/usr/local/lib/node_modules/n8n/node_modules/n8n-nodes-base/dist/nodes/DateTime.node.js:389:62)
2022-02-01T18:35:50.134524863Z     at Workflow.runNode (/usr/local/lib/node_modules/n8n/node_modules/n8n-workflow/dist/src/Workflow.js:492:37)

Integra_Ninja · February 1, 2022, 6:49pm

now i managed to get the ssl back, changing the .env email to another one i have.

But I still can’t update the n8n version. I think it might be because of the wait node, which it talks about in the logs.

Shirobachi · February 1, 2022, 10:01pm

it’s probably nothing but have you tried simply regenerate certificats?

Please provide:

how you run n8n (docker?)
Your n8n env (include just those connected to domain, webhook etc.)
How you generate certificates (certbot?)
Do you use CF (cloudflare)?

Also have a look for my config and try make it work properly:

File .env:
SERVER_USERNAME=simon
DOMAIN_NAME=hryszko.dev
SUBDOMAIN=n8n
N8N_BASIC_AUTH_USER=simon
N8N_BASIC_AUTH_PASSWORD='$2a$10$rm65qlh4PfLStpjmSnWZqumd1e2d1oQ8WyrbHWPnDngyieZS2cxhi'
GENERIC_TIMEZONE=Europe/Berlin
[email protected]



File docker-compose.yml:
version: "3"

services:
  n8n:
      image: n8nio/n8n:0.156.0
      restart: always
      ports:
        - "127.0.0.1:5678:443"
      environment:
        - N8N_BASIC_AUTH_ACTIVE=true
        - N8N_BASIC_AUTH_HASH=true
        - N8N_BASIC_AUTH_USER
        - N8N_BASIC_AUTH_PASSWORD
        - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
        - N8N_PORT=443
        - N8N_PROTOCOL=https
        - NODE_ENV=production
        - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
        - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
        - NODE_FUNCTION_ALLOW_EXTERNAL=moment,lodash
        - EXECUTIONS_DATA_PRUNE=true
        - EXECUTIONS_DATA_PRUNE_TIMEOUT=604800
        - DB_SQLITE_VACUUM_ON_STARTUP=true
        - EXECUTIONS_DATA_MAX_AGE=7
        - EXECUTIONS_DATA_SAVE_ON_ERROR=all
        - EXECUTIONS_DATA_SAVE_MANUAL_EXECUTIONS=false
      volumes:
        - /home/${SERVER_USERNAME}/.n8n:/home/node/.n8n
        - /home/${SERVER_USERNAME}/n8n-local-files:/files

corinja · February 3, 2022, 6:22pm

Hello. This thread has been helpful for me as I am also experiencing a Bad Gateway error after attempting to update the n8n version, following a successful renewal of the SSL Cert.

I have not been able to revert to an earlier version to get things back however. When I run

docker logs -ft ubuntu_n8n_1 I get the following output:

2022-02-03T18:01:19.361574601Z Initializing n8n process
2022-02-03T18:01:19.612455498Z 
2022-02-03T18:01:19.612487074Z 
2022-02-03T18:01:19.612492085Z INFO: Started with migration for wait functionality.
2022-02-03T18:01:19.612497136Z       Depending on the number of saved executions, that may take a little bit.
2022-02-03T18:01:19.612501844Z 
2022-02-03T18:01:19.612506170Z

It just seems to be hanging. Any suggestions for how to get past this would be gratefully received.

Also, can anyone tell me where to find the logs for my n8n container?

jan · February 3, 2022, 6:26pm

If that is the thing that gets displayed, then all you have to do normally is to wait. Because as the message says, can take quite some time to finish. Or does the container crash after that message?

corinja · February 3, 2022, 6:34pm

Thanks @jan I don’t see any sign the container has crashed. Is there some way I can confirm this?

If I need to wait that’s fine, and I just was expecting a response in seconds or a very few minutes.

jan · February 3, 2022, 7:00pm

Normally it should not take very long. Really depends totally on the number of executions you have saved. So could be anything between 1 second and probably even hours.

How many executions do you have saved (just a rough number is enough)?

If you type docker ps and that container does get displayed there, then it is still running.

corinja · February 3, 2022, 7:07pm

I’m not sure without being able to refer to the UI as it’s other people who have built our workflows. But it’s dozens at least. I’ll just hang on then and keep my fingers crossed

Useful, thanks. I can see the containers are still up:

CONTAINER ID   IMAGE       COMMAND                  CREATED             STATUS             PORTS                                                                      NAMES
f11072b467f4   n8nio/n8n   "tini -- /docker-ent…"   About an hour ago   Up About an hour   127.0.0.1:5678->5678/tcp                                                   ubuntu_n8n_1
6f52964ad150   traefik     "/entrypoint.sh --ap…"   About an hour ago   Up About an hour   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   ubuntu_traefik_1

Thanks a lot for your replies, I’m grateful.

jan · February 3, 2022, 7:58pm

I assume you are referring to workflows and not to executions? Because if you really mean executions then it should be in the second range.

benb · June 27, 2022, 3:28pm

I’m having Bad Gateway errors but with the DeepL node, I have tested my server reach to api.deepl.com and I have no packet loss, no problem reaching their API from a network or TLS certificate point of view
https://auto.osintukraine.com, plus this seems to be erratic, sometimes the nodes execute just fine, but the error is always 502 bad gateway.

Jon · June 27, 2022, 3:38pm

Hey @benb,

Can you opne a new thread and include as much information as possible including how you have n8n setup.

It sounds like if it works fine sometimes then it could be that the server is overloaded at the time but without a lot more it is hard to say.