N8n doesn't start after updating to 0.234.0

Describe the problem/error/question

This issue started happening after updating to 0.234.0 (using latest tag from docker image) from 0.232.0. The migrations ran successfully, then started crashing and doesn’t start.

What is the error message (if any)?

2023-07-04T09:24:50.315118447Z n8n ready on 0.0.0.0, port 5678
2023-07-04T09:24:50.333007551Z 2023-07-04T09:24:50.332Z | warn     | Migrations in progress, please do NOT stop the process. "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
2023-07-04T09:24:50.333257537Z 2023-07-04T09:24:50.333Z | debug    | Starting migration SeparateExecutionData1690000000030 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
2023-07-04T09:24:55.825905778Z 2023-07-04T09:24:55.822Z | warn     | License manager not initialized "{ file: 'License.js', function: 'isFeatureEnabled' }"
2023-07-04T09:24:55.827167343Z 2023-07-04T09:24:55.826Z | error    | Last session crashed "{ file: 'CrashJournal.js', function: 'init' }"
2023-07-04T09:25:06.018185406Z 2023-07-04T09:25:06.017Z | debug    | Lazy Loading credentials and nodes from n8n-nodes-base "{\n  credentials: 317,\n  nodes: 422,\n  file: 'DirectoryLoader.js',\n  function: 'loadAll'\n}"

Share the output returned by the last node

Information on your n8n setup

  • n8n version: 0.234.0
  • Database: MySQL
  • n8n EXECUTIONS_PROCESS setting (default: own, main): default, never changed
  • Running n8n via: docker
  • Operating system: Ubuntu 22.04.1 LTS

Hi @Karbust, I am so sorry you are having trouble. The log entry Last session crashed only suggests n8n has previously been terminated while executing a workflow, but it otherwise seems to be running.

Do you perhaps have a large amount of execution data in the execution_entity table, meaning the migration simply takes a lot of time here?

The migration isn’t even running, it’s on an infinite loop of crashing and starting again (container set to run unless-stopped).

But yes, my execution_entity table has around 1.3 million entries, the previous migrations took a few minutes to finish, this last one is stuck…

After attempting to run the last migration, I reached this:

The issue is this:

After changing the column directly on the database to match the column on execution_entity (mediumtext) the query ran successfully. However the migration keeps failing…

MySQL Data Types:
TEXT: A TEXT column with a maximum length of 65,535 characters
MEDIUMTEXT: A TEXT column with a maximum length of 16,777,215 characters.

1 Like

Thank you so much for sharing these details and please accept my sincere apologies for the trouble. This change was most not intended (and we have in fact fixed this very problem in the past, but it seems the most recent release brought it back).

The team is currently looking into how to best approach this, but in the meantime you would want to restore a backup from before the migration.

Thank you for the response. Unfortunately, due to my lack of responsibility, I didn’t do a backup :sweat_smile:

Will have to wait until it is fixed. I’m just using it for personal use and small monitoring, nothing major, can wait :laughing:

You might still be able to utilize the CLI to export your workflows and credentials, then import them into a fresh instance. Perhaps you want to give this a go? The CLI would be documented here:

This will also be fixed in 0.235.1

1 Like

The issue with using the commands is that my installation is on a docker container, it doesn’t even start, thus the commands don’t run properly.

Either it outputs the debug message or doesn’t output anything:

karbust@karbust-server:~# docker exec -u node -it n8n n8n export:workflow --all
karbust@karbust-server:~# docker exec -u node -it n8n n8n export:workflow --all
2023-07-04T14:53:28.486Z | debug    | Lazy Loading credentials and nodes from n8n-nodes-base "{\n  credentials: 317,\n  nodes: 422,\n  file: 'DirectoryLoader.js',\n  function: 'loadAll'\n}"
karbust@karbust-server:~# docker exec -u node -it n8n n8n export:workflow --all --output=backups/latest/
2023-07-04T14:53:44.342Z | debug    | Lazy Loading credentials and nodes from n8n-nodes-base "{\n  credentials: 317,\n  nodes: 422,\n  file: 'DirectoryLoader.js',\n  function: 'loadAll'\n}"
karbust@karbust-server:~#

Another way would probably be cloning the repo with the fixes, run the migrations and the docker container would probably start.

I don’t mind waiting.

1 Like

In this case, let’s really wait until a migration fixing this becomes available.

If you are familiar with SQL you could also clean up the executions_entity table outside of n8n already, and thus remove old executions you no longer need. This would speed up future migrations for sure.

Yeah, don’t really need them :sweat_smile:

Kept just the past 15 days of logs, just in case:

DELETE FROM execution_entity
WHERE stoppedAt < '2023-06-15 00:00:00';

Had logs all the way back to end 2021 :joy:

1 Like

Hi @Karbust, just wanted to let you know that [email protected] is now available via Docker, containing a fix for the MySQL/MariaDB migration: https://github.com/n8n-io/n8n/pull/6591

Perhaps you want to give it a shot?

1 Like

New version [email protected] got released which includes the GitHub PR 6591.

1 Like

Hey,

Thank you for the speedy release :smiley:

It is working but, had to manually delete the table execution_data create previously and let the migrations run it again, not sure if that’s the intended behavior.

Thank you

2 Likes

Huh, now that is strange. The new release should have fixed the previously broken migration. Though on the other hand, if it’s now working maybe we shouldn’t ask too many questions :see_no_evil:

Once again I am so sorry for the trouble this has caused.

2 Likes

I just ran into this last night trying to upgrade from 0.225.2 > 0.236.3

We run n8n via npm and after the upgrade, n8n would attempt to start then crash and continuously cycle between back and forth, though pm2 always showed it was running. We rolled back to 0.225.2 for now

Hey @jhambach,

What did the log say and which database are you using?

I did not have logging enabled, I’ll get that setup and retry tonight. We are running MySQL, preparing to migrate to postgres but wanted to get to the latest 0.x version first per the v1.0 migration doc

It could be that the migration failed as it can take some time and if it was stopped things go wrong. I would recommend restoring from a database backup if you have one then trying again or exporting the workflows and credentials in the current state and doing the migration now.

I took a snapshot as this runs on a vm so restoring wasn’t an issue. Will I need to re-import the workflows and credentials once I’ve changed my environment variables to postgres? I’m not worried about the execution history

Hey @jhambach,

Yeah you would need to import them once you swap so before you do change the options make sure you run the CLI commands to export the workflows and the credentials.

1 Like