N8n doesn't start after updating to 0.234.0

Karbust · July 4, 2023, 9:34am

Describe the problem/error/question

This issue started happening after updating to 0.234.0 (using latest tag from docker image) from 0.232.0. The migrations ran successfully, then started crashing and doesn’t start.

What is the error message (if any)?

2023-07-04T09:24:50.315118447Z n8n ready on 0.0.0.0, port 5678
2023-07-04T09:24:50.333007551Z 2023-07-04T09:24:50.332Z | warn     | Migrations in progress, please do NOT stop the process. "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
2023-07-04T09:24:50.333257537Z 2023-07-04T09:24:50.333Z | debug    | Starting migration SeparateExecutionData1690000000030 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
2023-07-04T09:24:55.825905778Z 2023-07-04T09:24:55.822Z | warn     | License manager not initialized "{ file: 'License.js', function: 'isFeatureEnabled' }"
2023-07-04T09:24:55.827167343Z 2023-07-04T09:24:55.826Z | error    | Last session crashed "{ file: 'CrashJournal.js', function: 'init' }"
2023-07-04T09:25:06.018185406Z 2023-07-04T09:25:06.017Z | debug    | Lazy Loading credentials and nodes from n8n-nodes-base "{\n  credentials: 317,\n  nodes: 422,\n  file: 'DirectoryLoader.js',\n  function: 'loadAll'\n}"

Share the output returned by the last node

Information on your n8n setup

n8n version: 0.234.0
Database: MySQL
n8n EXECUTIONS_PROCESS setting (default: own, main): default, never changed
Running n8n via: docker
Operating system: Ubuntu 22.04.1 LTS

MutedJam · July 4, 2023, 11:09am

Hi @Karbust, I am so sorry you are having trouble. The log entry Last session crashed only suggests n8n has previously been terminated while executing a workflow, but it otherwise seems to be running.

Do you perhaps have a large amount of execution data in the execution_entity table, meaning the migration simply takes a lot of time here?

Karbust · July 4, 2023, 11:22am

The migration isn’t even running, it’s on an infinite loop of crashing and starting again (container set to run unless-stopped).

But yes, my execution_entity table has around 1.3 million entries, the previous migrations took a few minutes to finish, this last one is stuck…

After attempting to run the last migration, I reached this:

github.com

n8n-io/n8n/blob/master/packages/cli/src/databases/migrations/mysqldb/1690000000030-SeparateExecutionData.ts#L15-L22


      
          		await queryRunner.query(
          			`INSERT INTO ${tablePrefix}execution_data (
          				executionId,
          				workflowData,
          				data)
          				SELECT id, workflowData, data FROM ${tablePrefix}execution_entity
          			`,
          		);

The issue is this:

github.com

n8n-io/n8n/blob/master/packages/cli/src/databases/migrations/mysqldb/1690000000030-SeparateExecutionData.ts#L9


      
          import type { MigrationContext, ReversibleMigration } from '@db/types';
          
          
export class SeparateExecutionData1690000000030 implements ReversibleMigration {
          	async up({ queryRunner, tablePrefix }: MigrationContext) {
          		await queryRunner.query(
          			`CREATE TABLE ${tablePrefix}execution_data (
          				executionId int(11) NOT NULL primary key,
          				workflowData json NOT NULL,
          				data TEXT NOT NULL,
          				CONSTRAINT \`${tablePrefix}execution_data_FK\` FOREIGN KEY (\`executionId\`) REFERENCES \`${tablePrefix}execution_entity\` (\`id\`) ON DELETE CASCADE
          			)
          			ENGINE=InnoDB`,
          		);
          
          
		await queryRunner.query(
          			`INSERT INTO ${tablePrefix}execution_data (
          				executionId,
          				workflowData,
          				data)

After changing the column directly on the database to match the column on execution_entity (mediumtext) the query ran successfully. However the migration keeps failing…

MySQL Data Types:
TEXT: A TEXT column with a maximum length of 65,535 characters
MEDIUMTEXT: A TEXT column with a maximum length of 16,777,215 characters.

MutedJam · July 4, 2023, 12:12pm

Thank you so much for sharing these details and please accept my sincere apologies for the trouble. This change was most not intended (and we have in fact fixed this very problem in the past, but it seems the most recent release brought it back).

The team is currently looking into how to best approach this, but in the meantime you would want to restore a backup from before the migration.

Karbust · July 4, 2023, 2:31pm

Thank you for the response. Unfortunately, due to my lack of responsibility, I didn’t do a backup

Will have to wait until it is fixed. I’m just using it for personal use and small monitoring, nothing major, can wait

MutedJam · July 4, 2023, 2:36pm

You might still be able to utilize the CLI to export your workflows and credentials, then import them into a fresh instance. Perhaps you want to give this a go? The CLI would be documented here:

Jon · July 4, 2023, 2:37pm

This will also be fixed in 0.235.1

Karbust · July 4, 2023, 2:57pm

The issue with using the commands is that my installation is on a docker container, it doesn’t even start, thus the commands don’t run properly.

Either it outputs the debug message or doesn’t output anything:

karbust@karbust-server:~# docker exec -u node -it n8n n8n export:workflow --all
karbust@karbust-server:~# docker exec -u node -it n8n n8n export:workflow --all
2023-07-04T14:53:28.486Z | debug    | Lazy Loading credentials and nodes from n8n-nodes-base "{\n  credentials: 317,\n  nodes: 422,\n  file: 'DirectoryLoader.js',\n  function: 'loadAll'\n}"
karbust@karbust-server:~# docker exec -u node -it n8n n8n export:workflow --all --output=backups/latest/
2023-07-04T14:53:44.342Z | debug    | Lazy Loading credentials and nodes from n8n-nodes-base "{\n  credentials: 317,\n  nodes: 422,\n  file: 'DirectoryLoader.js',\n  function: 'loadAll'\n}"
karbust@karbust-server:~#

Another way would probably be cloning the repo with the fixes, run the migrations and the docker container would probably start.

I don’t mind waiting.

MutedJam · July 4, 2023, 3:03pm

In this case, let’s really wait until a migration fixing this becomes available.

If you are familiar with SQL you could also clean up the executions_entity table outside of n8n already, and thus remove old executions you no longer need. This would speed up future migrations for sure.

Karbust · July 4, 2023, 3:27pm

Yeah, don’t really need them

Kept just the past 15 days of logs, just in case:

DELETE FROM execution_entity
WHERE stoppedAt < '2023-06-15 00:00:00';

Had logs all the way back to end 2021

MutedJam · July 5, 2023, 1:24pm

Hi @Karbust, just wanted to let you know that [email protected] is now available via Docker, containing a fix for the MySQL/MariaDB migration: https://github.com/n8n-io/n8n/pull/6591

Perhaps you want to give it a shot?

jan · July 5, 2023, 1:29pm

New version [email protected] got released which includes the GitHub PR 6591.

Karbust · July 5, 2023, 1:47pm

Hey,

Thank you for the speedy release

It is working but, had to manually delete the table execution_data create previously and let the migrations run it again, not sure if that’s the intended behavior.

Thank you

MutedJam · July 5, 2023, 1:48pm

Huh, now that is strange. The new release should have fixed the previously broken migration. Though on the other hand, if it’s now working maybe we shouldn’t ask too many questions

Once again I am so sorry for the trouble this has caused.

jhambach · July 26, 2023, 1:58pm

I just ran into this last night trying to upgrade from 0.225.2 > 0.236.3

We run n8n via npm and after the upgrade, n8n would attempt to start then crash and continuously cycle between back and forth, though pm2 always showed it was running. We rolled back to 0.225.2 for now

Jon · July 26, 2023, 4:05pm

Hey @jhambach,

What did the log say and which database are you using?

jhambach · July 26, 2023, 4:08pm

I did not have logging enabled, I’ll get that setup and retry tonight. We are running MySQL, preparing to migrate to postgres but wanted to get to the latest 0.x version first per the v1.0 migration doc

Jon · July 26, 2023, 4:18pm

It could be that the migration failed as it can take some time and if it was stopped things go wrong. I would recommend restoring from a database backup if you have one then trying again or exporting the workflows and credentials in the current state and doing the migration now.

jhambach · July 26, 2023, 4:35pm

I took a snapshot as this runs on a vm so restoring wasn’t an issue. Will I need to re-import the workflows and credentials once I’ve changed my environment variables to postgres? I’m not worried about the execution history

Jon · July 26, 2023, 4:48pm

Hey @jhambach,

Yeah you would need to import them once you swap so before you do change the options make sure you run the CLI commands to export the workflows and the credentials.