Use external module (npm) with queue mode in railway

yukyo · October 14, 2023, 1:26am

Describe the problem/error/question

Hello, I am testing a n8n instance running on Raiway in queue mode + 10 workers. It has been working very nice for the few days that I have been doing testing.

I have some functions running in normal VPS n8n that use some NPM modules, such as:

const cheerio = require("cheerio");
const AWS = require('aws-sdk');

And then I run some custom scripting using these modules, and I would like to run them on this testing instance, but I can’t figure out how to install or allow external modules to be executed on the Code module.

Dockerfile:

FROM n8nio/n8n:latest

ARG PGPASSWORD
ARG PGHOST
ARG PGPORT
ARG PGDATABASE
ARG PGUSER

ENV DB_TYPE=postgresdb
ENV DB_POSTGRESDB_DATABASE=$PGDATABASE
ENV DB_POSTGRESDB_HOST=$PGHOST
ENV DB_POSTGRESDB_PORT=$PGPORT
ENV DB_POSTGRESDB_USER=$PGUSER
ENV DB_POSTGRESDB_PASSWORD=$PGPASSWORD


ARG ENCRYPTION_KEY

ENV N8N_ENCRYPTION_KEY=$ENCRYPTION_KEY

CMD ["n8n start"]

railway.toml:

[deploy]
startCommand = "n8n start"
healthcheckPath = "/healthz"
healthcheckTimeout = 300
restartPolicyType = "on_failure"
restartPolicyMaxRetries = 8

Any ideas? Thanks!

Information on your n8n setup

n8n version: 1.9.3
Database (default: SQLite): Postgresql
n8n EXECUTIONS_PROCESS setting (default: own, main): queue
Running n8n via (Docker, npm, n8n cloud, desktop app): railway + docker
Operating system: railway

Jon · October 15, 2023, 4:36pm

Hey @yukyo,

You would need to make a custom image to include the packages you want to use. We don’t have any specific examples for Railway on this but we might already include both of those try adding NODE_FUNCTION_ALLOW_BUILTIN and / or NODE_FUNCTION_ALLOW_EXTERNAL and setting them to * to see if that works.

yukyo · October 15, 2023, 11:59pm

Hi @Jon I tried installing modules but I get this error:

[Region: us-west1]

=========================

Using Detected Dockerfile

=========================


context: bab22e743236f83525a05cec2950d94b

#0 building with "default" instance using docker driver

 

#1 [internal] load .dockerignore

 

#1 transferring context: 2B done

#1 DONE 0.0s

 

#2 [internal] load build definition from Dockerfile

#2 transferring dockerfile: 597B done

#2 DONE 0.0s

 

#3 [internal] load metadata for docker.io/n8nio/n8n:latest

 

#3 DONE 0.6s

 

#4 [1/3] FROM docker.io/n8nio/n8n:latest@sha256:35f27c4692019f1bf363bc15f4834d7429053e5cb9f493046495cc64936a224c

#4 DONE 0.0s

 

#5 [2/3] WORKDIR /usr/local/lib/node_modules/n8n

#5 CACHED

 

 

#6 [3/3] RUN npm install aws-sdk

 
#6 9.689 npm

Oct 16 01:56:31

ERR! code ERESOLVE

Oct 16 01:56:31

#6 9.694 npm ERR! ERESOLVE could not resolve

Oct 16 01:56:31

#6 9.694 npm ERR!

Oct 16 01:56:31

#6 9.694 npm ERR! While resolving: [email protected]

Oct 16 01:56:31

#6 9.694 npm ERR! Found: [email protected]

Oct 16 01:56:31

#6 9.694 npm ERR! node_modules/n8n-nodes-base

Oct 16 01:56:31

#6 9.694 npm ERR! n8n-nodes-base@"1.9.3" from the root project

Oct 16 01:56:31

Oct 16 01:56:31

#6 9.694 npm ERR!

Oct 16 01:56:31

#6 9.694 npm ERR! Could not resolve dependency:

Oct 16 01:56:31

#6 9.694 npm ERR! peer n8n-nodes-base@"1.9.2" from [email protected]

Oct 16 01:56:31

#6 9.695 npm ERR! node_modules/n8n-core

Oct 16 01:56:31

#6 9.695 npm ERR! n8n-core@"1.9.1" from the root project

Oct 16 01:56:31

#6 9.695 npm ERR!

Oct 16 01:56:31

#6 9.695 npm ERR! Conflicting peer dependency: [email protected]

Oct 16 01:56:31

#6 9.695 npm ERR! node_modules/n8n-nodes-base

Oct 16 01:56:31

#6 9.695 npm ERR! peer n8n-nodes-base@"1.9.2" from [email protected]

Oct 16 01:56:31

#6 9.695 npm ERR! node_modules/n8n-core

Oct 16 01:56:31

#6 9.695 npm ERR! n8n-core@"1.9.1" from the root project

Oct 16 01:56:31

#6 9.695 npm ERR!

Oct 16 01:56:31

#6 9.695 npm ERR! Fix the upstream dependency conflict, or retry

Oct 16 01:56:31

#6 9.695 npm ERR! this command with --force or --legacy-peer-deps

Oct 16 01:56:31

#6 9.695 npm ERR! to accept an incorrect (and potentially broken) dependency resolution.

Oct 16 01:56:31

#6 9.695 npm ERR!

Oct 16 01:56:31

#6 9.695 npm ERR!

Oct 16 01:56:31

#6 9.695 npm ERR! For a full report see:

Oct 16 01:56:31

#6 9.695 npm ERR! /home/node/.npm/_logs/2023-10-15T23_56_22_676Z-eresolve-report.txt

Oct 16 01:56:31

#6 9.699

Oct 16 01:56:31

#6 9.699 npm ERR! A complete log of this run can be found in:

Oct 16 01:56:31

#6 9.699 npm ERR! /home/node/.npm/_logs/2023-10-15T23_56_22_676Z-debug-0.log

Oct 16 01:56:31

#6 ERROR: process "/bin/sh -c npm install aws-sdk" did not complete successfully: exit code: 1

Oct 16 01:56:31

-----

Oct 16 01:56:31

> [3/3] RUN npm install aws-sdk:

Oct 16 01:56:31

9.695 npm ERR! Fix the upstream dependency conflict, or retry

Oct 16 01:56:31

9.695 npm ERR! this command with --force or --legacy-peer-deps

Oct 16 01:56:31

9.695 npm ERR! to accept an incorrect (and potentially broken) dependency resolution.

Oct 16 01:56:31

9.695 npm ERR!

Oct 16 01:56:31

9.695 npm ERR!

Oct 16 01:56:31

9.695 npm ERR! For a full report see:

Oct 16 01:56:31

9.695 npm ERR! /home/node/.npm/_logs/2023-10-15T23_56_22_676Z-eresolve-report.txt

Oct 16 01:56:31

9.699

Oct 16 01:56:31

9.699 npm ERR! A complete log of this run can be found in:

Oct 16 01:56:31

9.699 npm ERR! /home/node/.npm/_logs/2023-10-15T23_56_22_676Z-debug-0.log

Oct 16 01:56:31

-----

Oct 16 01:56:31

Oct 16 01:56:31

Dockerfile:25

Oct 16 01:56:31

-------------------

Oct 16 01:56:31

23 |

Oct 16 01:56:31

24 | # Install aws-sdk into the specified directory

Oct 16 01:56:31

25 | >>> RUN npm install aws-sdk

Oct 16 01:56:31

26 |

Oct 16 01:56:31

27 | CMD ["n8n start"]

Oct 16 01:56:31

-------------------

Oct 16 01:56:31

ERROR: failed to solve: process "/bin/sh -c npm install aws-sdk" did not complete successfully: exit code: 1

I always get the same error, any ideas?

yukyo · October 16, 2023, 12:35am

Ok the issue is solved by passing the command:

USER root

Before the npm install on the dockerfile… I feel like a dumb, hours testing different ways to install the package and everything was because of that command

Jon · October 16, 2023, 2:57pm

Hey @yukyo,

Don’t forget to end the file with USER node to set it back as well.

yukyo · October 16, 2023, 2:59pm

Like this @Jon?

Jon · October 16, 2023, 3:04pm

Yeah that will do it, You might not even need n8n worker anymore I think that should just be worker now.

You will also need NODE_FUNCTION_ALLOW_EXTERNAL so n8n knows what it can load.

yukyo · October 16, 2023, 3:07pm

Thanks, @Jon

So:

CMD ["n8n" "worker"] → CMD ["worker"] ? Or you mean the command isn’t needed at all? If yes, then how the image knows it’s a worker?

About the NODE_FUNCTION_ALLOW_EXTERNAL yes I know, thanks for pointing that out, but I include that in my project variables. (I try to make my Dockerfile much dynamic as possible to make the changes in one place):

Probably I should remove the ALLOW_BUILTIN variable, as it doesn’t seem I require it because I’m installing the packages anyway.

Jon · October 16, 2023, 3:38pm

Yup CMD ["worker"] should do it as we started assuming the command would be n8n x in v1.

system · October 23, 2023, 3:39pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.