Queue install on railway - community nodes not found

[rpock]

Hello together

Let me try to describe the problem …

Describe the problem/error/question

I installed the n8n template with workers on railway for running in queue mode. I tried everything I found regarding env, docker installation and so on, but on the workers the installation seems not to work.

On the main instance I was able to fix it with N8N_REINSTALL_MISSING_PACKAGES=true but this doesn’t work at the workers.

In the console everything seems to be ok when the docker container gets created.

Docker creation log

#1 transferring dockerfile: 2.31kB done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.n8n.io/n8nio/n8n:latest
#3 DONE 0.4s
#4 [1/4] FROM docker.n8n.io/n8nio/n8n@sha256:e2b40c42b622fea084c2575e7e3682aed47fa8daa71f146794374cf8c868a544
#4 CACHED
#5 [2/4] WORKDIR /home/nodes/
#5 DONE 0.3s
#6 [3/4] RUN npm install n8n-nodes-text-manipulation
#6 3.599
#6 3.599 added 20 packages in 2s
#6 DONE 3.7s
#7 [4/4] RUN npm install n8n-nodes-guuid-generator
#7 1.478
#7 1.478 added 2 packages in 461ms
#7 DONE 1.5s
#8 exporting to image
#8 exporting layers
#8 exporting layers 0.3s done
#8 writing image sha256:1635c08d6851b93b5473cf3278f93270f62fddaf2d92b87accb000fa27f8f9d0 done
#8 naming to us-west1.registry.rlwy.net/1d82ae4b-b1e9-4809-8ef4-5bd3c73b7fc9:1fb24cc6-711b-42b0-ad09-ed3d1390341d7 0.0s done
#8 DONE 0.3s
Build time: 6.39 seconds
================
Publishing Image
================

So in my opinion everything should be fine, both community-nodes are installed with all dependencies, but when the docker container is running, the nodes are missing.

I tried installing it globally, not globally, with root, without root, at different locations, tired to put the nodes directly in the docker container, and sure, I tried everything simultane on workers and main instance.

Main instance is only running because of the N8N_REINSTALL_MISSING_PACKAGES=true env.

I searched everywhere, tried everything, but it seems that I tried the wrong things

Here are the Docker file with some env.

With # marked lines are just a few other tries before

# Use the n8n image as the base
FROM docker.n8n.io/n8nio/n8n

ARG PGPASSWORD
ARG PGHOST
ARG PGPORT
ARG PGDATABASE
ARG PGUSER

ENV DB_TYPE=postgresdb
ENV DB_POSTGRESDB_DATABASE=$PGDATABASE
ENV DB_POSTGRESDB_HOST=$PGHOST
ENV DB_POSTGRESDB_PORT=$PGPORT
ENV DB_POSTGRESDB_USER=$PGUSER
ENV DB_POSTGRESDB_PASSWORD=$PGPASSWORD
ENV N8N_LOG_LEVEL=debug
ENV N8N_REINSTALL_MISSING_PACKAGES=true
ENV NODE_FUNCTION_ALLOW_BUILTIN=*
ENV NODE_FUNCTION_ALLOW_EXTERNAL=*
ENV N8N_LOG_OUTPUT=console
ENV N8N_LOG_LEVEL=debug
ENV N8N_COMMUNITY_PACKAGES_ENABLED=true
#ENV NODES_INCLUDE=[n8n-nodes-text-manipulation,n8n-nodes-guuid-generator]


ENV N8N_ENCRYPTION_KEY=$ENCRYPTION_KEY

#USER root
WORKDIR /home/nodes/
RUN npm install n8n-nodes-text-manipulation
RUN npm install n8n-nodes-guuid-generator

#RUN mkdir ~/.n8n/
#WORKDIR ~/.n8n/nodes
#WORKDIR /home/node/.n8n/
#COPY nodes /home/node/.n8n/nodes
#WORKDIR /urs/local/lib/node_modules/n8n
#WORKDIR ~/.n8n/nodes

#RUN npm install -g n8n-nodes-text-manipulation
#RUN npm install -g n8n-nodes-guuid-generator

#WORKDIR /urs/local/lib/node_modules/n8n
#RUN npm i n8n-nodes-text-manipulation
#RUN npm i n8n-nodes-guuid-generator

#RUN npm install -g n8n-nodes-text-manipulation
#RUN npm install -g n8n-nodes-guuid-generator

#RUN cd /usr/local/lib/node_modules/ && npm install n8n-nodes-text-manipulation
#RUN cd /usr/local/lib/node_modules/ && npm install n8n-nodes-guuid-generator

#RUN mkdir ~/.n8n/nodes
#RUN cd ~/.n8n/nodes
#RUN mkdir ~/.n8n/
#RUN mkdir ~/.n8n/nodes
#WORKDIR ~/.n8n/nodes
#RUN npm install -g n8n-nodes-text-manipulation
#RUN npm install -g n8n-nodes-guuid-generator

#RUN npm i n8n-nodes-text-manipulation
#RUN npm i n8n-nodes-guuid-generator
#USER node


CMD ["worker"]

What is the error message (if any)?

Node not found: n8n-nodes-guuid-generator.UUID Generator
Node not found: n8n-nodes-text-manipulation

Thank’s to anybody who will try to figure out, what’s going wrong with my community-node installation tries

Information on your n8n setup

• n8n version: 1.16.0
• Database (default: SQLite): Postgres
• n8n EXECUTIONS_PROCESS setting: queue
• Running n8n via: Docker via railway
• Operating system: railway

[n8n]

It looks like your topic is missing some important information. Could you provide the following if applicable.

• n8n version:
• Database (default: SQLite):
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app):
• Operating system:

[rpock]

It’s a bit strange, but I’ll try to explain, if anybody run’s in the same issue with a queue worker setup (on railway).

I looked at this before, but when I checked this today, it’s what I thought when I checked a duplicate node in the main editor:

The second TextManipulation is the community node as we see on the right side of it.The duplicate is only there, when I add the following ENV to the main n8n container:

N8N_REINSTALL_MISSING_PACKAGES=true

These re-installed community nodes are working, but as described above, the re-installed community nodes only works in the main n8n container, not in the workers. I’ve tried to add this ENV to the workers, but nothing happens there.

So the manual installed community node get’s successfully installed with the following docker configuration:

WORKDIR /home/nodes/
RUN npm install n8n-nodes-text-manipulation
RUN npm install n8n-nodes-guuid-generator

Don’t forget to set the following, otherwise it wont work

ENV N8N_CUSTOM_EXTENSIONS=/home/nodes/

But I think the N8N_CUSTOM_EXTENSIONS is the problem. No matter what path I’l take to install it (I tried everything, from the n8n documentation like ~/.n8n/nodes/ and much much more), it won’t work without this ENV pointing to the install directory, specified with WORKDIR above.

Here are the difference in the nodes:

The above one is the manual installed, the other one is the community node without the re-install env on the main n8n container.

I thought ok, no problem, let’s replace the community nodes with the manually installed one. Everything seems to be ok, but the manually installed community nodes are throwing an error in the workflow. I had to remove all of them (no matter if active or just unbounded in the workflow) to execute my workflows successfully.

Conclusion

It makes sense, that N8N_CUSTOM_EXTENSIONS points to custom extensions, not to community nodes so it seems they get’s processed differently.

My solution was to remove both community nodes and write my own code, resulted in faster node execution because of more efficient code which fit’s exactly my needs.

Perhaps there is one path in this n8n docker universe I’ve not tried and it will solve the whole problem with installing community nodes manually in queue mode with workers. If a n8n docker pro is reading this, let me know if I missed the glory path

The most simple way to fix this would be, that the workers would also run the function and re-install the missing community nodes (N8N_REINSTALL_MISSING_PACKAGES=true), but no idea if this is possible to integrate. Sure, the startup process of the worker container would be a bit longer, /healthz would take a bit longer to succeed … but who cares

Have a nice day

[MutedJam]

Hi @rpock, wleocme to the community!

I’m sorry you were having trouble with the community nodes, though I sure am glad you found a workaround. Right now your approach to manually install the community nodes for each worker is the approach working for many other queue mode users (see for example this similar thread), though I very much share your hope of seeing a simplified process one day .