Building n8n gives errors

Atul_Lohiya · June 24, 2022, 5:00pm

Hey!

As mentioned in a few of my other questions, I am trying to figure out installing, building and running n8n using source code to be used as a self-hosted solution in my organization. That is the only option that I have due to policies at my organization. Direct installation using any of the methods present in docs is not allowed.

Note that currently I am trying to set it up all on my personal machine but whatever steps I am doing like downloading code, installing dependencies, building etc will all go into a jenkins pipeline and the final thing would be formed into a docker image to run on our openshift clusters.

Note that my organization does not allow directly downloading anything from github or external npm repos either. So the entire n8n code and ALL its dependencies will need to be downloaded separately in a controlled environment and built and run to ensure everything works. Once that is established, the n8n code will be uploaded to internal git repo and dependent packages needed will be loaded into inhouse npm repository from where they will need to be installed during deployments using jenkins pipeline.

So there is no command that I can run manually for each individual package. I was expecting that since n8n is being used in prod environments of a lot of clients, the code on above repo would be up-to-date. I hope and think it is but I am not sure as tests are failing reporting this:

test/TelemetryHelpers.test.ts(1,72): error TS2307: Cannot find module ‘uuid’ or its corresponding type declarations.

I understand I need to install this dependency but I cannot be installing each individual dependency like this. I have an entire document recording whatever steps I am taking and output from bash console but I don’t see any way to attach the file here. If there is a way to share the file, please let me know.

So here is what I did on my personal machine with Windows 10 Home:

I created a folder in f:\n8n into which I cloned the repo so the root folder is f:\n8n\n8n
Into f:\n8n folder, I downloaded the n8n code with git clone “GitHub - n8n-io/n8n: Free and open fair-code licensed node based Workflow Automation Tool. Easily automate tasks across different services.”
I built it with “npm run build” but it fails saying lerna is missing
I installed ALL dependencies from root folder with “npm install”
I build again but this time it fails saying tsc is missing from packages\workflow and gulp is missing from packages\design-system
I CD into respective package folders and install ALL dependencies with “npm install”
I build again and this time it shows an error about uuid:

test/TelemetryHelpers.test.ts(1,72): error TS2307: Cannot find module ‘uuid’ or its corresponding type declarations.

I was expecting that when I install dependencies as I mentioned above, it should be working fine since this file is in design-system folder for which ALL the dependencies were installed in above steps.

Below is the build command and the output of it:

$ npm run build

[email protected] build F:\n8n\n8n

lerna exec npm run build

lerna notice cli v3.22.1

lerna info versioning independent

lerna info Executing command in 7 packages: “npm run build”

[email protected] build F:\n8n\n8n\packages\workflow

tsc

[email protected] build F:\n8n\n8n\packages\design-system

npm run build:theme

[email protected] build:theme F:\n8n\n8n\packages\design-system

gulp build:theme

[20:39:31] Using gulpfile F:\n8n\n8n\packages\design-system\gulpfile.js

[20:39:31] Starting ‘build:theme’…

[20:39:31] Starting ‘compileTheme’…

[20:39:34] Finished ‘compileTheme’ after 3.15 s

[20:39:34] Starting ‘copyThemeFonts’…

[20:39:35] Finished ‘copyThemeFonts’ after 69 ms

[20:39:35] Finished ‘build:theme’ after 3.23 s

test/TelemetryHelpers.test.ts(1,72): error TS2307: Cannot find module ‘uuid’ or its corresponding type declarations.

I searched for uuid package/module and I could find it in following places so it should not be failing to build. So how do I fix it now?

n8n/node_modules/request/node_modules/uuid
n8n/node_modules/temp-write/node_modules/uuid
n8n/packages/design-system/node_modules/uuid – this package is where the erring file is!
n8n/packages/design-system/node_modules/sockjs/node_modules/uuid
n8n/packages/design-system/node_modules/node-notifier/node_modules/uuid

Jon · June 27, 2022, 9:46am

Hey @Atul_Lohiya,

Looking at the steps you are taking you have not ran the lerna command which is what does a lot of the magic. The best solution might be to remove what you have done so far and follow the steps we have documented here: n8n/CONTRIBUTING.md at master · n8n-io/n8n · GitHub

Looking at the issues listed this should solve most if not all of your problems.

Atul_Lohiya · June 28, 2022, 12:04pm

Hi,

thanks for the response. As I mentioned in the steps, I am installing each package’s dependencies one by one so I know which package has what dependencies that are causing problems.

There is no lerna command in the root package.json for installation of packages. There is lerna command for building the packages. And that is what I am using. When I run command “npm run build”, it executes the lerna command “lerna exec npm run build”.

So all that is fine. What I was looking for is a single command that would install all the dependencies from all the packages so I don’t have to fix each package one by one.

Also, I was hoping that all these steps would be documented somewhere and that the things would already be in a stable state in terms of building and running.

But it doesn’t look like it looking at the compilation and installation errors that I am getting. Nonetheless, I am going through the steps at the link you mentioned. Will keep you posted.

Jon · June 28, 2022, 12:29pm

Hi @Atul_Lohiya,

You don’t need to go into each package one by one that is handled for you, If you run lerna bootstrap --hoist it should fetch all the packages needed and it does some linking so you don’t run into issues. You can find the lerna config file here: n8n/lerna.json at master · n8n-io/n8n · GitHub

More information on how the Lerna bootstrap command works can be found here: lerna/commands/bootstrap at main · lerna/lerna · GitHub

Atul_Lohiya · June 28, 2022, 1:56pm

Yes I understand now that I have gone through the docs. Earlier I didn’t know where to look for the steps. This was helpful, thanks much.

One question is about the tech stack used: can you help me with what technologies are used both on client and server side? I didn’t see any comprehensive list of techs that n8n uses. I am only aware of typescript and vue. How about any messaging queues that might be getting used to run the workflows and may be something else?

Is there any doc which explains the entire architecture of n8n as in how requests are recieved on server and executed and how communication between various nodes happens specially when there is a delay or failure?

Jon · June 28, 2022, 2:18pm

Hey @Atul_Lohiya,

I don’t think we have a diagram to show that, We use Typescript, Vue and Sqlite out of the box but if there was any scaling required that would introduce a database like Postgres or MySQL, Redis and Bull for queues (Configuring queue mode - n8n Documentation).

Requests are recieved based on the triggers most of which are HTTP based, The communication between nodes is possibly less important as with n8n it isn’t like each node needs to run on its own container like some other products everything is ran in the one instance.

Atul_Lohiya · June 28, 2022, 3:32pm

@Jon,

I was running the steps mentioned in the docs to setup n8n but this seems to be stuck at

npm install -g windows-build-tools

for windows. I have Windows 10 Home. It has been almost 40 mins or so and it is stuck after showing below deprecation warnings:

$ npm install -g windows-build-tools
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: Node.js now includes build tools for Windows. You probably no longer need this tool. See https://github.com/felixrieseberg/windows-build-tools for details.

Please also notice the last warning about build tools.

To ensure I have at least the version of node.js mentioned in the docs or higher, I installed node.js v16.14.2 as that is what is allowed at my org at this point. This installed python 3 too. It also comes packed with windows build tools but due to internet connectivity issues or something else may be, this last step of installing the build tools failed.

After that I am continuing with rest of the steps but it seems to have been stuck at step 2 itself i.e., installing the build tools.

Any ideas what could be going wrong or should I abort and try running again? Will it download the tools again?

Jon · June 28, 2022, 3:47pm

Hey @Atul_Lohiya,

That looks like it is probably still running, You could cancel it and try again to see what happens it should be fairly quick to catch up if it has already installed some of it. Although it could be for Windows Build Tools it has to fetch other dependencies and that is where it is slow.

Atul_Lohiya · June 28, 2022, 4:18pm

It is still stuck, let me try it again.

Meanwhile, a few questions about execution of workflow:

Is there a doc that explains how workflow executes?
What happens if a workflow fails at any node? Will it resume the next time it runs from point of failure? This has to be case based because not every workflow is fit for resuming from point of failure and some workflows may need that capability to resume. I think if it is trigger based workflow, it will should not resume because the workflow should execute based on the current trigger and not previous trigger.
Does n8n use redis for message queueing by default with a single instance of it running? Almost all tools that I tried to understand so far use a message queue irrespective of number of instances of workflow tool because it allows better scalability and performance as next node may be executed by another instance (if more than one are there), provides fault tolerance as in if workflow execution fails at a node, it can be resumed based on messages or something.

I also went through the queue mode docs and I think that is the default way for me to go. That is because we deploy our apps on atleast two docker containers on our openshift clusters. That means there would be two instances of n8n running at least.

Now I have never worked with redis before so is there anything that helps me understand how it works specially with n8n and how to troubleshoot if something doesn’t work? I know I can do a google search but just asking if you or n8n docs have something handy to get started.

How do I setup MySql db for n8n?
Is there any installation guide or any DB scripts that create the needed schema, tables, stored procs etc to store the execution and other workflow information like status for execution of nodes and workflow as a whole and whatever else?
Is there any doc that explains the relations between various DB tables/views to understand what data goes where and how it all links to each other?
Are there any code or config changes needed to enable MySql instead of default SQLite?

Jon · June 29, 2022, 8:45am

Hey @Atul_Lohiya,

There is nothing technical that explains how a workflow executes but it depends on what you are after, We have documentation that covers setting up workflows and the data structure but really for a workflow to run it depends on what trigger you are using.

Workflow Docs: Overview - n8n Documentation

If a workflow fails what happens can depend on what the failure was and how your workflow has been built. We have options to retry the node and the ability to call another workflow on error for notifications. You can also tell nodes to just continue on error then you can check the output and build out your own logic for saving to try again if needed.
No, n8n has no need for Redis in a single instance mode. While it could possibly help in some instances for most cases it would just be an extra service for users to worry about. Your comment about scalability is correct which is why if you have multiple instances of n8n running that is where we do use Redis for the reasons you have mentioned in a single instance environment this is unlikely to be a problem.
Set up a MySQL database and user like you normally would then update the environment options for n8n and it will take care of creating the schema on startup.
DB Setup: Supported databases and settings - n8n Documentation
DB Structure: Database structure - n8n Documentation

Atul_Lohiya · June 29, 2022, 1:42pm

Ok. I will have to install MySql first and then configure n8n to use that instead of SQLite. Just to note, I am currently doing all this on my Windows 10 Home machine so it will be single instance.

Can you please confirm if it is necessary to run

npm install -g windows-build-tools

step as mentioned in docs “if I first install node.js” first?

The above command kept hanging forever! I had been trying to run it since yesterday but it consistently hung. I was finally able to run it successfully just now based on inputs on Stuck at `Still waiting for installer log file...` · Issue #208 · felixrieseberg/windows-build-tools · GitHub and Cannot install windows-build-tools · Issue #179 · felixrieseberg/windows-build-tools · GitHub with command

npm install --global --production windows-build-tools --vs2015

I installed node.js v16.14.2 which also installs python 3 and windows build tools. Though in my case even that failed so I had to be running the above command manually to ensure things don’t break. This is the log screenshot:

Actually I just noticed, it says it was able to install visualstudio2019buildtools v16.11.16.0 but failed to install visualstudio2019-workload-vctools. Can you tell the difference? is the failed module required?

Jon · June 29, 2022, 1:49pm

Hey @Atul_Lohiya,

We don’t control that package but if the package says it is needed then it will be required, I am not sure why it is failing but maybe the log has more information.

Atul_Lohiya · June 29, 2022, 3:19pm

It only supports VS version 2015 and 2017, that’s why it was failing!

Jon · June 29, 2022, 3:25pm

Well that is a pain, I wonder why that is.

Atul_Lohiya · July 4, 2022, 11:46am

HI,

I have been able to create my first workflow with triggers. Thanks for the support so far. This helped me understand things like webhook and triggers better. And that also made raise new questions!

When a webhook/trigger node is used, there is a new path and thus a new webhook url automatically generated for each instance of that node. Is it possible to generate the same webhook url no matter what? The reason is: there can be multiple users each with multiple workflows listening to triggering events from a common app. That app needs to have one single n8n webhook that it can invoke irrespective of user and workflow. The app cannot keep multiple webhook urls for each such instance of webhook/trigger.
I see there is this .n8n folder which has the config file which stores an encryption key for credentials. Docs also mention that this folder is required for that very reason. In containerized environment it may not be possible to retain that without mounting a volume. And that is not allowed in my org as of now. Doc at Docker - n8n Documentation also mention that the folder is not explicitly required when using alternate DBs but recommended. The encryption key can be passed in env variable.

I was looking for a more dynamic approach to it rather than statically providing the key in a file or env variable.

So is there a way the credentials (and whatever else goes into that file) can be persisted in DB and read across restarts? Once n8n restarts, it can read that info from DB and put it in the said folder for its working. Whenever, the file is updated, that info can also be persisted in DB.
What encryption algorithm is used by n8n for encrypting and decrypting the credentials? Can a single encryption key be enough for both encryption and decryption or a public-private key pair is needed? If a pair is needed, how to inform n8n of the private key assuming the public key can be passed in env variable (or read from DB if that is possible).
That information is needed to generate the correct key.

Jon · July 4, 2022, 12:14pm

Hey @Atul_Lohiya,

The URLs are randomly generated for trigger nodes, I have not yet seen this cause any problems with any of the apps we currently support. Most of the time if the third party system doesn’t allow multiple webhook URLs you can work around it with the events you subscribe to and you can then filter them out in your workflow. If the URL was always the same I am not sure how the node would know which user or node instance it is meant to run for.

Can you share more about the system you are integrating with?

If you are not allowed to use volumes the only option is to use an env variable or a config file that would need to be mapped somehow to the container. The encryption key is used to keep the credentials safe, Maybe a crazy solution would be to use the API to create credentials everytime the instance is started up but that is going to be a lot of work.
Only one encryption key is needed which should be more than enough assuming this key is kept safe, The encryption is done using AES and can be found here: n8n/Credentials.ts at 0af3ccf35f4e63322e83b523b62c705d29bb37c9 · n8n-io/n8n · GitHub

Atul_Lohiya · July 4, 2022, 1:22pm

I could actually figure it out! There is this “path” field which actually makes the webhook url unique. SO if this is set to the same value in any instance of webhook node, they all would be able to listen on the same url and thus be notified by a common app at the same time! That is my use case actually and probably true for anyone. It does not matter who the user is or which instance of webhook node is getting triggered. If consumers of an app need to be notified of an event from that app, all the users need to be notified the same way. The app can invoke a single webhook url which is present in multiple webhook instances for multiple users’ workflows. And that should do the trick.
Let me know if my understanding right or would create a problem!
API would be the ideal thing and surprisingly n8n does not have that. That to me looks like a default path to take to decouple various parts of n8n. Nonetheless, will need to check what can I Do about it!
Will check the docs. I was just trying to understand how to generate the encryption key and provide via env variables. If there is just one key that is needed then it works but if it was public-private key pair, then it won’t be possible to pass via env variables too.

Jon · July 4, 2022, 1:35pm

What is it you are acutally trying to do? I have used one URL for multiple users and have used a workflow to work outwhere it should go. If you have one URL and say 2 workflows using that same URL how would it know which workflow to run? We also don’t allow the same URL internally when activating a workflow anyway so being able to set it to a static value would break something.
There is an API that can be used to add credentials to n8n but having an API to add an encryption key which is loaded at startup may not make much sense.
You can generate the encryption key however you want it is just a string and be any phrase or set of characters that you want to use. There is no need to worry about key pairs

Atul_Lohiya · July 4, 2022, 1:52pm

You are right about not allowing same webhook url on multiple webhook nodes! It throws error when I try to do that! I don’t understand the restriction really. What I am trying to do is just normal scenario I guess, may be I need to understand it better. This is what it is hypothetically:
There is an app say new employee onboarding. There are multiple other apps/people who are interested in knowing as soon as an new employee is onboarded - finance, background check, development team, infra team etc., so that they can initiate their own workflows say opening a bank account, initiate a detailed background check, order development machines with needed software preinstalled, setup phone, desk etc.
Now each user from these four teams goes in and creates a workflow starting with a webhook node. How does the onboarding app notify each app via those four or more workflows? Onboarding app won’t be maintaining a list of multiple webhook urls. The app can only invoke one url. And just like pub-sub design pattern, all the workflows that have the same webhook must be run! It really does not matter who the user is. If a workflow is using that webhook, it will be run. I am not sure what am I missing in this wrt to n8n working.
It makes sense if you think decoupling and dynamic execution specially in conatainerized scenarios but that is true in any case really. The keys and such things are sensitive pieces of data and should be not accessible and provided statically. That’s where APIs provide better security and scalability.
Not just that, it makes way more sense to allow fetching things like DB passwords for alternate DBs, that is a very obvious use case where static value should never be provided but be fetched from a vault/secrets manager or something.
So the same key is used for encryption and decryption?

Jon · July 4, 2022, 2:18pm

I would say if it is one app you only need one node in one workflow, Assuming it is sending the data the workflow can deal with the data and filtering it out. I have not yet seen an app that only allows one outbound webhook which could be why I am not able to picture it.
It would make sense but that is not how n8n is built and we do have APIs for making credentials but it would need to be done on instance creation and if you were running in scaling mode you would need a way to share that decryption key between all application nodes in the cluster.
Yes the same key is used for encryption and decryption, This is how the AES implementation works.