I am experiencing slow execution of nodes with low utilization of system resources.
Google Drive Node:
I am using Google Drive nodes to find all files in a folder and download them. To test, I limited it to 20 results, which is maximum 200 MB in total. I have a very stable 10 MB/s connection. Still it takes multiple minutes to execute. I can’t see any fast downloading connections using tcptrack either.
Text Splitting Node:
I am using the Recursive Text Splitter Node to split these documents into chunks. My systems resources are barely used, both RAM and CPU. It takes way over 20 minutes to complete though.
Information on your n8n setup
n8n version: 1.75.2
Database (default: SQLite): Postgres
n8n EXECUTIONS_PROCESS setting (default: own, main): I can’t see the env to be set when looking at the docker variables, so I assume its default. I didn’t change it either.
Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
Operating system: Running it on a Jetson Orin Nano Developer Kit, it’s OS is based on Ubuntu Desktop
Also, I realised running the workflow is leading to my browsers RAM consumption skyrocketing. However, I am accessing the n8n web ui from a different device than the n8n docker container is running on.
Is n8n doing the text splitting using the browsers computers resources? I assumed all the actual work will be done on the n8n host and the accessing webclient only does frontend workload like rending the canvas, etc.
Okay, I’ve been toying around with EXECUTION_DATA envs and CLI vs. WebUI, and I narrowed it down to this, maybe someone could confirm:
WebUI doesn’t display the actual state of the workflow. Due to trying to make all execution data accessible to the user in the WebUI, n8n pushes a lot of data like the read in binary files, the extracted text, etc. This leads to the high browser memory consumption. It also explains, why I didn’t saw any resource utilization - the workflow was long executed and done, the UI didn’t update yet tho.
There is an issue with saving anything bigger than 2 GB to the database, due to string limitations in JS. That means, if all nodes combined at some point try to process data >2 GB, the execution will fail. This can be prevented somewhat by disabling all “while process” related storing of EXECUTION_DATA, leading to only the final push to DB failing and the workflow completing sucessfully still.
The given heap size of the internal nodejs can get problematic when working with larger amounts of data, also due to the internal storing of EXECUTION_DATA, e.g. copies of the same documents on all nodes after the first one dealing with it. Passing the respective NODE_OPTIONS fixes this.
Am I right? And is there any way to exclude nodes or data fields from the final DB push or stop n8n from pushing the whole EXECUTION_DATA to the WebUI and instead just push “success/error”?