Hi, I’m an Integromat user who is looking to run an operation-heavy part of the workflow on n8n.
Describe the issue/error/question
I’ve built a workflow that runs successfully with very few items. It gets data from Excel, iterates over it, IF condition takes care not to process the header row, makes API call to Email Finder service for each row, aggregates data, and uploads them back to Excel with 2 different calls. There are also 2 Set nodes and 2 Code nodes (index items and count rows).
I couldn’t run the workflow successfully with 2000 items with (7-10MB JSON). I read community posts and learned that n8n can’t handle big amount of data in a workflow. I’ve tried split batches but it didn’t help. Also tried it with non-manual execution. I know I can call another workflow, but wanted to avoid this option.
So I removed all the nodes that are not absolutely necessary and I’m sending data from Integromat in batches. However, n8n can not even process 100 items. What’s worse, n8n.cloud constantly freezes to the point, when it works only if I’m not doing anything.
I’ve tried to delete the executions but I didn’t succeed. I disabled the option to save failed executions. I’m not saving execution progress. Upgraded and downgraded to 0.201.0 / 0.200.1 respectively.
What is the error message (if any)?
top-right corner - Connection lost
execution log - Status unknown
when deleting execution - Problem deleting executions Request failed with status code 503
on random occasions - Problem loading data Request failed with status code 503; 503 Service Temporarily Unavailable nginx; Problem loading workflows Request failed with status code 503
Please share the workflow
I do not have the Start node in my original workflow.
Share the output returned by the last node
Successful runs with very few items pass data to Integromat webhook:
[
{
"data":"Accepted"
}
]
Information on your n8n setup
n8n version: 0.201.0 and 0.200.1; currently 0.200.1
Database you’re using (default: SQLite): no idea
Running n8n with the execution process [own(default), main]: no idea
Running n8n via [Docker, npm, n8n.cloud, desktop app]: n8n.cloud
Hi @antitalent, I am very sorry to hear you’re having trouble. A 503 status would indicate the n8n instance behind the web server is unavailable which is usually due to a lack of memory. I had a quick look at your server logs and could indeed spot a couple of FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory errors.
From looking at your workflow, could you perhaps move the part starting from the email finder into a sub-workflow? Then use a Split In Batches node in your main workflow to split the execution data in batches of 1, call your sub-workflow using the Execute Workflow node. As long as your sub-workflow doesn’t return the full dataset this should reduce the memory footprint significantly (as the memory used by the sub-workflow will become available again after the sub-workflow finishes).
If that’s not an option for you, you might want to consider self-hosting n8n. In this case you’ll be able to monitor the memory consumption and can allocate more memory to your instance as needed.
I know this isn’t a great experience and there is an ongoing initiative for improving the memory handling at n8n, but I don’t have a timeline for this unfortunately.
Thank you @MutedJam. Would you be able to delete all my execution history, please? I’m not able to do it - when I click “Executions” n8n.cloud freezes and I have to wait for a couple of minutes to access it again.
Hey @MutedJam, just wanted to give you a quick update. Splitting the data and calling a sub-workflow works fine but I’m not sure 5000 executions would be enough for me.
Therefore I decided to self-host but ofc I have no idea what I’m doing and I have little faith I would be able to configure anything or solve any server issues (and there will be some, I’m sure). So I’m willing to go back to the cloud once the memory issues are resolved. But for now, I’m using the self-hosted solution, so you’ll probably hear from me again
So, I am a big fan of self-hosting things in general (I even wrote an entire tutorial on how to do so with n8n).
That said, sub-workflows alone shouldn’t be a problem on n8n cloud. This is because only the parent execution would be counted. So even an execution like below (using these example workflows) would count as 1 execution, despite the sub-workflow running 37 times:
