Facing connection lost error for handling large data

Questions
1

I have this workflow where i need to handle large data(almost 48MB) which i get from a sub workflow but n8n gives connection lost error in mainworkflow . i think it is crashing there is no way i can reduce the data because i need all data to be processed further . We are on Enterprise Plan. can you please guide what else can be done

2

Sounds like you have hit the memory limit, @mylearnings_abu
This may help

3

Welcome @mylearnings_abu to our community! I’m Jay and I am a n8n verified creator.

The core issue is that n8n loads all items into memory at once - 48MB of raw data will easily hit the limit. Instead of passing the full dataset through the sub-workflow output, restructure so the sub-workflow returns only IDs or references, then process the actual data in batches in the main workflow using a Split in Batches node (batch size 50-100). If you need to fetch the data per record, do it inside the batch loop with an HTTP Request or database node. This keeps memory usage flat regardless of total dataset size.

4

Hi @mylearnings_abu
This might help:

5

Actually in my case the subworkflow is returning data from db all these data are needed and then passed on further to do some manipulation which is again a subworkflow , so there is no way i can reduce anything in 48MB

6

Also i have one more query like if we have a json file in s3(array of objects) how can we read data in batches ? Is it like we GET complete file and then perform batching?

7

Hi @mylearnings_abu you are correct, you can just download the JSON from the AWS S3 node, and once the file is there, you can extract its JSON content by using the extract from file node, and once you have the JSON extracted, you can either add a loop which is quite easy or setup a code node if you want some specific operations to be performted on the data. One problem I can see right away is that the file that you would get would be very huge, as n8n currently does not support streaming partial reading. So just be mindful of the memory usage.

8

Better to redesign your workflow to work in batches (fetch some data, process it, rewrite/modify), instead of fetching all data and then processing it