It looks like it has run correctly, but there is a memory error on last step.
The memory issue might be because i’m running it manually (so a copy of the data is saved), but again I get an unknown error when I run it automatically (through the scheduler).
Also : the data is correctly split in chunks (according to memory docs, this is what should be done)
What is the error message (if any)?
There is no error message when I run it through the scheduler, and an “out of memory” error when I run it manually.
Please share your workflow
my workflow is here below : don’t know why it doesnt show :
Share the output returned by the last node
it is executed on a schedule so I cant see the output
Information on your n8n setup
n8n version: 1.0.5 (on cloud)
Database (default: SQLite): cloud (so i guess sqlite?)
n8n EXECUTIONS_PROCESS setting (default: own, main): no idea
Running n8n via (Docker, npm, n8n cloud, desktop app): cloud
I suspect this is a memory issue, when we talk about splitting it into chunks we tend to also mean making a sub workflow to handle the data so rather than loading 22000 items in one go and making 22000 Postgres calls which would also return the items we would put part of it in a subworkflow so you would handle say 1000 at a time then when the subworkflow finishes the memory is released and the main workflow can do the next loop.
In your example I would put the sub workflow between the HTTP Request node and the If and have that do the Item Lists and Postgres.
Hello Jon,
Thank you for the reply.
One question about this : will each sub-workflow execution count as one workflow ? because it might end up doing a lot of workflow executions ? (precisely, the hubspot API returns 500 contacts so I need to call it 100 times, and if I do 101 workflow executions per day the count is going to get high pretty fast).
Or is there a way to “buffer” items (I haven’t found one yet), so that for example I only trigger a sub-workflow when I have 5k lines ?
also : why isn’t the data dropped in progress when it’s no needed by the workflow anymore ?
also : is there a chance it gets better if I drop the unnecessary fields ? or will I have the same problem anyways because all data is saved temporarily to memory ?
I have just given it a quick test and currently sub workflow execution counts don’t appear to count towards cloud limits.
The data isn’t dropped because you might still need the data later in the workflow, We don’t have a node to clear all data at the moment, If you dropped the fields you don’t need it would make it a bit better but the best solution is to use sub workflows.
All shows green, but the execution panels shows a “failed job”
(by the way there is a problem because the job did take a lot more time than 0.14s).
I tried making a recursive workflow but it didn’t work either. The workflow called itself - no problem there, but then executions started failing after they took too much time (because first executions dont “end” until all recursive subworkflows are called).
You need to clear the data before returning to the main workflow. you can do this by using a set node set it to keep only set (not setting any fields) and set it to run only once at the settings of the node (the cog icon)
Edit: the last node executed is returning data to the main flow automatically. so just put a set node at the end with the settings I mentioned.
You did not set it to execute only once it seems. Please see if it works with that setting set.
Although with empty items it should still be fine without the setting.
I just tried that (adding “execute once”) to the “set” node at the end of the sub-workflow, but i’m getting the same issue.
I’ve checked the data and the subworkflow does return just one empty row instead of 500
all sub-workflows finish but the parent workflows logs says “execution failed in 0.118s”
What does the output of the main workflow look like now you have the set node to only run once and I would also add an item like a bool for finished or something so that it moves to the parent workflow, If there is no output item it will just stop.
the output is correctly passed back to the parent (101 items of just one empty object)
so again the whole workflow seems to be alright (and has terminated correctly) but the status keep saying “failed in 0.2s” - although 1- it succeeded and 2- it lasted a lot longer.
