Describe the problem/error/question
My workflow is failing and I have no clue why.
The job is running, then at some point it “fails”, and all jobs are ticked green.
a weird step : i have 44* 1000 contacts, but only “23k inserted” is displayed
after running it manually I actually get this :
It looks like it has run correctly, but there is a memory error on last step.
The memory issue might be because i’m running it manually (so a copy of the data is saved), but again I get an unknown error when I run it automatically (through the scheduler).
Also : the data is correctly split in chunks (according to memory docs, this is what should be done)
What is the error message (if any)?
There is no error message when I run it through the scheduler, and an “out of memory” error when I run it manually.
Please share your workflow
my workflow is here below : don’t know why it doesnt show :
Share the output returned by the last node
it is executed on a schedule so I cant see the output
Information on your n8n setup
n8n version: 1.0.5 (on cloud)
Database (default: SQLite): cloud (so i guess sqlite?)
n8n EXECUTIONS_PROCESS setting (default: own, main): no idea
Running n8n via (Docker, npm, n8n cloud, desktop app): cloud
Operating system: N/A?
Welcome to the community
I suspect this is a memory issue, when we talk about splitting it into chunks we tend to also mean making a sub workflow to handle the data so rather than loading 22000 items in one go and making 22000 Postgres calls which would also return the items we would put part of it in a subworkflow so you would handle say 1000 at a time then when the subworkflow finishes the memory is released and the main workflow can do the next loop.
In your example I would put the sub workflow between the HTTP Request node and the If and have that do the Item Lists and Postgres.
Thank you for the reply.
One question about this : will each sub-workflow execution count as one workflow ? because it might end up doing a lot of workflow executions ? (precisely, the hubspot API returns 500 contacts so I need to call it 100 times, and if I do 101 workflow executions per day the count is going to get high pretty fast).
Or is there a way to “buffer” items (I haven’t found one yet), so that for example I only trigger a sub-workflow when I have 5k lines ?
also : why isn’t the data dropped in progress when it’s no needed by the workflow anymore ?
also : is there a chance it gets better if I drop the unnecessary fields ? or will I have the same problem anyways because all data is saved temporarily to memory ?
I have just given it a quick test and currently sub workflow execution counts don’t appear to count towards cloud limits.
The data isn’t dropped because you might still need the data later in the workflow, We don’t have a node to clear all data at the moment, If you dropped the fields you don’t need it would make it a bit better but the best solution is to use sub workflows.
Executing this workflow did work - but the workflow crashed again
All shows green, but the execution panels shows a “failed job”
(by the way there is a problem because the job did take a lot more time than 0.14s).
I tried making a recursive workflow but it didn’t work either. The workflow called itself - no problem there, but then executions started failing after they took too much time (because first executions dont “end” until all recursive subworkflows are called).
So I don’t have a solution right now.
You need to clear the data before returning to the main workflow. you can do this by using a set node set it to keep only set (not setting any fields) and set it to run only once at the settings of the node (the cog icon)
Edit: the last node executed is returning data to the main flow automatically. so just put a set node at the end with the settings I mentioned.
Well, same problem :
the sub-jobs all executed successfully, but the parent job won’t terminate correctly (and i don’t know why it sais “failed”).
i checked the output of “set”, all items are empty, so the subjob does return 50k empty lines
You did not set it to execute only once it seems. Please see if it works with that setting set.
Although with empty items it should still be fine without the setting.
I just tried that (adding “execute once”) to the “set” node at the end of the sub-workflow, but i’m getting the same issue.
I’ve checked the data and the subworkflow does return just one empty row instead of 500
all sub-workflows finish but the parent workflows logs says “execution failed in 0.118s”
What does the output of the main workflow look like now you have the set node to only run once and I would also add an item like a bool for finished or something so that it moves to the parent workflow, If there is no output item it will just stop.
The parent workflow looks ok
i’ve done it this morning again to show :
the output is correctly passed back to the parent (101 items of just one empty object)
so again the whole workflow seems to be alright (and has terminated correctly) but the status keep saying “failed in 0.2s” - although 1- it succeeded and 2- it lasted a lot longer.
Can you share the workflow?