HTTP module was working fine, then after about 750 runs started throwing this error. I have N8N_DEFAULT_BINARY_DATA_MODE=filesystem, so I’m guessing the filesystem filled up. But I’m running on containers, so I can’t log in and check.
I’m going to reboot and turn off the “Save executions” options to see if that fixes this. If not, I’ll remove the N8N_DEFAULT_BINARY_DATA_MODE environment variable for now.
Request: if this -is- because the filesystem filled up, please handle this error better.
What is the error message (if any)?
Error: ENOENT: no such file or directory, mkdir '/home/node/.n8n/binaryData/workflows/CiCtWg8yX1sQ1NpA/executions/35256'
Please share your workflow
Entire workflow is too large, but here’s the HTTP module:
Update: I was able to shell into the worker container in my cluster and confirmed that the filesystem is full. Will see if turning off “Save execution data” solves this problem. Otherwise I’ll turn off keeping binary data in the filesystem.
Another update: Turning off the “Save execution” options does not affect whether the binary data is cleaned up from the server after execution is complete. So there needs to be better handling of binary data to avoid all of this.
The binary data should be cleaned up as part of the standard execution cleanup but the workflow would need to finish so if your workflow has a 750 item loop it likely wouldn’t clean it up until that workflow finishes.
I think @ivov may have made some changes to binary data cleaning recently to fix some issues though so maybe an update could help.
Hey @Jon, thanks for replying quickly. I can confirm that’s NOT the case. This workflow is set up to process one airtable row at a time and then exit. A schedule trigger kicks off a new execution every 3 seconds to work on another row. I was able to log into the container and could watch the number of subfolders in /home/node/.n8n/binaryData/xxxxx/executions/ continue to grow as each new execution ran.
I’m trying to update my configuration to use S3 for binary data, but haven’t gotten it working yet.
Perfect, So you basically have a workflow that runs every 3 seconds and it reads one item from airtable and downloads a binary file and the issue at the moment is the binary data is not being cleared as expected?
In theory would be the below be enough to reproduce it?
That is correct, glad you could reproduce it so easily.
I was able to turn off filesystem storage for binary data but now my workflows take a LOT longer to run. (Like 60ish seconds per execution instead of 4-8 seconds). So I’m going to go back to filesystem storage for binary data and just keep clearing out that directory manually for now.
So I have updated my n8n instance to 1.21.0 and cleared my binaryData folder then activated the workflow and now the files are created (will create an internal bug ticket for that now) but the contents are no longer kept so I believe an upgrade will solve this for you.
Temporary fix (in case someone finds this thread later):
If you’re on a full unix install, edit your crontab to run the script below periodically.
If you’re running n8n in containers, you have to turn on crond & put the script in the right place:
Figure out how to shell into your container. If you’re using docker, you probably want docker attach. If you’re running in Elastic Container Service like I am, open up Cloudshell and use aws ecs execute-command help to figure out all of the command line options you need.
su to root if you’re not already root
crond start
Put the shell script below into /etc/periodic/15min or hourly depending on your needs.
Note: the above works on a running container ONLY. It will not persist between new container launches unless you modify your Dockerfile. Since my job runs on a schedule, I did this on all of my worker containers. You might need to do that on your webhook container(s) or your main container too.
The +2 means delete files with last modified more than 2 mins ago. My workflow takes no more than 20 seconds to execute so that’s a safe age for me, but adjust for your needs. Also change the search directory if you have n8n installed somewhere else.
Hopefully N8N fixes this soon and this workaround is no longer needed by the time you’ve found this thread.
That is interesting, In my test with my scheduled workflow the folders were created but the contents were empty. As your path has undefined in it though did you use a saved and scheduled workflow or were you manually running the workflow?
A manual run results in the folder and file being created and not deleted until the timer kicks in as there is a chance you may be working on your workflow and would need to access the data.
To fix removing the empty folders we have the PR below being worked on which will resolve the last part of this.
I did turn saving the executions back on for this workflow. It’s running from a schedule.
I’m not sure if this is related, but if I monitor executions in the UI, they show up as “[UNKNOWN]” or something like that when they’re running. (I didn’t take a screenshot but I can try to if that’d be helpful.)
Once they finish executing, the name of the workflow is displayed correctly in the executions list. This is on 1.21.0.
