Split binary file data into individual items

Describe the problem/error/question

Hi I would like to split a single binary file into several binary files.
I’ve found this docs:

That do the opposite, so from x items to 1 item

Do you have some code example that allow me to do that?

What is the error message (if any)?

Please share your workflow

I am downloading a CSV file, binary file, 550MB ish, and I would like to split it in x Binary items

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:

I am downloading a CSV file, binary file, 550MB ish, and I would like to split it in x Binary items

Hi @Gabriele_Bracciali, this seems be to be similar to what you were asking for over here:

You can use pretty much the same approach and have your workflow return the smaller binary files produced by the “Move to file” node in the sub workflow back to the parent workflow. This potentially costs a lot memory though, so make sure to monitor this closely.


Hi @MutedJam , yes actually I haven’t thought about that because I was believing that there was a way to do it without downloading the file.

Hm, I can’t seem think of any way to do this for a file downloaded from an HTTP location, all approaches i can think of would start with fetching that one big file in the first step. You could do it outside of n8n itself through the Execute Command node and a tool like curl or wget, but it would still have to be downloaded.

Or are you perhaps downloading your CSV file from an SSH/SFTP server instead?

I am downloading the file from AWS, but so far I am using the solution above and it’s going well, do you have other idea?

hi @MutedJam , following the workflow above, the files that we are going to create will be deleted after the process? or they will stay in memory?

I don’t think AWS S3 files can be manipulated before downloading them, so I don’t think there’s any way around this.

The example workflow does not delete anything. It would, however, write the file into a /home/node/.n8n/temp/ folder, so to remove the file you could simply run rm /home/node/.n8n/temp/* through an Execute Command node at the end of your workflow to clear this temp folder. This will be easy if you use the new version of the Split In Batches node which has a “Done” output.

okay thanks @MutedJam , so I should add the “execute node” with the command, in the write workflow when the split in batches node will finish, right?

Exactly, that would clear up the temp directory :+1:

1 Like

thank you so much for your help, have a good one!

1 Like

You’re most welcome, have a great afternoon as well!

Hi @MutedJam , I’ve received an error in the “Read line” node:
ERROR: stdout maxBuffer length exceeded

Hi @Gabriele_Bracciali, this could suggest you have very long lines in your file (or perhaps even a single very long line?) . You would need to adjust the splitting logic in the parent workflow in this case.

Which result are you seeing on the Count lines in file node? Is it more than 1 line? If so, you could try reducing the linesPerBatch number on the Prepare batches node in a first step (perhaps try 50 instead of 100 lines per batch).

hi @MutedJam I was setted the value you meant at 1000, maybe it’s for that

1 Like

Hi @MutedJam
Actually after debug I saw that the problem was in the “Read” workflow:
I’ve added a node for remove files

This node was the cause of workflow stuck and crashing, once I remove it everything run smooth

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.