Possible 2GB limit for (S)FTP file download/upload

Describe the issue/error/question

I have a simple workflow downloading some files via SFTP and subsequently uploading them to multiple other SFTP servers. Files below 2GB are processed successfully, but files above 2GB fail.
The error is: ERROR: Error finding file: /root/.n8n/binaryData/4479f5cae949-0e0d-4202-ad88-d0794801c2bd
Interestingly, when I exec into the k8s pod where the worker process runs, the file /root/.n8n/binaryData/4479f5cae949-0e0d-4202-ad88-d0794801c2bd is there and has the correct size.

My n8n server and workers run in Kubernetes and communicate via a redis queue.

What is the error message (if any)?

ERROR: Error finding file: /root/.n8n/binaryData/4479f5cae949-0e0d-4202-ad88-d0794801c2bd

Error: Error finding file: /root/.n8n/binaryData/4479f5cae949-0e0d-4202-ad88-d0794801c2bd
    at BinaryDataFileSystem.retrieveFromLocalStorage (/usr/local/lib/node_modules/n8n/node_modules/n8n-core/dist/BinaryDataManager/FileSystem.js:177:19)
    at Object.execute (/usr/local/lib/node_modules/n8n/node_modules/n8n-nodes-base/dist/nodes/Ftp/Ftp.node.js:501:44)
    at Workflow.runNode (/usr/local/lib/node_modules/n8n/node_modules/n8n-workflow/dist/Workflow.js:658:28)
    at /usr/local/lib/node_modules/n8n/node_modules/n8n-core/dist/WorkflowExecute.js:585:53

Please share the workflow

Share the output returned by the last node

ERROR: Error finding file: /root/.n8n/binaryData/4479f5cae949-0e0d-4202-ad88-d0794801c2bd

Information on your n8n setup

  • PostgreSQL
  • latest image n8nio/n8n, n8n version 0.212.1
  • Kubernetes (Server as k8s Deployment, workers as StatefulSet)

Forgot to mention, that the pod has 10GB RAM assigned and NODE_OPTIONS=“–max-old-space-size=8000”
N8N_DEFAULT_BINARY_DATA_MODE=filesystem

Hi @Janusz_Gryszko, welcome to the community :tada:

[email protected] introduces support for streaming binary data to the FTP node. Could you give this latest version a go and confirm if you’re still having trouble with large files?

1 Like

Hi @MutedJam,
thanks for the tip. I’ll try it out and get back to you asap.

hi @MutedJam,
I finally had time to upgrade n8n to the version you suggested and my workflow was able to successfully process a 6GB file, so it looks like the streaming support for binary data actually works.
Thanks again for the useful response! :slight_smile:

2 Likes

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.