Cannot list files of a big SFTP

Questions
1

Describe the problem/error/question

I have a very simple workflow which consists of listing all files under a certain directory on a SFTP.
It should be noted that this SFTP has quite a lot of files, which amount to approximately 1,100,000.

When running this workflow, after a few minutes of processing, the workflow fails in the listing FTP node.

What is the error message (if any)?

RangeError: Maximum call stack size exceeded
    at Object.execute (/usr/local/lib/node_modules/n8n/node_modules/n8n-nodes-base/dist/nodes/Ftp/Ftp.node.js:489:46)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Workflow.runNode (/usr/local/lib/node_modules/n8n/node_modules/n8n-workflow/dist/Workflow.js:706:19)
    at /usr/local/lib/node_modules/n8n/node_modules/n8n-core/dist/WorkflowExecute.js:656:53
    at /usr/local/lib/node_modules/n8n/node_modules/n8n-core/dist/WorkflowExecute.js:1058:20

Please share your workflow

Share the output returned by the last node

Information on your n8n setup

  • n8n version: 1.27.2
  • Database (default: SQLite): Postgres
  • n8n EXECUTIONS_PROCESS setting (default: own, main): default
  • Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
  • Operating system: Amazon Linux 2023
2

hello @ESOPred

Usually in such cases there should be some kind of pagination be implemented or filter, but I see that the FTP node does not have such an ability.

And all objects are in one folder?

3

Hello @barn4k ! Thank you for your prompt reply. :wink:

Yes, that’s what I was looking (and hoping) for when using this feature for the first time, as it would have also helped me to reduce the memory print of each workflow execution.

All objects are indeed in one folder, which is not very convenient for my use case I suppose.

4

Hey @ESOPred,

This looks like a bug we need to fix, I have let the team know so we can get this sorted. Although that many files in one directory is generally not a good idea and I have seen specialist file transfer products have issues with less.

5

This will be fixed soon in fix(FTP Node): Fix "Maximum call stack size exceeded" error when dealing with too many files by netroy · Pull Request #8657 · n8n-io/n8n · GitHub

6

New version n8n@1.30.0 got released which includes the GitHub PR 8657.

7

Thank you very much for making this fix so quickly!