Download large file from s3

[sultan]

Hi, I am trying to download a json file from S3, file size is 2 gb and got the following error:

Error: Cannot create a string longer than 0x3fffffe7 characters

n8n is running in docker and the machine RAM is 32 gb

[Jon]

Hey @sultan,

Can you try setting the environment variable below to see if it helps?

N8N_DEFAULT_BINARY_DATA_MODE=filesystem

[RicardoE105]

This is a known limitation of the way n8n handles files by default. As @Jon suggested, changing the binary data mode to the filesystem should do the trick.

ERROR: Cannot create a string longer than 0x3fffffe7 characters

Yes, that will currently not be possible with n8n. The reason is the design. It was simply never written to handle huge files. That is maybe something that will change in the future but that would require a bigger refactor. Technical explanation. All data in n8n gets saved as JSON. All binary data gets converted to base64 and so also saved as a string. The maximum size of a string in Node.js is 1GB. Considering that base64 increases the file size, does it mean that the maximum file size n8n can…

[sultan]

@Jon Thanks for your replay,
Yes it did solve the issue for the S3 Node, but when I tried to read it using the SpreadSheet File node I faced the same error:
Error: Cannot create a string longer than 0x3fffffe7 characters
And I have noticed another issue, where I have faced an issue loading the old saved credentials after restating the containers error: credentials could not be decrypted
I have used N8N_ENCRYPTION_KEY environment variable where the encryption key didn’t change.

I have another question, when using the N8N_DEFAULT_BINARY_DATA_MODE=filesystem environment variable where are the files saved, is it still in memory or as a file in the container ? If it is a file inside the container how to make sure that the file is deleted after the workflow is successfully done so the that the container don’t take a lot of space.