Handling Large Binary Files (200–500MB) in HTTP Node

[jabbson]

Hey @Inphinity_Design hope all is good. A few things you could try or may have already tried:

• Use Filesystem Mode for Binary Data Storage

  • By default, n8n stores binary data in memory, which can cause crashes or performance issues with large files. To mitigate this, set the environment variable N8N_DEFAULT_BINARY_DATA_MODE=filesystem. This will store binary data on disk instead of in memory, significantly improving stability when working with large files. You can also specify the storage path with N8N_BINARY_DATA_STORAGE_PATH if needed. Note that filesystem mode is not supported in queue mode
    Binary data environment variables
    Binary data | Scaling.

• Process Files in Batches

  • If you need to process many files, use nodes like “Loop Over Items” or “Split In Batches” to handle files in smaller groups. This reduces memory usage and avoids overloading the system
    Convert from base64 to image/jpeg produce 0 bytes
    Reading big files

• Use External Storage (Enterprise Only)

  • For even larger scale or distributed setups, n8n Enterprise allows you to store binary data in AWS S3. This offloads storage from your local disk and is suitable for high-volume or long-term storage needs. You must configure the relevant S3 environment variables and have an Enterprise license
  • External storage

• Avoid Passing Large Files Between Nodes

  • Where possible, avoid passing large binary data between multiple nodes. Instead, consider passing URIs or file paths and letting external services handle the heavy lifting. This is especially important if you are running in queue mode, where filesystem and S3 modes are not supported
    BinaryData options on self-hosted instance

• Adjust Memory Limits

  • If you must process large files in memory, increase the Node.js memory limit using the --max-old-space-size option. However, this is only a temporary solution and not recommended for very large files or high-throughput environments
    Convert from base64 to image/jpeg produce 0 bytes.

• Split Workflows or Use Sub-Workflows

  • If you need to process multiple large files, consider splitting your workflow into sub-workflows. This ensures that each execution only handles one file at a time, releasing memory and resources after each run
    Workflow not stopping even when all the nodes are executed successfully
    How to best handle large amounts of data?

---

Summary:

• Set N8N_DEFAULT_BINARY_DATA_MODE=filesystem for disk storage.
• Process files in batches.
• Use S3 storage if on Enterprise.
• Avoid passing large files between nodes; use URIs where possible.
• Increase memory limits if absolutely necessary.