Download large file and upload to OneDrive

Hi all,

I am looking for the best way to download a large file (a tar file of a directory on my website host - not in public_html) and move it to a OneDrive folder.

I SSH into the host, create the tar but not sure what the best workflow is to get the file and upload it.

If I download the file, where is it stored?
I could move the file to the FTP directory on. my host then download it via FTP but not sure how to get it up to onedrive (as there is a size limit and my file might be > 1GB).
If I download it and can access it I could sync it to OneDrive using rsync or similar outside of n8n.

My n8n install is on an old linux laptop using docker.

Any thoughts or ideas welcome.

Thanks.

Hey @PaulBR1,

It sounds like it is all possible which is handy, What you could do is use the SSH node to access the server and create the archive then use the FTP node and set it to SFTP and use that to download it.

Where it gets tricky is I don’t think n8n writes the file to disk until you tell it to so it may sit in memory (@MutedJam may need to confirm this), so it would be a case then of adding in the Onedrive node and telling it to use the data property and you should be sorted.

If the file is larger than your limit then it may be a case of splitting the archive into chunks on the first step and uploading the chunks for usage later.

Yes, as @jon said n8n would keep data in memory, so working with large files or huge datasets can become a problem really fast, especially when using multiple nodes or manually running workflows. So you’d need to monitor your memory usage very closely to avoid running into out of memory errors.

Personally, I’d only use n8n to execute commands for uploading such large files onto a cloud storage rather than actually fetching them in n8n. So my suggested approach would be to set up something like rclone on the host (which would handle compression and chunking for you when needed), then configure the OneDrive connection in there.

That way, you’d still be able to use n8n to control the flow and handle errors through the SSH node, but wouldn’t have to worry about the memory consumption.

On the n8n front I know @kik00 was looking into improving handling of large files (but I’m currently keeping him busy with cloud-related requests). So the above suggestion might change in the future :slight_smile:

1 Like

Thanks @jon and @MutedJam, that makes sense.

I did setup the sftp and downloaded a small test file which it stored as a binary file so couldn’t get to it.

I have used rclone before so will add that to my linux laptop where n8n is installed (can’t add it to the shared website host) and play with controlling it via its API to initiate the download from my host and upload to OneDrive as suggested.

I did setup the sftp and downloaded a small test file which it stored as a binary file so couldn’t get to it.

This sounds somewhat unexpected to me though tbh. While I think large files will cause trouble, the process of downloading a file from FTP, then uploading it to OneDrive should still work like so:

Example Workflow
{
  "nodes": [
    {
      "parameters": {
        "fileName": "={{$binary.data.fileName}}",
        "parentId": "CA5C20CA5A0862D9!120",
        "binaryData": true
      },
      "name": "Microsoft OneDrive",
      "type": "n8n-nodes-base.microsoftOneDrive",
      "typeVersion": 1,
      "position": [
        680,
        300
      ],
      "credentials": {
        "microsoftOneDriveOAuth2Api": {
          "id": "18",
          "name": "Microsoft Drive account"
        }
      }
    },
    {
      "parameters": {},
      "name": "Start",
      "type": "n8n-nodes-base.start",
      "typeVersion": 1,
      "position": [
        240,
        300
      ]
    },
    {
      "parameters": {
        "path": "/pub/example/pop3-browser.png"
      },
      "name": "FTP",
      "type": "n8n-nodes-base.ftp",
      "typeVersion": 1,
      "position": [
        460,
        300
      ],
      "credentials": {
        "ftp": {
          "id": "78",
          "name": "test.rebex.net"
        }
      }
    },
    {
      "parameters": {
        "fileName": "={{$binary.data.fileName}}",
        "parentId": "CA5C20CA5A0862D9!120",
        "binaryData": true
      },
      "name": "Microsoft OneDrive",
      "type": "n8n-nodes-base.microsoftOneDrive",
      "typeVersion": 1,
      "position": [
        680,
        300
      ],
      "credentials": {
        "microsoftOneDriveOAuth2Api": {
          "id": "18",
          "name": "Microsoft Drive account"
        }
      }
    }
  ],
  "connections": {
    "Start": {
      "main": [
        [
          {
            "node": "FTP",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "FTP": {
      "main": [
        [
          {
            "node": "Microsoft OneDrive",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

Did you enable the binary data switch on the OneDrive node?

I didn’t try the onedrive upload because the file will be very large.

Decided to jump straight to rclone test :slight_smile:

1 Like