Issue downloading files with HTTP Request in File Name

[MotazHakim]

Describe the problem/error/question

I’m trying to download all files (Images/Videos/PDFs) from a website, and it uses API so I wanna download all the content to my Server.
So, I started with HTTP Request node to log in and get Auth Session ID to use in the next HTTP Requests in header auth, and the next HTTP Request node that list all the files I’m talking about, it’s about 9000 Files and every file has its own metadata, so after executing this node it got laggy because the output was huge so I’ve pinned it, since it’s not changeable, and here’s an example of the output of this node

Every doc has huge metadata, and all I care about is the file name and the ID of the file that will be used to get the download URL, as this screenshot

So, for that, I’ve used a Code node and used a javascript code to grab only the ID and filename, and the output of the node is :

So, the next node is HTTP Request to convert the ID to a URL to download the item, and here’s a node screenshot

The next node is HTTP Request to get this file, and i’ve used the URL from the previous HTTP request node, and in Response Format I set it to File and output field : data and executed and here’s a screenshot of the node:

also, in Json, I’ve found this Header Response that also has the file name + extension as this screenshot

and i’ve found in the server the files is being downloaded but with dummy name and without Extention as this screenshot

and if i’ve tried to open a file in Photo viewer, it works! and the content of metadata file is :

{"fileName":"","mimeType":"application/x-perl","fileSize":4396277}

So how to fix this issue!!? to use the file name I’ve already got in the Code node in the HTTP Request that downloads the files with file name + extentions

Please share your workflow

Share the output returned by the last node

Information on your n8n setup

• n8n version: 1.1.1
• Database (default: SQLite): SQLite
• n8n EXECUTIONS_PROCESS setting (default: own, main): default
• Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
• Operating system: Ubuntu 22
• Docker-Compose Content:

version: '3.3'
services:
    n8n:
      image: n8nio/n8n
      container_name: n8n
      ports:
        - '5678:5678'
      volumes:
        - /root/n8n:/home/node/.n8n
        - /root/n8n/files:/files
      environment:
        - UID=1000
        - GUID=1000
        - N8N_PERSONALIZATION_ENABLED=false
        - N8N_DIAGNOSTICS_ENABLED=false
        - N8N_DEFAULT_BINARY_DATA_MODE=filesystem 
        - N8N_LOG_LEVEL=debug
        - N8N_LOG_OUTPUT=console
        - EXECUTIONS_DATA_PRUNE=true
        - EXECUTIONS_DATA_MAX_AGE=72
        - EXECUTIONS_DATA_PRUNE_MAX_COUNT=0
      restart: unless-stopped

[MotazHakim]

an update
I’ve used a Write Binary File node to write the file into the host with the name I’ve from the Code node, but now I’ve the double files, one with filename + ext and one with a dummy name and without ext, so is there a way to remove the dummy files?, I’ve tried an execute code it’s not working!

[EmeraldHerald]

Hi @MotazHakim Glad you got this mainly sorted with your update!

Using the write Binary files with a filename is the best option here. The binaryData folder should only be used for temporary storage, and n8n will automatically clean that up, so that will sort out any “dummy” files.

[MotazHakim]

Thanks a lot for your reply
But the dummy data is not removed automatically, so I’ve done a cronjob command to delete file every 2 minutes

[Jon]

Hey @MotazHakim,

That is a bit worrying the default time to keep the files is 24 hours, did you have any files that stayed for longer than that?

[MotazHakim]

Hello @Jon
Thanks for your reply
Like I said, I’ve done a cronjob to delete all files on this folder every 2 minutes for my task because it was taking double the size of what I downloaded.

i was wondering if I could rely on these dummy files and be downloaded with a custom name and extension instead of using WRITE BINARY FILE node

[Jon]

Hey @MotazHakim,

You could try to use those files if you renamed them but using the write binary file node lets you put them in a better location.

My understanding was you set up the cron job because the files were not automatically removed which was the part that seemed worth investigating further but if it is just a space issue it doesn’t matter as much unless your cron deletes something in progress.

Maybe a better solution would be to change the deletion time by setting N8N_PERSISTED_BINARY_DATA_TTL and N8N_BINARY_DATA_TTL to 2

[MotazHakim]

i’ve set the cronjob to delete the files that last modified 2 min ago !
and renaming the dummy files is what i was looking for !