Runtime problem with the HTTP request node

Questions
1

Hello,

In my workflow I use this JSON file very often, which is updated every hour: https://pbs.twimg.com/hashflag/config-YYYY-MM-DD-HH.json

To do this, I make an HTTP Get request every hour where of course I set the date with variables. From time to time I get a 404 error with some URLs and I can’t figure out why since I can access the page manually with my browser, and when I debug this node on a completely different n8n environment than the one I’m using (in this case my company one), the node runs correctly.

Here are several screenshots of the result I get in n8n. I used this exact address: https://pbs.twimg.com/hashflag/config-2022-01-10-16.json

What can I do to avoid these errors?

Capture d’écran 2022-01-10 à 18.30.55
Capture d’écran 2022-01-10 à 18.30.551700×817 333 KB

Capture d’écran 2022-01-10 à 18.31.02
Capture d’écran 2022-01-10 à 18.31.021700×817 178 KB

Capture d’écran 2022-01-10 à 18.31.12
Capture d’écran 2022-01-10 à 18.31.121700×817 205 KB

2

Hey @arthurcorre, n8n would simply return whatever the webserver behind https://pbs.twimg.com sends it. For the 404s I could imagine the file you’re looking for might not have been generated yet. For the garbled data, I wonder if that’s how the server serves incomplete files (e.g. files which are currently being uploaded or otherwise processed).

Without in-depth knowledge about that server I don’t think anyone would be able to tell with certainty, especially since the URL now serves what appears to be valid JSON.

So my suggestions for the next steps would be:

  • Verify if this does indeed only happen for very fresh data by delaying your requests for say 1 hour. In other words, deduct 1 hour from the timestamp of the file you are fetching. This can for example be done with the Date & Time node.
  • To rule out intermittent problems, enable the retry option on the Settings tab of your HTTP Request node:
    image