Hi pb,
I am sorry to say that I do not have any updates for you. I was able to reproduce your problem from one of our internal n8n instances, but the Http Request node only fails (timeouts) for https://www.britishcouncil.org.au/ but it works for all other webpages i tried. I tried all kinds of http request options but nothing worked. It’s working from any local environment I tested.
It looks like the culprit is the britishcouncil webserver configuration, since there is also this strange curl issue where we receive Access Denied. Maybe they block IP Ranges, I don’t know.
I suspect what has happened is the website is limiting requests from the digital ocean IP range, the only way to know for sure would be to get in touch with whoever runs the site and see if they can tell you or allow your droplet IP.
You might be able to use a proxy and route the requests through that and see if that helps but you may hit a similar problem. Out of interest have you tried a droplet in a different region to see if that has the same issue?
Above is exactly right - only working locally, not on digital ocean - currently resorting to sending the url to google app script and using this to dl the file as a work around but straight in n8n would be incredible!
Just done some testing and a US droplet appears to fail but the same version of n8n running on a nuc I have at home is working fine. It also fails when I run it on another VPS which is really odd.
I have then set up a UK droplet and that has also failed but on all 3 VPS options curl worked fine, To make things more interesting if I run it from my desktop from source all is good but if I enable a VPN and set it to US it works, If I change the location to somewhere else in the UK, Spain or Canada it also fails.
I am not actually sure what to check next, We don’t have anything in the node that would be blocking it and it is working some of time but by changing regions I can make it work and fail from the one machine. The other side of it is I can see the request working when n8n fails using curl so it can’t just be the region on it’s own.
My gut feel is the request is being blocked based on some kind of fingerprint using the user agent and IP but it will need a bit more testing to confirm that.
Hey all, I too have an instance where I could reproduce the problem. From playing around a little this seems to work fine when setting certain user-agent values for me (but not all). Out of curiosity, could you try this exact workflow @pb84, setting a user agent of curl/7.68.0?
Hi @pb84, this isn’t a bug but the respective webserver not liking the idea of being scraped.
You might be able to work around this by emulating the requests made by other tools (as I’ve done in the recent example). But ultimately only the admins of that server will be able to say exactly which rules they apply to incoming requests (and they can also change these without notice).
