HTTP Request not working for some websites

Hi everyone. Are there any known restrictions to the HTTP node? When running below, it doesn’t execute (just times out):

@Jon, thanks in advance for offering to look at this, creating a new issue as mentioned.

Original thread: Monitoring Website content changes with n8n - #8 by pb84

1 Like

Hey @pb84,
if i try your http request to https://www.britishcouncil.org.au/ it works. What version of n8n are you using to get a timeout. Are you running the desktop app, are you using n8n cloud or do you self-host n8n?

Apologies using Version 0.191.1; self-hosted on digital ocean

Hey pb,
I just tried your http request with our docker image version 0.191.1 and the http request works fine.

To make sure it’s not a network problem from your digital ocean droplet, could you try out your http request locally using docker

docker run -it --rm -p 5678:5678 --name n8n n8nio/n8n:0.191.1

or our desktop app.

Hi Marcus

Yep it works no problem on the desktop app, so must be an issue with the self hosted version we’re using. The self hosted version allows us to run the HTTP request for every other website I’ve tried, though (e.g. https://www.google.com)

Thanks for your solution re digital ocean. I have tried that and get this error:

Hey @pb84,

I think @marcus wants you to run that command locally rather than on the Digital Ocean droplet itself. That error is saying you already have port 5678 listening locally, If you wanted to fix that I would just change the port parameter and use something like -p 5679:5678 instead.

To then access n8n it would be IP:5679 instead of IP:5678.

Thanks @Jon, feel like I might have done something wrong here:

Tried to run above but not sure what to do now. Happy to jump on a private thread on discord if easier

Hey pb,
@jon was right, I just wanted to see if your http request works outside of your digital ocean setup. Now that you have run n8n mapping host port 5679 to docker port 5678 you access your n8n using your digital ocean ip address with port 5679, e.g. 100.150.200.1.:5679

Now that we have established that https://www.britishcouncil.org.au/ is working outside of your digital ocean droplet, the problem seems to be network related and not n8n related. Why it’s not working I don’t know. If you did make lot’s of requests your IP adress could have been blocked or rate limited for some time. But that is just a guess

Hey @pb84,

That looks fine, So n8n in the container is listening on port 5678 and the host is redirecting port 5679 to port 5678 in the container.

Tried with a different network and still times out :confused:

Didn’t make any amount of unreasonable requests for my IP to blocked/rate limited either. Any thoughts for next steps?

Hey pb, sorry I tried it out several ways and it works for me. It’s especially hard to retrace since you are getting a timeout instead of an error. Do you experience this problem with any other website or is it just the britishcouncil page?

Haven’t had a problem with any other website. I don’t even think we get an error it just never finishing executing; just spins on the http node for ever

Hey @pb84,

Few questions below so we can try and work this out…

When you are connecting is it mid workflow with multiple inputs?

Are you sending any data to the site or just doing a get request on the main home page?

Is there any proxy between docker and the internet?

Does it always fail or is it only sometimes?

Thanks very much, Jon

When you are connecting is it mid workflow with multiple inputs?

Negative, literally this (the mwe) doesn’t run for me (on self host)

Are you sending any data to the site or just doing a get request on the main home page?

Exactly as above, simply a GET request which causes it to fail

Is there any proxy between docker and the internet?

Nothing that I have added or that I am aware of

Does it always fail or is it only sometimes?

It just runs forever - doesn’t fail explicitly but hangs as shown below:

I have also done some more tests:

  • Reinstalled on a new DigitalOcean droplet (following this video: https://www.youtube.com/watch?v=br91GmhMNxg&t=901s) - same problem
  • Other colleagues/friends of mine who have installed n8n as per the above video (all of them who have n8n on a digital ocean droplet w/ docker) are having the exact same issue

Hey pb84,
can you login to your droplet and try out this

curl https://www.britishcouncil.org.au/

curl https://britishcouncil.org.au/

Do you mean they can’t reach britishcouncil.org.au or do they generally have this problem?

Will try the curl request

Re colleagues who have the same error - yes exactly the same as me - with just British council so far

Very Interesting, this is what I get:

No issues with the curl command for other websites (e.g. Google)

Hey pb,
hmm that is interesting :thinking:. Could you try curl again setting the user-agent header.

curl --header "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" https://www.britishcouncil.org.au/

Could you also try out this workflow http request setting an user-agent header.

Thanks Marcus,

Unfortunately didn’t work (both in terminal and then cannot finish executing in n8n):

And also didn’t work using just one user-agent if I might have misunderstood what you meant,

Hi Marcus, any updates?