403 error when using the http Get Request

[Judge.F.Jumbo]

I just want to read the home page info and send it to openai to determine the topic or niche of the website from the home page - I have look and can not seen to find any answers - should i pass a User_Agent in the header - if some how do i do that? Thanks Jim

[barn4k]

hello @Judge.F.Jumbo

Use https. HTTP nowadays is relatively rarely used.

[Franz]

To pass a User Agent simply add an Header Parameters.

BUT I can retrieve the URL with N8N even without a User Agent.
Have you already made multiple requests to the same domain? If so, you might be getting blocked. In such cases, it helps to add a Wait Node after each HTTP Node. To prevent the workflow from being interrupted, you can set the On Error option to Continue (using Error Output) in the HTTP Node under Settings. The branch there should then pass along a message like URL could not be loaded or something similar.

[Judge.F.Jumbo]

thanks - same error with https

Thanks - yes I can do the same if i put the url in - however once i try use json (see the image I attached)to have the url as a variable (i have a number of url i want to check) it is then that it fails - so something is happening different between the two methods- thanks for your help

[barn4k]

What’s inside the sixth URL item? The error is showed for it, but you are showing the result of the first item.

Open the table view and check the url.