Hi- I am trying to set up an automation that will check an affiliate website and check the links to see if they match the description on the website and work, however, after getting all the links, filtering/extracting only the external links (which works up to this point), I can’t avoid bad gatway/forbidden errors on different website.
I’ve tried splitting in batches, adding waits and also putting in headers so it looks like it’s coming from a browser but no luck.
Any idea on how to solve this?
Thank you in advance