New at scraping - handling URL redirects and metadata

Questions
1

Hi everyone, I am wondering if anyone has found a method to extract content when working with a redirecting URL (e.g. a Google News redirect to a news source). In a normal browser, the link would redirect after a second, but the standard HTTP request node doesn’t allow the redirect occur.

Also, I’ve been using sample javascript from various tutorials to remove metadata and unwanted elements from web pages, but nothing seems to work well. I’ve also tried to prompt an LLM for the code based on the input/output, but no luck either. What have people found effective?

Thanks!

2

In the HTTP node, select add option at the bottom:

Screenshot 2025-03-12 at 7.26.27 PM
Screenshot 2025-03-12 at 7.26.27 PM816×456 12.3 KB

3

Thanks, I’ve activated this option but it doesn’t seem to work.

For example this link in an HTTP request node:

Doesn’t load the source from the redirect link:

4

Not sure if it was a paste issue or what, but I would advise extracting the real link from the continue= parameter, then url decoding it, and then using that url.