As some of you may know i’m keeping tabs on ransomware victim in the dark web and scraping the blogs of the ransomware groups. They keep changing the anti-scraping protection and I’m looking to bypass it to keep scraping the site.
Is there I way I can bypass this.
At this point in the flow, I need it to then click the button that shows on the webpage and continue on the flow,
any ideas?
I’m thinking maybe to just add the cookie that its trying to set and see if that will do it
yeah needed to enable Full response
and add the Cookie in, works now.
spoke too soon
passing the cookie to HTTP nodes further down the line only works for the first result when iterating
How are you referencing the value for the Cookie?
Since you’re passing the values without expression, all the requests will use this value. Is it possible, that the URL expects different cookie values?
Doesnt seem like it, on first glance. Why would the first HTTP node work and then the first iteration on the next http node with the cookie value, but not any iteration after that?
I also tried passing the entire header from the first request too, didnt make any difference.
i think the site isnt the most stable it is working ok again at the moment. strange
