HTTP request - cookie support?

Help me Build my Workflow
1

Describe the problem/error/question

I am working on a scraper to populate a vector db. the http request node is setup to pull the sitemap. But there is one site behind cloudflare and it appears to be blocked because it’s trying to set and retrieve a cookie. Must be a bot thing. Although it’s not prohibited by their robots file. Is there a way to support cookies in this node?

the page I’m trying to GET is https://seaworld.com/san-diego/sitemap.xml

thank you

What is the error message (if any)?

Forbidden - perhaps check your credentials?
Attention Required!  Please enable cookies. Sorry, you have been blocked You are unable to access seaworld.com Why have I been blocked?
This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.

What can I do to resolve this?
You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.

Cloudflare Ray ID: 91090436bfc82aa3 •

Please share your workflow

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

Cloudflare protection is hard to bypass - which is its purpose. I believe there are services that can do it, but it’s not as easy as setting a cookie.

1 Like