Trying to webscrape using n8n

[Manas_Rathi]

Describe the problem/error/question

Tried to webscrape using n8n and the http requests node. The webpage I wanted to scrape loads data dynamically using Javascript, and thus the data is need is not in the html code. I need to tell the http request to wait for a few seconds. I tried using custom JS and Python code but both environments do not have the libraries I need. What do i do

What is the error message (if any)?

Please share your workflow

{
“meta”: {
“instanceId”: “3ae984276f26fdccef57a1b9bc1694b82a13e4f6272e3c1d101b9cb4444cc809”
},
“nodes”: [
{
“parameters”: {},
“id”: “ae38dc3d-85b4-4b25-af69-46e6a280ea3b”,
“name”: “When clicking "Execute Workflow"”,
“type”: “n8n-nodes-base.manualTrigger”,
“typeVersion”: 1,
“position”: [
760,
380
]
},
{
“parameters”: {
“options”: {}
},
“id”: “c5afbe14-92fc-4f1e-928e-57426ff19dd7”,
“name”: “HTTP Request”,
“type”: “n8n-nodes-base.httpRequest”,
“typeVersion”: 4.1,
“position”: [
980,
380
]
}
],
“connections”: {
“When clicking "Execute Workflow"”: {
“main”: [
[
{
“node”: “HTTP Request”,
“type”: “main”,
“index”: 0
}
]
]
}
}
}

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

• n8n version:
• Database (default: SQLite):
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app):
• Operating system:

[EmeraldHerald]

Hi @Manas_Rathi

In this case it’s most recommended to use something like https://apify.com/ and connect to their service’s API using the HTTP Request node. That way, you can transform or move this data however you need If you’re self-hosting, you could also look into something like Puppeteer.