Hello,
I am using an HTTP Request to call a POST request from the Crawl4AI API, which I have in a separate container. The request is working fine, but it seems to me that the “crawler_params” that I am sending in JSON in the body are not working properly. I am using parameters such as “only text”: True and “verbose”: True. But I can’t see the debug data, and the output doesn’t display only text.
I suspect that the syntax of the parameters I am sending may be incorrect.
{
“nodes”: [
{
“parameters”: {
“method”: “POST”,
“url”: “http://host.docker.internal:11235/crawl”,
“authentication”: “genericCredentialType”,
“genericAuthType”: “httpHeaderAuth”,
“sendBody”: true,
“specifyBody”: “json”,
“jsonBody”: “={{({\n "urls": $json.loc,\n "crawler_params": {\n "simulate_user": true,\n "magic": true,\n "override_navigator": true,\n "user_agent": "Mozilla/5.0 …",\n "verbose": true\n },\n "extra": {\n "word_count_threshold": 10,\n "only_text": True\n }\n})}}”,
“options”: {}
},
“type”: “n8n-nodes-base.httpRequest”,
“typeVersion”: 4.2,
“position”: [
1000,
20
],
“id”: “8d2dd4d2-532a-47ee-9d15-0a8ce1bef0f7”,
“name”: “HTTP Request1”,
“credentials”: {
“httpHeaderAuth”: {
“id”: “HIDDEN”,
“name”: “Authorization”
},
“httpCustomAuth”: {
“id”: “HIDDEN”,
“name”: “Custom Auth account”
}
}
}
],
“connections”: {
“HTTP Request1”: {
“main”: [
]
}
},
“pinData”: {},
“meta”: {
“templateCredsSetupCompleted”: true,
“instanceId”: “ba562a290ad69bfaafa1ccca754949fc8f2d803bfcdf727b648b4007bed0af04”
}
}
I do not share the result, because the output of the scrapper is a very large amount of data ( 3.8 MB )
I ask because I would like to use the parameters properly. Right now, my scraper is downloading the entire page, ultimately for the current project I would like to limit this to the first 2000 or so words as this is sufficient for me.
- n8n version: 1.78.0
- Database (default: SQLite):
- n8n EXECUTIONS_PROCESS setting (default: own, main):
- Running n8n via Docker
- Operating system: Windows