Automated Web Ingestion Pipeline in n8n for LLM Knowledge Bases & RAG (75% Token Reduction + WAF Bypass)

Built with n8n
, ,
1

Hi everyone! :waving_hand:

I put together a practical architecture pattern for automating web scraping and knowledge base ingestion in n8n without getting blocked by Cloudflare/anti-bot WAFs and without polluting the LLM context with heavy HTML.

### Architecture Overview

```

[Schedule Trigger] ➔ [Read Target URLs] ➔ [HTTP Node: MESSORA API] ➔ [Embeddings] ➔ [Vector DB / Pinecone]

```

### The Problem it solves:

1. **WAF/Anti-bot blocks**: Standard HTTP nodes or headless browser nodes fail on sites protected by Cloudflare or DataDome.

2. **Token waste**: Passing raw HTML through LLMs consumes huge context windows. Clean Markdown reduces token load by ~75%.

### How to configure the HTTP Request Node in n8n:

- **Method**: `POST`

- **URL**: `https://api.messora.dev/v1/extract\`

- **Auth**: Generic Header Auth (`Authorization: Bearer YOUR_MESSORA_API_KEY`)

- **Body JSON**:

```json

{

“url”: “={{ $json.url }}”,

“only_main_content”: true

}

```

The response returns clean Markdown under data.markdown ready to feed directly into text splitters and embeddings.

I wrote a detailed step-by-step tutorial with sample JSON workflows here:

:backhand_index_pointing_right: Full Guide: Building an automated web ingestion pipeline in n8n for LLM knowledge bases - DEV Community

Hope this helps anyone building RAG and AI workflows in n8n! Feedback and questions are welcome.