I need help building a workflow with 22k products

Help me Build my Workflow
1

I need help building a workflow to process 22k products coming from an API. What’s the best method to do this? I’m using version 2.1.4 and the installation is on my server using Docker Desktop, Portainer, PostgreSQL and Redis.

1 Like
2

For 22k products, the best approach is batch processing with pagination. Don’t pull everything at once fetch in smaller batches (e.g. 100–500), process incrementally, and use Redis/DB to track progress. Splitting ingestion and processing into separate workflows helps avoid memory and timeout issues on v2.1.4 Docker setups.

1 Like
3

Hi,

I already have a tested instruction and a working approach for handling large product volumes (20k+ items) coming from APIs.

In short, the architecture is based on:

  • batch processing with pagination

  • state tracking via Redis and/or a database

  • separated workflows (ingestion → processing → persistence)

  • retries, API rate limiting, and idempotency

  • avoiding loading large datasets into memory at once

I can:

  • adapt the existing instruction to your specific API

  • help you build a stable workflow in n8n 2.1.x

  • advise when to use Redis vs PostgreSQL

  • help avoid common issues like timeouts, memory spikes, and duplicated executions

If this sounds useful, feel free to share more details about the API and update frequency.

4

Okay, I followed your advice and used pagination of 100 per product, totaling 221 pages, and I put them in a loop where I process them and upload them list by list via HTTP using PUT. The only “problem” I’m having now is the execution time, which is taking around 40 minutes. Do you have any idea how I can reduce this?

1 Like
5

Great progress! Pagination at 100 per page is the right approach. The execution time is likely long because each page is processed and uploaded sequentially. To reduce it, you can process multiple pages in parallel instead of one by one, separate data processing from uploading by using a second workflow or sub-workflow, and batch multiple pages into a single PUT request if the API allows. Also, make sure you’re only sending necessary fields and reusing connections where possible. With parallel execution and batching, you can usually bring a 40-minute run down to just a few minutes.