API Response: My API returns a meta object with current_page, last_page, and total (e.g., 41 items across 5 pages).
Data Structure: The actual items are in an array called data.tours.
Goal: I need to loop through all pages (1 to 5), extract each tour, and store it in Pinecone
My Questions:
What is the recommended way to build the loop? Should I use a Wait node with a recursive HTTP request, or the Loop Over Items node?
Since I’m sending data to Pinecone, should I use the Split Out node immediately after the HTTP request to process each tour individually, or is there a better way to batch them?
How do I pass the current_page increment back to the HTTP Request node without losing the context of the last_page limit?
you don’t need a wait or loop node at all. the http request node has built-in pagination now. it handles the whole thing internally.
definitely drop a split out node right after the http request. you want to flatten that data.tours array into individual items before they hit pinecone.
you don’t pass it back manually. just use {{ $pageCount + 1 }} in the pagination settings. n8n tracks the counter under the hood. for the stop condition, just check if {{ $response.body.meta.current_page >= $response.body.meta.last_page }}. it reads the limit straight from each response so it knows when to quit.
In the HTTP Request node options there’s a Pagination setting, use “Update a Parameter in Each Request” mode and set your page query param to {{ $pageCount + 1 }}, then for the completion expression pick “Other” and use {{ $response.body.meta.current_page >= $response.body.meta.last_page }}. After that just use a Split Out node on data.tours before sending to Pinecone.
Adding to what @A_A4 said — the HTTP Request node’s built-in pagination is definitely the right call here. For your specific case with total, limit, and page-based URLs, set the pagination mode to “Update a Parameter” and point it at your page offset or page number field.
For the Pinecone batching question: definitely add a Split Out node right after the HTTP Request to explode the tours array into individual items. Then you have a choice:
If your Pinecone index accepts batch upserts: use an Aggregate node to collect all items back into a single array, then one HTTP Request node to the Pinecone /vectors/upsert endpoint with the whole batch. This is faster.
If you’re embedding each tour before storing: pipe each Split Out item through an Embeddings node first, then the Pinecone node processes them one by one (which is fine — n8n’s internal queue handles the throughput).
The key thing to avoid: don’t try to manually track the page number with Static Data or expressions — the built-in pagination handles it cleanly and you don’t lose context between iterations. Just configure the “Complete When” condition to check when the returned tours array length is less than your limit value (that signals you’ve hit the last page).
What does your API’s pagination look like — offset-based or cursor-based? That affects which pagination mode to pick.
Hey @yelinaung , just solved the exact same thing today!!!
Use “Update a Parameter in Each Request”, set the page param to Type: Query, Name: page and value {{ $pageCount }} and Complete Expression to something like this, just adapt it to your API {{ $response.body.data.tours.length === 0 }}.
Avoid relying on next page URLs from the API response, I got burned by that today where the URL kept returning page=1 forever. Letting n8n control the counter with $pageCount is much cleaner.
Split Out on data.tours right after and you’re good to go.
