Hello,
I’m currently working on transferring all orders from Order Desk to BigQuery using a workflow created in n8n. While I’m not a specialist in data pipelines, I would appreciate your guidance on the best approach to achieve this efficiently.After reviewing the Order Desk API, I noticed that each request can retrieve up to 500 orders. Initially, I attempted to implement pagination, but I struggled to find a schema that fits well with n8n’s structure.
Since our daily order volume is below 500, I created a flow that works as follows:
- It initiates a request for orders starting from a specified date and sends the data to BigQuery.
- The start date is then incremented by one day, and the process repeats until the end date is reached.
- The flow stops once all dates in the range are processed.
Although this approach works to some extent, it is not standard. As a result, the data in BigQuery often does not match the data in Order Desk, with discrepancies that vary in size.
We need a reliable method to ensure all orders are accurately transferred to BigQuery while maintaining daily updates.
A guidance or direction to appropriate technical support would be greatly appreciated, as the current inconsistencies are impacting our reporting.
Thank you in advance for your help!
It looks like your topic is missing some important information. Could you provide the following if applicable.
- n8n version:
- Database (default: SQLite):
- n8n EXECUTIONS_PROCESS setting (default: own, main):
- Running n8n via (Docker, npm, n8n cloud, desktop app):
- Operating system:
Thanks for posting here and welcome to the community! 
Can you share more about what these discrepancies are in order to address them?
Which limitations did you encounter with the pagination?
Can you share your workflow to see your current configurations?
Tip for sharing your workflow in the forum
Pasting your n8n workflow
Ensure to copy your n8n workflow and paste it in the code block, that is in between the pairs of triple backticks, which also could be achieved by clicking </> (preformatted text) in the editor and pasting in your workflow.
```
<your workflow>
```
Make sure that you’ve removed any sensitive information from your workflow and include dummy data or pinned data as much as you can!
As an idea, you might be benefiting from our Dedupe node in case the discrepancies you described involve duplicates.