I know there’s been a bunch of threads on pagination, and with all of them I’ve tried to implement the fixes but couldn’t seem to find anything that worked.
Basically I’ve got a scraping tool that paginates with, with the code node checking whether its the final page or not, and using an if node to loop back around. The problem is that the data then comes out in separates runs.
I needed the data in a single run to then go through a second loop which will further extract data for individual leads, this can’t be done since after the first run the ‘done’ stays open, and all future data flows straight through.
The fix?
Use a code node to aggregate all the data coming out (4 x all the data for example), then mark the first run with a secret code, and filter out anything not with this code, I can then pass this run onto the actual loop.
Funny solution, but has anyone else found a better solution, want to know for future reference.
I should add that for some reason there seems to be a bug that the final results shows 152 items for example, but in reality there is only 38. Strange.
Example Workflow
Information on your n8n setup
- n8n version: 1.93
- Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
- Operating system: Mac