Pagination with generic API

Describe the problem/error/question

I’ve created a workflow that makes pagination and transform json to xml. The problem it that the node json → xml to Split in Batches. I have more than 7000 products and it’s update just first 15 items. The rest of item stay between json/xml and split in batches.

What is the error message (if any)?

Please share your workflow

Share the output returned by the last node

Nothing happens. Show me that execute nodes but not all items are executed.

Information on your n8n setup

  • n8n version: 1.1.1
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main): default
  • Running n8n via (Docker, npm, n8n cloud, desktop app): npm
  • Operating system: Mac

Hi @cesar_dev , thanks for posting your workflow.

What is your trigger here? I did not see a trigger but it seems like you are trying to trigger on that http request node, but it could be that you did not copy your trigger here and that is why I don’t see it.

Could you include a sample of what your json looks like? That can help me mock up the data.

It seems like you are making an api call, and then receiving and splitting up the results. Then you are making some if checks, then applying some code. That part makes sense to me.

You then convert your JSON to XML and then split in batches of 3. Could you explain that part of your script in more detail so I can understand what you are trying to do with that 7000 products?

It seems the reason its not working is because you are trying to output 2 items out of the false output of the if node. It will not work like that, and will only output one time. you need to put the espera 3s and the api call to bling inside the loop, not outside of it. See example below.

Here are my comments on your code:

Hello @djangelic,

Thanks for your help. Let me explain my workflow.
I’ve created a loop to get all items from Bling API and I split in items and get the information that I want to. I have to convert to XML because it’s the format that API accept to use POST method for update it. I have to split in 3 pieces because the API accept 3 items per second. The problem is when updated 3 items on “Atualiza produto no Bling” node the items stay in “XML” node and not run to the rest of workflow.
Take a look at the image and you will see there is 465 items between “xml” node and “separa em grupos” node" and only 10 items on “atualiza produto no Bling”.

Best regards,

Ok I understand, you still need to edit that loop at the end starting with separa em grupos to look like this instead for it to work correctly and loop correctly:

Hi @djangelic,

Thanks for help me with that. I believe that is something I’m doing wrong.
I edit and follow your recommendation, but still not work.
I will send a link with a video with the problem. You can see that when it’s split in batches and goes to done, nothing happens. This API accept just 3 items per second. If it’s more than that, it will be refused.

There is another video that I join split In batches node “done” with the node “Espera 3s”. At this time, you will see that try to update the items on Atualiza produto no Bling node but the split in batches node from “done” it isn’t 3 items. There is sometimes that has 8 items and them not work properly.

I’m learning about n8n.

This is the workflow that I’ve done.

Thank you so much if can help me with that.

Best regards,

Whoops, my apologies I forgot a key step. See updated workflow below. The last loop step needs to connect back to split in batches node so it will move on to the next batch:

Hello @djangelic,

It’s almost there. How I have pagination, when finished first loop on split in batches, it goes to next page until has items. When got items, the workflow goes to until split in batches and goes to “done” and set node that inform Loops ended. I send to you an image that shows many items between “XML node” and “Split in batches node”, but not goes to “loop” and “Wait node”.
Screenshot 2023-08-09 at 15.21.57

Best regards,

How are you splitting the data on the split in batches? Is it still set to 3? I recommend add a success set node after the http request to clear out any data the http request node generates, so at the end of the loop, you have a list of success messages from your api call. This should help you better troubleshoot. Something like this:

1 Like

Hello @djangelic,

Unfortunately it didn’t work.
When pagination does not meet the conditions, it repeats until it does. But the Split in Batches node has already executed once and therefore it doesn’t repeat. She goes the “Done” path and does nothing. Items arrive at the node and just sit there without executing them.

Thanks a lot for help.

Best regards,

I wonder if you’re running out of memory on your computer/browser and it’s not finishing. I see that sometimes when my workflows handle too much data at once.

My work around is to build sub workflows, to minimize the amount of data that the UI needs to process (sub playbooks are run silently in the background and serve the data back).

Are you looping on the same data over and over? You may want to do an if check in the loop to do the wait if that is the case, so it stays in the same loop till the function is complete then move back to split in batches to move to the next 3 items.

1 Like

Hello,

I will try.

Best regards,