"SplitInBatches" causing workflow to hang & crash

[mmac]

I’ve created a workflow querying username info to the Twitter API. There are 2 options:

(1) Query the API 1 user at a time
(2) Query the API with up to 100 users, in a comma separated list, at a time

Option 2 is obviously the more efficient, and will help me not hit API limits as quickly.

So I’ve set up the following Workflow to query the Twitter usernames from MySQL, SplitInBatches of 100 each, aggregate the username fields, set the usernames as a comma separated list, GET from the API, remove “data” from the response so the keys can be renamed, and finally update MySQL with the data received from the Twitter API:

This works until we reach about 200-300 items returned from the 1st MySQL step, which is 2-3 batches. Then it hangs, crashes in the browser, I can’t view the execution, etc.

As an alternative, I created this, which uses option (1), above. It works flawlessly with as many as 600 items returned from the 1st MySQL step (that’s the size of my list, so I can’t test bigger numbers).

But, this approach has the disadvantage of eating up my API limits very quickly.

I have replicated the problem in both the cloud and self-hosted n8n, so I don’t believe it’s an issue with server load or something similar. I was told via email support that version 0.201.0 solved this, but it did not. Email support asked me to let them know if I was still having trouble with this issue after upgrading to 0.201.0 in the cloud, which I did, to which they replied I needed to come to the forums, so here I am.

I would very much like to be able to run this with option 1 - getting 100 users at a time. Any ideas on how to make it work, and not crash, would be appreciated.

[joeldeteves]

What happens if you move everything past the http request outside the loop?

[mmac]

I couldn’t get that to work correctly - it only passes the last batch through to “Items List” → “Rename Keys”.

In this run, there were 4 batches of 5 each, and only the last batch moves through the IF=True flow:

I need all of them to move through and get to the MySQL update, so then I tried this, but that broke as well:

I’ve also tested running this API call with 100 usernames at a time, multiple times in a row in Postman, to ensure it wasn’t an issue with the API itself, and it completes successfully there.

[joeldeteves]

If it’s not passing all items outside the loop, something is wrong with your workflow though I’m sorry to say I’m not sure what.

Take a look at this workflow I have which uses similar logic and see if it helps:

This workflow passes multiple items outside the loop (is it possible your Item Lists inside the loop are messing with the output?)

Also note if you have anything inside the loop that returns an empty item, you’ll need to set it to Always Output under the node settings and add logic outside the loop to deal with that (like I have here).

[mmac]

Woo-hoo!

I didn’t know that you could have something leave the IF → False and go to 2 places. So like yours, I now have 1 route in the first IF=False go back to the loop, and one route in the first IF=False go to a second IF, then to the Item Lists → Rename Keys and it appears to have worked:

Thank you!

[joeldeteves]

Awesome! Glad to hear that worked!