I have built a lead gen workflow with different APIs, where user can input the number of profiles to scrape per run. I am worried that the workflow will crash. Is there a way to gauge how much profiles can be scraped per run?
I’ve dealt with this too! The best way is to just start small and work your way up.
I usually start with like 20-50 profiles per run and check if anything breaks. Then slowly increase the batch size until I see errors or timeouts in the execution logs.
A few things that helped me:
-
Use the Split in Batches node to break it into smaller chunks (maybe 10-20 at a time)
-
Add some Wait nodes between batches so you don’t hammer the API too fast
-
Instead of storing everything in one variable, save results to a database as you go - otherwise memory gets crazy
For rate limits, I play it safe. Like if an API says 100 requests/min, I’ll do maybe 50-60 max with some delays between calls. Better to be slower than getting banned lol.
Which APIs are you scraping from btw? Might be able to give you a better estimate.