Scraper Execution time limit

Help me Build my Workflow
1

I am working on Website scraper workflow where i will scrape webpages, personalize email and send emails.
I am fetching 500 records from apify/appollo. Then i am running Loop of these 500 items one by one and performing scraping, sending email etc. But this process is not running for all 500 items. It stops after 30-40 loops.

I am using starter plan of n8n. Tell me what could be the issue? I need to scrape all the urls present in the apify 500 items.

Thanks

2

Hi,
On n8n Cloud Starter every workflow execution is automatically cancelled once it has been running for 5 minutes.
If each loop (scrape → personalise email → send) takes ~8–10 s, the 5-minute wall will be hit after roughly 30–40 cycles, which is exactly what you’re seeing. n8n simply marks the execution as cancelled (timeout), so no explicit error shows up. A second but related bottleneck is the small container behind the Starter plan (≈ 320 MB RAM) which can also kill very large single-executions.

Option 1: Parent/Child Workflow Pattern

Split your workflow into two parts:

  • Parent: Gets 500 items → Split into batches of 10-15 → Execute child workflow for each batch
  • Child: Process one batch (scrape + personalize + send emails) in under 60 seconds

This keeps each execution well under the 5-minute limit while processing all your data.

Option 2: Self-hosted n8n Instance

Move to your own server (Google Cloud, AWS, etc.) where you control:

  • Unlimited execution time
  • Memory allocation
  • No timeout restrictions

I personally use Google Cloud VM for heavy workflows like this - costs ~$10-15/month but gives you complete control.

What about upgrading the plan?

Pro plan increases timeout to 40 minutes and doubles RAM, but costs $50/month. If you’re processing 500+ items regularly, self-hosting is more cost-effective and gives you unlimited resources.

Quick recommendation: Start with Parent/Child pattern (works immediately), then consider self-hosting if you need more scale or have multiple heavy workflows.

Both solutions will get you processing all 500 items reliably instead of dying at 30-40.

1 Like