Tuning n8n for long running tasks

Questions
1

Describe the question

We’re running AI automation workflows to generate images and movies. Those usually takes between 5-15 minutes to finish and involves multiple workflows ranging from very specific, like handling the actual call to a remote AI provider, to expose a webhook and how to respond back to our services when done. Our k8s cluster is running with many smaller instances (cpu:2,mem:8) so I thought I would reconfigure our n8n from one instance (cpu:0.5,mem:2) to use workers instead and scale dynamically based on the current traffic. But we seem to run into issues with validity window and n8n not being able to respond in time. We also have one remote API provider which returns generated images as inline base64 instead of an url. This was mitigated by mem:2 and max_old_space but I still see that the memory consumption only increases, never decreases, hinting me of a memory leak. Our currency is set to 20 (tried higher but that only triggered mem issues faster), and lower caused “Offer expired” faster. We’re not delivering our results in real time, so a few workers should be able to handle the load at their own pace. And I will try to scale our workers depending on queue length when we can get 1 worker stable.

So my questions are, what is the recommended way to tune n8n to handle long running tasks? Can I change or remove the validity window of tasks? And how come the n8n instance become unresponsive when it only handles dispatch and at most one user having the UI open?

What is the error message (if any)?

  • Task (1UEsQ7iT) rejected by Runner with reason “Offer expired - not accepted within validity window”
  • Our LB in front of n8n returning 504 when trying to proxy to n8n (no clue why it doesn’t respond in time

Please share your workflow

Too sensitive to share, but we’re using a hierarchy of workflows, with up to a depth of 5 (atm) workflows.

  • webhook workflow (handles webhook, dispatch to other workflow, and callback when done)
    • workflow A (if we want more than 1 image back we configure this here to loop)
      • workflow B (some other logic specific to our use case, like QA)
        • workflow C (and some more logic specific to our use case
          • workflow handling calling of remote API, and returns a generated image/movie

Information on your n8n setup

  • n8n version: 1.97.1
  • Database (default: SQLite): postgres
  • n8n EXECUTIONS_PROCESS setting (default: own, main): default
  • Running n8n via (Docker, npm, n8n cloud, desktop app): docker, k8s, gcp
  • Operating system: linux
2

Seeing something similar. I have very long running task. Scaping thousands of sites in a loop, can take 6+ hours. At some point I end up seeing:

Task (jXHrjlUP) rejected by Runner with reason “Offer expired - not accepted within validity window”

What’s the suggestion to avoid running into these things?