I am building a workflow that has a webhook trigger. Each webhook receives a single item (an account record). I then need that item to process all the way through to the end of my workflow, because I have some nodes that are inserting calculated values into unique columns in MySQL. Things are working fine when a single webhook is received. But I also need to plan for when multiple webhooks are received at the same time, like when 100 records are imported into the webhook source and all of them are received by n8n at nearly the same time. In testing, these multiple webhooks tend to succeed 5% of the time because they are trying to reuse the same value for the unique columns. Ideally, if 100 webhooks were received at once, they would be processed in order, one at a time, through the entire workflow. As it is, they seem to be executing all at once and causing failures.
Iâve tried splitting into batches, setting the webhook Respond property to âWhen Last Node Finishesâ and playing with the ordering of my MySQL nodes, all to the same result.
You could save your webhook data to a MySQL table and with a second workflow incrementally process the data.
The second workflow would just poll the database and whenever there is new data it loads all of them and processes them.
A nicer solution would be, to use transactions with the MySQL nodes. But afaik n8n does not support transactions at all right now.
Some feature requests about transaction support for database nodes:
@MutedJam, perhaps those three feature requests should be merged?
Thanks for the suggestion @FelixL - we initially started to go down the path of transactions, ran into Unknown Errors and gave up. Good to confirm why it didnât work. I will mull over the 2nd database idea (we already use MySQL so would need a âtransactionsâ table). It sound like a simpler version of queue mode. I would like to handle this in one instance and one workflow if possible. If we come up with an alternate idea I will post it here.
did not think about queue mode, but afaik itâs used with multiple workers to spread the load over multiple servers, which would not solve your problem.
Maybe there is a setting in n8n which wonât allow n8n to execute multiple workflows at once?
Maybe someone from the n8n Team has some insight about that?
I am not aware of any throttling options for incoming webhooks on our roadmap right now unfortunately. But perhaps you might want to try something like Convoy @hndmn (Reddit - Dive into anything)? Their website suggests this might be the right tool for the requirement:
Avoid blasting too much events to a single endpoint at once with flexible rate limiting controls. Rate limits can be configured at both high level and per endpoint.
I have not used this myself though, so donât have any first hand experience to share here.
This is a great idea. We decided on a temporary workaround for our immediate need, but I will file Convoy away for future use. I can certainly see use cases for controlling the flow of webhooks.
I have a follow up question for you @MutedJam. After finding a way past the first breakdown, we have run into another one. We are stress-testing our workflow by sending n8n 3,000 webhooks from Zoho CRM. We are watching the All Executions page and the first 430 records were processed without any errors. We still donât have any errors, but processing did stop about 10 minutes into the batch and has stopped processing data. There are no failed executions.
We are currently running on a starter account on n8n.cloud and are aware of the 5,000 monthly execution limit. According to the dashboard though we arenât yet close to the limit.
What are we able to do on a cloud instance to troubleshoot? We can see on the webhook source (Zoho CRM) that 430 webhooks were sent. So the issue may be with that system. But it led us to consider how could we investigate in n8n (cloud) in the future? If we self-host we know weâll have server logs. If we opt for cloud, what is the equivalent?
Hi @hndmn, I am afraid there is no equivalent to the server-side logs youâd get on a self-hosted instance in n8n cloud. Log streaming is available on enterprise plans, but these are more audit logs rather than traditional server logs.
While I hope this will change at some point, at the moment the only way to get this level of insight would be to host n8n yourself rather than using n8n cloud.
Without knowing how exactly you are batching your data itâs also hard to make an educated guess as to whatâs happening here. From your screenshot it seems a lot of webhooks are being sent around the same time 19 Apr 8:41.
Which response did you get for the webhooks for which you couldnât see any execution in n8n? Could there have been any error response (5xx status) suggesting the n8n instance was simply receiving too much traffic around the same time?
Nevermind, just re-read your message. Yep, it seems like only a limited number of requests were sent, so the problem does indeed seem to be with the source system rather than n8n here.
Also thanks for clarifying on the current logs status. It helps us in decision making.