Continuing the discussion on this topic, I’d like to understand better the differences of using Redis and RabbitMQ for bigger operations.
I’ve seen users use RabbitMQ like this:
Receive webhook in n8n workflow
Save it in RabbitMQ
Use RabbitMQ Trigger in another workflow to process the queue
And I’ve also seen the documentation on queue mode recommending Redis.
I’m confused as to what is difference. Does Redis replace RabbitMQ? Do they work together? Does Redis remove the need to receive the requests in an n8n webhook node?
n8n uses redis as its queue. so if you enable queue mode it will use redis. where the main instance/webhook instances will send things to the redis queue for the worker instances to work on.
RabbitMQ I use because it has nice native nodes that allow you to set the parallel executions of the workflow. So this makes the control over your queue very nicely manageable.
Often you have batches of large data sets for example. and then other times where your server isnt doing anything. You could scale with kubernetes to deal with this. Another option is to scale to max load you need. But the better way is to just put everything in a rabbitMQ and then let the servers deal with it when they get around to it. and then you can still scale the workers if you notice the queue never gets cleared or too slow.
Technically you can use other queues to do the same and redis as well, but I just like the native rabbitMQ nodes
SQS queue length, autoscaling workers (min/max) based on this. Data offload to S3 when needed. guaranteed max 1 time execution, If processing fails (for whatever reason, the message comes available again to other workers)
Queue mode uses Redis for its internal queueing.
I Use RabbitMQ inside workflows, it has nothing to do with the internal Queue mode of n8n.
My goal is to always make sure the server is not overloaded. When just using the normal queue mode it will use the servers if it can. So for example you have a concurrency of 100 and have 1000 webhooks coming it wanting to process stuff. and then a schedule trigger triggers something else it has to wait for the 1000 webhooks to complete before it can start. and it will do that 100 at a time.
Also can help with workflow design as you can just offload to the RabbitMQ and it will process as it goes. Without you having to make sure the server can handle it.
