Uncontrollable RabbitMQ

Questions
#1

Hi.
I finally got around to writing this question.
I set up a little workflow:

WorkFlow 
{
  "nodes": [
    {
      "parameters": {
        "queue": "test-rabbit",
        "options": {
          "jsonParseBody": true,
          "onlyContent": true
        }
      },
      "name": "RabbitMQ",
      "type": "n8n-nodes-base.rabbitmqTrigger",
      "typeVersion": 1,
      "position": [
        250,
        480
      ],
      "credentials": {
        "rabbitmq": "RabbitMQ n8n"
      }
    },
    {
      "parameters": {
        "functionCode": "var time = new Date().getTime() + (5 * 1000);\n\nwhile (new Date().getTime() <= time) {}\n\nreturn 'Done!';\n\n"
      },
      "name": "FunctionItem",
      "type": "n8n-nodes-base.functionItem",
      "typeVersion": 1,
      "position": [
        470,
        480
      ]
    }
  ],
  "connections": {
    "RabbitMQ": {
      "main": [
        [
          {
            "node": "FunctionItem",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

I added a simple time loop to simulate a 5-second run and left it off.
Put about 30 messages in the queue.
I started the flow, and what do I see?
I see 30 floo running, even though there’s only one listener in the rebbit.

image
image2020×1150 166 KB

How?
No new messages should be accepted until the current ones are handled in success or error.
This leads to a large uncontrollable load on the server and high LA.

Home made message broker
#2

n8n starts a new workflow execution for each message it receives. It would currently also technically not be impossible in n8n to wait until the workflow execution finishes and only then start the next one. What would be possible is to implement some kind of a rate limiting that for example just all X seconds a new execution gets started.

If you need that kind of behavior please create a new topic under “Feature Request”.

#3

Rebbit, on the other hand, works by the push mechanism.
While the listener is busy, Rebbit can not send him a new message.
For this reason, it seems to me not right to make a limit to start the listener by time.

#4

WebHook can wait for the last node to execute, why can’t RebbitTrigger do that?

#5

It has technical reasons that are sadly too complex and time-intensive to explain in this context right now.

Not saying it would not be possible to do, just saying that it would need core changes that would need more time and are so very unlikely to happen now, as we simply have many more high priority things to work on. We have just limited resources and we have to make sure to work on the things that matter most.

#6

I know what you mean.
I hope you will someday have time to get this task out of the backlog.
I think that in the queue nodes documentation, you should warn users that with a large flow of messages, the load will be uncontrollable.

#7

Hi Jan,

Can you specify more detail about how to implement rate limiting?
I could not find any delay node within n8n.

Fan

#8

Welcome to the community @fanshaohua-fan !

There is currently none (is wip), but that is also not even needed here. You could in the simplest case just read one message at a time and after each message you wait for a certain amount of time before emitting it.

#9

@jan , Thanks for the quick response.

If I understand you correctly, I should modify the RabbitMQTrigger.node.ts file by adding some wait logic around emitting(line: 138).
similar as the comment you made on 7051

#10

Yes, that is correct. That sound then make sure that n8n does not get totally overloaded. You can then set the values depending on the size of your instance and how long those executions run.