Pagination with Body parameter

Questions
,
1

Hello n8n lovers!

I’ve used the HTTP node pagination feature already several times. Now I’m dealing with a json-rpc API that to fetch data I need to do a POST request. In the body of the request I must pass the limit (fix value) and the offset.

I can’t manage to update the offset in each call, the body of the first call would be something like:

{
  "method": "call",
  "params": {
    "service": "object",
    "method": "execute_kw",
    "args": [
      "X",
      "Y",
      "z",
      "res.partner",
      "search_read",
      [[]],
      {
        "fields": [
          "id"
        ],
        "limit": 500,
        "offset": 0
      }
    ]
  }
}

How should I set up the http call to update the offset field of the body?

  • n8n version: 1.21.0
  • Database (default: SQLite): postgresql 14.8
  • n8n EXECUTIONS_PROCESS setting (default: own, main): queue
  • Running n8n via (Docker, npm, n8n cloud, desktop app): kubernetes
  • Operating system: amazon linuex
2

Any ideas??? :wink:

3

Do you need help keeping track of the offset or knowing how to structure the http request?

If it’s keeping track of the offset you can likely use some of the built in HTTP variables they give you, like $pageCount which gives you the the pagination count.
Then you could do something like {{ $pageCount * 500 }} in the offset.

Let me know if that makes sense

4

Hello Liam,

What I want to know is how to structure the HTTP request in the node so that the body change in each request.

I know how to do it when a query parameter is changing in each request but not when is part of the body.

5

$pageCount will increment for each request in the pagination within the actual pagination section.

If you want to change the offset by 500 each time, use {{ $pageCount * 500 || 0 }} The || 0 part of the expression sets the default value to 0 if $pageCount is undefined, which it is on the first one. (If their data starts at 1 instead of 0 then you should put {{ $pageCount * 500 + 1 || 0 }} )
Then you just set type to body.

See that in action in this node

I set up an endpoint to return the data back, and this is what this sends back, showing it’s working.
Screenshot 2024-02-05 at 4.42.17 PM
(Edit: huh I notice now this picture does not show it working. But I promise this does infact work)

I suspect you’ll run into an issue because it doesn’t let you nest that json in the pagination section, it looks like it needs to be under params/args.

You can still make that work, it will just be a lot more complicated because you’ll need to set up a loop instead of just letting the HTTP node handle it. I made the whole thing here:

You will need to adapt this to work for you in terms of the amount of loops you have run.

I tested by pinning 3 items in the webhook and passing those in to make the loop run 3 times.

In the body of the json in the HTTP request {{ $('Loop Over Items').context["currentRunIndex"] * $json.limit || 0 }} is setting the offset. $('Loop Over Items').context["currentRunIndex"] gets the current loop of the Loop Over Items node

$json.limit is from the Set Variable For Batch SIze/limit node. I set the size there so that you only had to change it in one place. I set it to 50 for my testing, you can go in and change it to 500.


Let me know if any of this doesn’t make sense and if you have any follow up questions or need help getting it to work for your usecase.

6

Hello Liam,

Thank you very much for your answer, but I just wanted to know how the HTTP Request node could handle the pagination using its standard features, I wanted to avoid having a loop.

7

The first part of my response show’s just that, using just the http node.

It doesn’t seem possible to nest it deeper into a json structure using just the built-in feature though.

8

Thanks a lot for your time, my point from the beginning was how to nest the variable into the actual body (like the one I share) which is generated in previous nodes.

9

Set the input field to expression then write out the json as usual and add in your parameters in {{ }}

10

Where? In the “Value”?

image

What name do I give it? When leaving it blank it doesn’t work.

I tried putting in the standard body section but the pageCount wasn’t increasing;

image

11

Thee $pageCount only works in the pagination section.

And in the pagination section you’re only able put key/value pairs not json, which is what i was saying would be your issue here and why I suggested the custom loop.

In the key value section in the json, it adds the data you enter into the json but it does not accept json. For example, if you enter:
Name: limit
Value: 500

That will be added to the body as

{
   "limit":"500"
}

So if you add Json to the Value it would look like this:

{
   "fieldName":"{"fields": [ "id"],"limit": 500,"offset": 0}"
}

It adds it as a string, not adding as json. I have seen APIs accept data like this but it’s not common, since they will need to parse the json to make it valid.

If you wanted to try, you would take an “key” from the json, which is what the “Name” field actually converts it to. So you would do:

Name: params

Value : { "service": "object", "method": "execute_kw", "args": [ "X", "Y", "z", "res.partner", "search_read", [[]], { "fields": [ "id" ], "limit": 500, "offset": {{ $pageCount * 500 || 0 }}* } ] }

Then in the regular body section you would still need to add this:

{
   "method": "call"
}

The output of this will look like it’s working but it won’t be valid json, so it likely won’t work.

The easiest way forward will likely be just trying the loop. You can copy and paste what I have, it shouldn’t take much more to get it working

Hope that all makes sense

12

Thanks for the time you’ve put into this!
I will leave the loop solution I already have in place.

1 Like
13

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.