HTTP batch get.list - loop and merge through runs. (newbie question)

[Kamil_Muras]

Describe the problem/error/question

What is the error message (if any)?

Please share your workflow

I am coming from make.com and am amazed by n8n. Automating is becoming my life and with n8n I hope I will learn lots of new stuff. Unfortunately make had some features out of the box, where I need to learn getting same effects on n8n. I am just at the beginning so please be understanding. I need to change the way of thinking into more professional.

Could anyone help me merging batches of data. I am able to list only 50 results at one http request with the information which was the last item number. To get the rest I need to include “start”:{value of “next”} from the last batch of data in the request. This is how one batch looks like

[
  {
    "result": [
      {
        "ID": "97833",
        "TITLE": "deal title",
        "TYPE_ID": "SALE",
        "STAGE_ID": "NEW",
        "PROBABILITY": "48",
        "CURRENCY_ID": "PLN",
        "OPPORTUNITY": "0.00",
        "IS_MANUAL_OPPORTUNITY": "N",
        "TAX_VALUE": null,
        "LEAD_ID": "71603",
        "COMPANY_ID": "0",
        "CONTACT_ID": "66603",
        "QUOTE_ID": null,
        "BEGINDATE": "2024-02-03T03:00:00+03:00",
        "CLOSEDATE": "2024-02-10T03:00:00+03:00",
        "ASSIGNED_BY_ID": "19",
        "CREATED_BY_ID": "1",
        "MODIFY_BY_ID": "1",
        "DATE_CREATE": "2024-02-03T12:31:52+03:00",
        "DATE_MODIFY": "2024-02-03T12:35:18+03:00",
        "OPENED": "N",
        "CLOSED": "N",
        "COMMENTS": null,
        "ADDITIONAL_INFO": null,
        "LOCATION_ID": null,
        "CATEGORY_ID": "0",
        "STAGE_SEMANTIC_ID": "P",
        "IS_NEW": "Y",
        "IS_RECURRING": "N",
        "IS_RETURN_CUSTOMER": "N",
        "IS_REPEATED_APPROACH": "N",
        "SOURCE_ID": "3",
        "SOURCE_DESCRIPTION": "x",
        "ORIGINATOR_ID": null,
        "ORIGIN_ID": null,
        "MOVED_BY_ID": "1",
        "MOVED_TIME": "2024-02-03T12:31:52+03:00",
        "LAST_ACTIVITY_TIME": "2024-02-03T12:31:52+03:00",
        "UTM_SOURCE": null,
        "UTM_MEDIUM": null,
        "UTM_CAMPAIGN": null,
        "UTM_CONTENT": null,
        "UTM_TERM": null,
        "LAST_ACTIVITY_BY": "1"
      }
    ],
   "next": 50,
   "total": 1402,
    "time": {
      "start": 1706954368.239976,
      "finish": 1706954368.284911,
      "duration": 0.044934988021850586,
      "processing": 0.023025989532470703,
      "date_start": "2024-02-03T12:59:28+03:00",
      "date_finish": "2024-02-03T12:59:28+03:00",
      "operating_reset_at": 1706954968,
      "operating": 0.778087854385376
    }
  }
]

[n8n]

It looks like your topic is missing some important information. Could you provide the following if applicable.

• n8n version:
• Database (default: SQLite):
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app):
• Operating system:

[jan]

Welcome to the community @Kamil_Muras ! Great to have you with us!

Did you check out the pagination functionally? The docs can be found here:

Pagination | n8n Docs

[Kamil_Muras]

Thank you, hopefully I will ask less stupid questions in the future ^^

[jan]

Glad to hear that it was helpful. There are no stupid questions. It is normally a sign that we have to improve things and make it clearer and easier discoverable for starters.

Have fun!

[Kamil_Muras]

This means that my newbie attitude is valuable here ^^. What I have noticed - the easiest way of learning new things is looking at working examples. I have read the pagination documentation but I lack some knowledge that would let me jump straight into it, so I am looking at YouTube videos which , what I assume on the basis of n8n development pace , might be little outdated. All of this makes learning little too long because I am still learning pagination ^^. Hopefully it will help.

[Kamil_Muras]

Unfortunatelly I am not able to get through this, anyone could help me and show example how to setup this scenario ?
My main issue is how to get the data from response because I see no input that I could process or put into pagination setting. This is the point where I am right now:

[Ruslan_Yanyshyn]

Hay @Kamil_Muras What the response you are getting or error in this case?

[Kamil_Muras]

is response the same as output? I dont know where should I look for response

[Ruslan_Yanyshyn]

When you start this node what the output or response do you have right now with all your last settings?

[Kamil_Muras]

[Ruslan_Yanyshyn]

I meant, when you implemented pagination in node you’ve show in a post above?

Does {{$response.body["0"].next}} shows the Value of 50?

According to the specifications provided by your API provider, you should include it in your subsequent request as a ‘start’ parameter. For more details, please refer to: https://training.bitrix24.com/rest_help/general/lists.php

[Kamil_Muras]

Now the result was I had 3 identical runs with same data because I have limited pagination runs to 3
I think I start to understand what is going on but not sure.
I think I have issue with putting start value into the query or getting response value.
I dont know how to check what value does " `{{$response.body[“0”].next}} " return.
RIght now it looks like this because query as JSON is the only working one:
And I dont know how to change query from json to fields so instead of this: { “order”:{ “STAGE_ID”:“ASC” }, “filter”:{ “ASSIGNED_BY_ID”:17 }, “select”:“*”} I could use fields as query parameters. I started a new topic here: Converting JSON query to fields in HTTP request

I think It has strong connection in the way of using pagination and putting the “next” value in “start” query parameter.

This is a screenshot from browser developer, I think there is no “start” qery value

[Kamil_Muras]

Hey Ruslan, thank you for your answer about fields, but it’s not very connected with the topic. Would you be so kind and give an answer on another topic that I have created so I can credit you and close the topic. Link: Converting JSON query to fields in HTTP request

[Kamil_Muras]

Finally I managed to work it out. Thank you everyone, and am very sorry for being such a noob.

@jan : Mistakes that I have made:

1. Was using query parameters as Fixed instead of Expression ( bad habit from make.com, which was simplifying many things)
2. Did not understand that I can fetch some data from response body by this: {{$response.body.next}} - additionally I didn’t know I need to use “Expression” also. Similary I was looking for “make.com” click and drag - need to learn some coding.

Once again thank you very much, I promise I will add some value in the community once I get used to everything.

[jan]

Amazing, and thanks a lot @Kamil_Muras for sharing your experience. It is not just helpful for other users, also for us to know where we should improve product and/or documentation.