Addition assignment (+=) operator with the Set node in n8n

[BigAndini]

Describe the issue/error/question

I’d like to request data from an api that provides pagination only and it’s not possible to request all data at once. I’ve searched for solutions but didn’t find any suitable.

• What I did is to initialize the limit and the offset with a Set node, I do use a limit of 200 sets and I’ll start at an offset of 0.
• Do the first http request and get the first 200 entries
• the api gives the amount of data sets which I collect with the “total amount of CRM data” Set node
• CRM - Data collects the CRM data and brings the needed data into a table
• the if statement is my problem. I’d like to check if all requested data is larger or equal the total amount of data, however for that I would need to calculate in some variable how many datasets I’ve already requested from the api.

How can I set the next offset for the next HTTP Request? I would need a Set node which is capable of doing a Addition assignment operator. So I start at 0 and I know I’ve requested 200 datasets, the if statement checks if all data was received and the Set node needs to increase the offset by 200, so the next HTTP Request will start with limit 200 and offset 200.

I would appreciate it if you could give me an idea how to implement that in n8n or link to an article which I didn’t found, yet.

Information on your n8n setup

• n8n version: 0.216.2
• Database you’re using (default: SQLite):
• Running n8n with the execution process [own(default), main]: I don’t know, it’s simply the desktop app
• Running n8n via: desktop app as of now

[BigAndini]

I do the variables for the http request like this, however it looks like I don’t get the right input variables when I choose the other input from the dropdown:

It looks like the HTTP Request doesn’t interpret the second input correctly. I would expect that the HTTP Request uses the according settings depending from which input it’s triggered.

Regards,
Andreas

[MutedJam]

Hi @BigAndini, welcome to the community

How can I set the next offset for the next HTTP Request? I would need a Set node which is capable of doing a Addition assignment operator. So I start at 0 and I know I’ve requested 200 datasets, the if statement checks if all data was received and the Set node needs to increase the offset by 200, so the next HTTP Request will start with limit 200 and offset 200.

Based on this description it seems to me you can use the $runIndex variable (documented here) for this. Here’s a quick example workflow:

For the first execution this HTTP Request node would use an offset value of 0, then 200, then 400 etc.:

Is this what you had in mind?

[BigAndini]

Hey @MutedJam,

thank you for your reply. Yes, that makes things much easier. Now I need to collect the data from the HTTP Request into a variable to further work on it later in the flow. So I found some code to collect the information, however it seems like the new Code Node will not work with this code anymore:

const allData = []

let counter = 0;
do {
  try {
    const items = $items("HTTP Request - CRM", 0, counter).map(item => item.json.results);
                    
    const aja = items[0].map(item => {
      return { json: item }
    })    
    
    allData.push.apply(allData, aja);
    //allData.push($items("Increment", 0, counter));
  } catch (error) {
    return allData;  
  }

  counter++;
} while(true);

I’ve adjusted the $items element to the name of my HTTP Request - CRM Node.

These are the errors I get:

When I click on the “fix” link I get a syntax error:

What’s a nice way to collect all the data from the different rounds ot http requests, so it’s possible to merge t hem with the upper branch of the flow?

Regards,
Andreas

[BramKn]

Hi @BigAndini

Because I didn’t want to mess around with code nodes every time. I created a community node that deals with the iterations and also has an option to combine data for further processing.
Keep in mind that if you have a lot of data the server might not like it.

n8n-nodes-iterator

[BigAndini]

I saw an example with some kind of code, however my api does not give a next or previous page within the api. Therefore I need to walk through the offsets and limits on my own. This looks like it’s working cool with @MutedJams input. However the data collection part to further work on it is not working as it should be.

So as of now we have only about 500 records, but the crm api let us only request 200 at a time. So this should work even when we have 10k to 20k records in the database, I think.

Thank you.

Regards,
Andreas

[BramKn]

Yeah it should work, just wanted to mention this community node as it does everything you need without needing to mess around with code.
using offsets etc. also is possible not just setting next page values.

[MutedJam]

Tbh the node solution suggested by @BramKn is way more elegant. But if you want to implement this using the standard nodes you might want to try out this approach: Merge multiple runs into one | n8n workflow template

It doesn’t throw any errors on the current version of n8n: