HTTP Request node and node data formats - unintuitive behavior or I don't get it

Questions
1

Hi all, hopefully this is me being stupid. I really do NOT get this is supposed to work. I am using an HTTP request node to query the Okta API to get a list of users. Okta will return a simple array with user objects like this:

[ 
  { "id": "id0",
    "profile": {
      "firstName": "hey",
      "lastName": "ho silver"
    }
  },
  # etc. pp
]

I connected the REST node to an iteration node, because I am only interested in the value of a single property (“id”). Now it get’s funny. It seems that the REST node is putting the whole array returned from Okta into another array, which looks like this if you open it:

[
  [
    {
      "id": "XXXXxxxxxxx.......",
      "status": "ACTIVE",
      "created": "2018-07-23T15:25:00.000Z",
      "activated": "2018-07-23T15:25:01.000Z",
      // etc. etc.

Which does not make sense to me, but okay.

Now I thought “okay, let’s remove the outer array” with a function node doing this: return items[0]; (cause this should do the trick, right?!) I get this result:

Error: Always an Array of items has to be returned!
    at Object.execute (/Users/tm/.npm/_npx/a8a7eec953f1f314/node_modules/n8n-nodes-base/dist/nodes/Function.node.js:74:19)
    at processTicksAndRejections (node:internal/process/task_queues:93:5)
    at async /Users/tm/.npm/_npx/a8a7eec953f1f314/node_modules/n8n-core/dist/src/WorkflowExecute.js:395:47

Since we have an array which contains only another array this should not be an issue, right?! But no.

Also when I switch to the “table” view - all the elements are displayed correctly!! So the node does somehow recognize that there are multiple elements.

Thinking maybe this is just normal I try to extract the “profile” sub-object using a FunctionItem node (return item["profile"] as single code line), but now I get this:

Error: No data got returned. Always an object has to be returned!
    at Object.executeSingle (/Users/tm/.npm/_npx/a8a7eec953f1f314/node_modules/n8n-nodes-base/dist/nodes/FunctionItem.node.js:78:19)
    at processTicksAndRejections (node:internal/process/task_queues:93:5)
    at async Promise.all (index 0)
    at async Workflow.runNode (/Users/tm/.npm/_npx/a8a7eec953f1f314/node_modules/n8n-workflow/dist/src/Workflow.js:481:34)
    at async /Users/tm/.npm/_npx/a8a7eec953f1f314/node_modules/n8n-core/dist/src/WorkflowExecute.js:395:47

When I (just for testing) replace this with return item; - it works and returns everything.

I just don’t get it. Can somebody elaborate maybe?

image
image1650×1284 257 KB

2

okay, I officially give up. I tried even more things now:

let rv = [];

// this changes
for (i of items[0]) {
  rv.push({"json": [i]})
}

return rv;

result: items[0] is not iterable

//...
for (i of items) {
//...

result: one big array-in-array

//...
for (i of items.json) {
//...

result: items.json is not iterable

//...
for (i of items.json[0]) {
//...

result: Cannot read property '0' of undefined

I just. Don’t. Get. It.

3

@flypenguin Check the example below. You have to map the data that the HTTP node returns to something n8n “understands”. Also, check the n8n data structure here.

{
  "nodes": [
    {
      "parameters": {},
      "name": "Start",
      "type": "n8n-nodes-base.start",
      "typeVersion": 1,
      "position": [
        250,
        300
      ]
    },
    {
      "parameters": {
        "url": "https://oktamockup-be4ax5fm3jgo.runkit.sh/",
        "options": {}
      },
      "name": "HTTP Request",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 1,
      "position": [
        570,
        300
      ]
    },
    {
      "parameters": {
        "functionCode": "const results = []\n\nconst users = items[0]\n\nfor (const user of users.json) {\n  results.push({ json: user})\n}\n\nreturn results;\n\n"
      },
      "name": "Map data to n8n data structure",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        850,
        300
      ],
      "notesInFlow": true
    },
    {
      "parameters": {
        "functionCode": "const results = []\n\nfor (const item of items) {\n  results.push({ json: { id: item.json.id } })\n}\n\nreturn results;\n"
      },
      "name": "Pick just id",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        1090,
        300
      ]
    }
  ],
  "connections": {
    "Start": {
      "main": [
        [
          {
            "node": "HTTP Request",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "HTTP Request": {
      "main": [
        [
          {
            "node": "Map data to n8n data structure",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Map data to n8n data structure": {
      "main": [
        [
          {
            "node": "Pick just id",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}
4

yup; i read that page (even before, multiple times). but - thanks to your input I got it working!!

as a side node to the devs: I really find this is super confusing and unintuitive and error-prone and annoying. this cost me half the day. also now every time I use and function node I have to think “which array is now in which object under which key”. that should be handled transparently in my opinion.

for the recod - the working code is copied 1:1 from @RicardoE105 :

// INPUT seems to be:
// items = [
//   {
//     "json": [ the okta object ... ]
//   }
// ]

const rv = []
const users = items[0];
for (const i of users.json) {
    rv.push({json: i})
}
return rv;

if at all possible this would be the way better solution:

return items[0];

nobody has to think here.

1 Like
5

As a fellow dev do I totally agree that it can be very very confusing if the data structure is not known. But if it is known it should not be anymore. The Function-Node is the lowest level node that exists in n8n and all it expects is the data in a specific format, which is in the simplest form [ { json:{} } ], and also expects the data to be returned in exactly the same.

return items[0]; can not work as it does not follow that predefined schema of how n8n works internally. I agree that if the json property would not be there, it would be easier but as n8n can also supports binary data, that limitation is there.

If you look for a simpler version of that code you can also write:

return this.helpers.returnJsonArray(items[0].json);

It then takes care of the json part for you.

We could also create another Function-Node which only works with the json part of the data. But to be honest, do I not want to end up with too many different Function-Nodes (I kind of still regret having created two different ones already) as I really believe the normal Function-Node is not too hard to understand. And again it is the “low-level dev node” where that kind of complexity should be OK.

6

all right, from an architectural standpoint - yes. I also thought that this is the solution for the “binary data” thing after I posted. whatever you say, I still stand by the “error prone” part :wink:

maybe here’s an idea: dumb down the function node and default to json → json translation (and error out on binary data), then add an option to enable “raw” mode which drops back to the current state of things?

I consider basically any way better than the “object(s) with a ‘json’ key in array” structure, when all I want to do is to pass an array around. btw it also is pretty confusing to retrieve an array from a REST API, and have that treated as monolithic object in the following nodes.

also the display is super weird: if you open the node it will show you a “[ [ ...” structure, which is just not what is being sent around.

I get it now, but I think here is a lot room for improvement. on the other hand - thank you, it might be that this is just THE product I was looking for :slight_smile: (maybe even with the confusing data structure :wink: )

7

clarification: this is what is being shown in the nodes if you look at the data:

this is how you have to treat the data in code:

// the ACTUAL input seems to be:
items = [
  {
    "json": [ "the array from the okta API" ]
  }
]

or am I missing something?

8

Yes, I totally agree there with you. Is for sure more “error prone” and does also not make it easier. But the thing that I wanted to achieve with the Function-Node is that users can do as much as possible. For that reason exposing the raw data was the only way to do that. As also I saw that it can be a little bit confusing in the beginning, did I create the FunctionItem-Node which operates on a per-item basis and only on the json-part of the data. But because of the per-item part would it not help in this use case. Generally did I realized that the node causes much more confusion than it actually helps. I personally, for example, never use it.

The just mentioned FunctionItem node is exactly what I had planned as this simple version you are more or less talking about.

About receiving an Array from an REST API. That is actually also one thing I regret. What I should have done is to simply return that array automatically as separate items (esp. as an array directly under json is not officially allowed). So that, what you are trying to do, would then never have been needed. But I did sadly not think about that originally, and once I realized that mistake, was it sadly too late as it would then have broke a lot of workflows. So until we have node-versioning in place, can we sadly not change that. It is, however, on the list as soon as we have it.

About the raw mode. Agree that this could make things simpler but could also cause again quite some confusion because then the code in the Function-Node would be totally different depending on the activated mode. So also not really perfect.

Then what you mentioned with the [ [ ... structure. Yes, that is also a little bit confusing because of the json part. As that is the only thing that gets displayed in the UI. So here your slightly changed example:

items = [
  {
    "json": { a: 1 }
  }
]

and if we remove the { json: ... } part we end up with:

items = [
  { a: 1 }
]

Hope that makes now more sense, and why it is confusing but still kind of correct at the same time.

Very glad to hear that n8n in generally THE product you were looking for. Even with the INITIALLY ( :wink: ) confusing data structure.

9

ha, thanks for that long answer. I kinda disagree … since you’re not on v1.0 yet (are you? :wink: ), so breaking changes should be possible. although super hard, yes.

on a related note - i am trying to implement an okta event sink, which is a bit special and not “only” web hook, and I stumbled over a possible licensing issue. I just tweeted to @n8n about this cause I can’t open more posts here on my first day.

could you help me out here? cause if I understand that correctly …

10

Ah yes we do breaking changes but we try to keep them to an absolute minimum:

The thing here is more the difference between what we can do (make a breaking change as we are not 1.0 yet, so theoretically nobody can complain) and what we should do (unnecessarily break the workflows of very very many people). Just because we theoretically can, does not make it a good thing to do, especially if there is not a very good reason that we have to do it right now and we can avoid a breaking change in a not to distant future once node-versioning is in place.

I replied to your tweet with a link to our FAQ which should help to answer your question.