Data Transformation, Arrays, Object, Elements, Items, Functions, etc

I’ve been playing around with n8n since the end of January, and managed to build quite a few working workflows. I even started hosting my first real production workflows for customers a couple of weeks ago. I really like n8n, it’s crew and community, and it’s giving me the same vibes as a had when I started with FileMaker some 30+ years ago. And n8n is now part of my every day development, alongside with FileMaker and AppDrag.

The thing that made me wonder, though, is that it sometime takes me a couple of minutes to build a workflow, and sometimes I can get stuck for hours on something that is apparently very simple. The majority of my workflows has some kind of data transformation, and although the Item Lists and Set nodes can help you stay on the low-code side of things, before you know it, you need the Function or Function Item nodes, and you’re in all-code. And since that’s the part I get stuck in most of the time, I started to dive into this subject. I read and re-read the docs on key concepts, expressions, Function nodes, JavaScript code snippets, etc. as well as community posts tagged data-transformation, transform-array and transform-string. It finally struck me that the syntax of the functions and expression actually differs depending on where you are. I started to notice the difference in the way that sample/mock data was being created in examples. In some function nodes you’ll find:

return {
    employees: [
        { firstName: 'John', lastName: 'Doe' },
        { firstName: 'Anna', lastName: 'Smith' },
        { firstName: 'Peter', lastName: 'Jones' }
    ]
};

In others, you’ll find:

return [{ json: 
{
    "employees": [
        {"firstName":"John", "lastName":"Doe"},
        {"firstName":"Anna", "lastName":"Smith"},
        {"firstName":"Peter", "lastName":"Jones"}
    ]
}
}];

For somebody who’s really new to JavaScript, I first had to understand that there’s actually a difference between arrays in JavaScript and JSON. And as far as I understand now, data can be going back and forward between the two in function nodes.

I then also noticed the different behavior between the Function and Function Item nodes when returning the results …

return result
return [{json: result}]
return [{json: {result}}]

When returning this result:

[
  {
    Productnumber: 'test123',
    EAN_barcode: 123456,
    ....

the first and second will create an object:

[
  [
    {
      "Productnumber": "test123",
      "EAN_barcode": 123456,
      ....

and the third and array:

[
  {
    "result": [
      {
        "Productnumber": "test123",
        "EAN_barcode": 123456,
        ....

And then there’s the different data formats that the data can be in, being:

And to clarify my description of the different JSON structures:

array … [{…:[{…},{…},{…}]}]

[
  {
    "employees": [
      {
        "firstName": "John",
        "lastName": "Doe"
      },
      {
        "firstName": "Anna",
        "lastName": "Smith"
      },
      {
        "firstName": "Peter",
        "lastName": "Jones"
      }
    ]
  }
]

object … [[{…},{…},{…}]]

[
  [
    {
      "firstName": "John",
      "lastName": "Doe"
    },
    {
      "firstName": "Anna",
      "lastName": "Smith"
    },
    {
      "firstName": "Peter",
      "lastName": "Jones"
    }
  ]
]

multiple items … [{…},{…},{…}]

[
  {
    "firstName": "John",
    "lastName": "Doe"
  },
  {
    "firstName": "Anna",
    "lastName": "Smith"
  },
  {
    "firstName": "Peter",
    "lastName": "Jones"
  }
]

And as for the multiple items, to further illustrate this structure by comparing it to other environments:

Screenshot 2022-03-19 at 13.22.40

To those of you who have been working with n8n for a long time, this is probably obvious and second nature, but for newbies, or at least to me, this is far from straight forward.

Finally, like almost anywhere, it comes down to context. And although I get the feeling that I’m finally getting my head around this, data transformation still remains an enigma to me. Maybe the introduction of the JMESPath query language for JSON might bridge the gap between low-code and all-code. Especially with the help of a template workflow with examples of the most commonly used data/array transformations in n8n.

P.s.: this is my own and current interpretation of how I understanding that things work, and maybe I’ve got it completely wrong. If so, please so not hesitate to tell me.

5 Likes

This may further illustrate the different methods and syntaxes that can be used to find an object by a key value and return another key value.

As well as the different approaches in a workflow

This a solution to the post Get a value of a specific trello tag

1 Like

but what about instead of searching for the key result, you want to search for the key itself and have those results listed?