Writing JSONL to file

[zenweasel]

Describe the problem/error/question

I want to write JSONL to a file. JSONL is a more compact format for JSON and has one JSON object per line. I also do not want the data Stringified.

So I have a code node before it that basically turns all the JSON into one large string. I then want to write that string, unaltered. If I try to use “Write Binary File” is says the input contains no Binary data. If I try to use “Move JSON to Binary” it won’t work because my data is no longer JSON, it’s just one big string. Also since my code node can only return an array of objects I have to wrap my string with an object and an array, neither of which I want. I also tried the CSV export but it also wants to manipulate the data, which is not what I want, I want to just write the data to a file.

I am just about ready to write a custom code for this, but it really seems like this is probably possible out of the box and it’s probably starting me right in the face.

What is the error message (if any)?

I either get the “No Binary Data” from write binary file or I get unexpected results from the “JSON to Binary” node.

Information on your n8n setup

• n8n version: 1.x
• Database (default: SQLite): Postgres
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app):
• Operating system: Linux

[jan]

Sadly not on a computer right now so can not post a solution but your approach with Move to Binary node should work. You probably just have to play around with the settings a little bit.

[zenweasel]

Coming back to this and still banging my head up against the wall. Have spent quite a bit of time playing around with the options with always the same results. I definitely feel like this is user error here but I am out of ideas.

Here is the code that I am using to convert the JSON array to JSONL

let fileData = '';
for (const item of $input.all()) {
  if (item.json) {
      const jsData = JSON.stringify(item.json, null, 0);
      if (jsData) {
          fileData += jsData + "\r\n";
      }
  }
}

console.log("fileData", fileData);

return [ {fileData: fileData } ];

I am pretty sure that I have played with every option for Move Binary Data and nothing produces a single JSONL file for all elements. The closest I got was producing a JSON file that looks like this:

which is close but it’s still wrapping the whole thing in a JSON object which it should not and will make the upstream system reject it. Getting really desperate to get this working now so any help would be greatly appreciated.

[MutedJam]

which is close but it’s still wrapping the whole thing in a JSON object which it should not

Can you try disabling the “Convert All Data” option on your Move Binary Data node and enable the “Use Raw Data” option like below?

Sounds like these settings might be the missing piece here.

[zenweasel]

Thanks. I had tried that before and it wasn’t working for me.

What worked for me seems to be not to use the “Move Binary Data” at all and use a regular code node with this snippet, which solves both the format issue and being able to upload to S3

items[0].binary = {
  data: {
    data: new Buffer(items[0].json.fileData).toString('base64')
  }
};
return items;