Custom node dev - How to deal with binary data?

shrey-42 · August 30, 2021, 12:13pm

I’m a bit stuck with the code for dealing with binary data, in my custom node dev process.

One of my functions in GenericFunctions.ts, that deals with downloading binary content of files, is facing issues and is defined as follows:

export async function directusApiAssetRequest(this: IHookFunctions | IExecuteFunctions | IExecuteSingleFunctions | ILoadOptionsFunctions | IWebhookFunctions, method: string, path: string, ID: string, dataPropertyName: string, qs: IDataObject = {} ): Promise<any> { // tslint:disable-line:no-any

    const credentials = await this.getCredentials('directusApi') as { url: string, accessToken: string };

    if (credentials === undefined) {
        throw new NodeOperationError(this.getNode(), 'No credentials got returned!');
    }

    const params = credentials;
    const url = params.url!.replace(/\/$/, "") || null;
    const accessToken = params.accessToken! || null;

    const optionsAsset: OptionsWithUri = {
        headers: {
            'Content-Type': 'application/json',
            Authorization: `Bearer ${accessToken}`,
        },
        method,
        qs,
        uri: `${url}/${path.replace(/^\//, "")}`,
        json: true,
    };

    try {
        const resFile = await this.helpers.request!(optionsFile);
        const file = resFile.data;

        const res: any = await this.helpers.request!(optionsAsset);
        const binaryData = Buffer.from(res);
        const data = binaryData.toString('base64');
        const binary = {
            [dataPropertyName]: {
                data: "",
                fileName: 'fileName',
                mimeType: 'mimeType'
            } as IBinaryData
        };
        binary![dataPropertyName] = await this.helpers.prepareBinaryData(binaryData);

        const json = {};
        const result : INodeExecutionData = {
            json,
            binary
        };
        return result ;
    } catch (error) {
        throw new NodeApiError(this.getNode(), error);
    }
};

But, i’m seeing this error while/before compiling the code:

nodes/Directus/GenericFunctions.ts:136:56 - error TS2339: Property 'prepareBinaryData' does not exist on type '{ prepareBinaryData(binaryData: Buffer, filePath?: string | undefined, mimeType?: string | undefined): Promise<IBinaryData>; getBinaryDataBuffer(itemIndex: number, propertyName: string): Promise<...>; request: RequestPromiseAPI; requestOAuth2(this: IAllExecuteFunctions, credentialsType: string, requestOptions: (UriO...'.
  Property 'prepareBinaryData' does not exist on type '{ request?: RequestPromiseAPI | undefined; requestOAuth2?: ((this: IAllExecuteFunctions, credentialsType: string, requestOptions: (UriOptions & CoreOptions) | RequestPromiseOptions, oAuth2Options?: IOAuth2Options | undefined) => Promise<...>) | undefined; requestOAuth1?(this: IAllExecuteFunctions, credentialsType: s...'.

136         binary![dataPropertyName] = await this.helpers.prepareBinaryData(binaryData);
                                                           ~~~~~~~~~~~~~~~~~

Would appreciate any help in figuring this out!

harshil1712 · August 30, 2021, 12:19pm

Hey @shrey-42,

We changed how the Binary Files are handled in n8n in the version 0.135.0. You can read more about it here: n8n/BREAKING-CHANGES.md at master · n8n-io/n8n · GitHub

This might help you with debugging.

shrey-42 · August 30, 2021, 12:32pm

Hey @harshil1712 , thanks for the link.

Although, as i understand, the 2nd point in the link (concerning binary data) illustrates the change for incoming binary data only. Whereas, i’m currently trying to write the binary data after an API (request) call.

So, i’m unclear as to how to deal with that.

RicardoE105 · August 30, 2021, 3:52pm

Can you point me out to the endpoint you are trying to use?

Do not think the HTTP it’s downloading the data as the property JSON is set to true. That would try to parse the response to JSON. You do not want that. Else the binary data would get messed up.

json - sets body to JSON representation of value and adds Content-type: application/json header. Additionally, parses the response body as JSON.

shrey-42 · August 31, 2021, 7:59am

Hey @RicardoE105 ,

I ended up solving this issue by avoiding the helpers function altogether.

I directly created the binary object, e.g.:

const binaryPropertyName = additionalFields.binaryPropertyName as string;
let fileName = additionalFields.fileName as string;
let binaryData, mimeType, fileExtension;

binaryData = Buffer.from(JSON.stringify(response));
mimeType = 'application/json';
fileExtension = 'json';
fileName = `${fileName}.${fileExtension}`;

const data = binaryData!.toString('base64');
binary = { [binaryPropertyName]: {data,fileName,mimeType} as IBinaryData} as IBinaryKeyData;

returnItems.push({ json: response, binary });

So, didn’t get to debug what the issue really was when i tried this with
binary![dataPropertyName] = await this.helpers.prepareBinaryData(binaryData);

jan · August 31, 2021, 8:34am

You can do that but be aware that your node will break very soon as setting it directly will not be supported anymore.

shrey-42 · August 31, 2021, 8:36am

@jan
Ah, good to know now as i’m still building it.

Is binary![dataPropertyName] = await this.helpers.prepareBinaryData(binaryData); indeed the recommended way then?

If it is, would love to know more about how to use it correctly. Any documentation/source files to go through?

jan · August 31, 2021, 8:47am

Yes, that will actually be the only way moving forward. We are changing how n8n works with binary data internally. Right now does the data get passed along as base64, in the future will the actual data be saved somewhere else and we will only pass along a reference to that data (like an ID). That will enable n8n to finally work with very large binary data, as no longer all the data has to be kept in memory.

Sorry there is not special documentation about that yet but a lot of other nodes that use it already. Also is the great thing about TypeScript that the documentation is kind of built in as it clearly defines how the data should look like that it expects:

So expects as first parameter a buffer, and then optionally filePath and mimeType.

Looking at your code, is the problem you are facing less about how you use that method (even though you first set data manually and then totally overwrite it, so the lines before that call are unnecessary) the problem is more that you define that this can be of very many different types and two of those (IHookFunctions, ILoadOptionsFunctions) do not have helpers.prepareBinaryData() defined. Guess if you remove those it should work fine, or if not you are at least closer.

shrey-42 · August 31, 2021, 8:52am

Thanks for the info! Will try these modifications.

shrey-42 · August 31, 2021, 9:36am

Finally it works. I just removed-

But, how to pass/retrieve the filename, fileExtension properties?
Do they still need to be set after the call?

This doesn’t seem to take those parameters-

jan · August 31, 2021, 9:43am

Glad to hear!

You can provide the filename as filePath. n8n will then use it as filename and extract the fileExtension also automatically.

shrey-42 · August 31, 2021, 9:57am

Great, thanks!

shrey-42 · August 31, 2021, 10:13am

Hi, i do have one more query.

In another use-case i have the raw data and need to create a Buffer and pass it on to the binary property. My current implementation looks like this:

const exportType = additionalFields.export as string ?? null;
let binary : IBinaryKeyData = {};
if(exportType){
    const binaryPropertyName = additionalFields.binaryPropertyName as string;
    let fileName = additionalFields.fileName as string;
    let binaryData: Buffer, mimeType, fileExtension;

    if(exportType == 'json'){
        binaryData = Buffer.from(JSON.stringify(response));
        mimeType = 'application/json';
        fileExtension = 'json';
        fileName = `${fileName}.${fileExtension}`;
    } else if(exportType == 'csv') {
        binaryData = Buffer.from(response);
        mimeType = 'text/csv';
        fileExtension = 'csv';
        fileName = `${fileName}.${fileExtension}`;
    } else if(exportType == 'xml') {
        binaryData = Buffer.from(response);
        mimeType = 'application/xml';
        fileExtension = 'xml';
        fileName = `${fileName}.${fileExtension}`;
    } else {
        binaryData = Buffer.alloc(0);
        mimeType = "";
    }
    const data = binaryData!.toString('base64');
    binary = { [binaryPropertyName]: {data,fileName,mimeType} as IBinaryData } as IBinaryKeyData;
}

I guess i need to replace

    const data = binaryData!.toString('base64');
    binary = { [binaryPropertyName]: {data,fileName,mimeType} as IBinaryData } as IBinaryKeyData;

with

binary![binaryPropertyName] = await this.helpers.prepareBinaryData(binaryData, fileName, mimeType );

?

Also, will i still need to provide the fileName & fileExtension manually or will there be a different property available, that will have to be passed for the filepath which will then automatically return fileName & fileExtension ?

jan · August 31, 2021, 10:39am

That would be the same as in the other case. You would provide only the fileName. As the fileExtension is part of it, can n8n figure it out by itself. So you can simply test it. You should then see if there are problems or all works as expected.

harshil1712 · September 30, 2021, 3:03pm

Hey @shrey-42,

How is it going? Did you get it working? If yes, can you please share the solution

shrey-42 · September 30, 2021, 3:59pm

Hey @harshil1712 ,
yes, this is what i ended up deploying, for the 1st use-case:


export async function directusApiAssetRequest(
    this: IExecuteFunctions | IExecuteSingleFunctions,
    method: string,
    path: string,
    ID: string,
    dataPropertyName: string,
    qs: IDataObject = {}
): Promise<any> {

    const credentials = (await this.getCredentials('directusApi')) as {
        url: string;
        accessToken: string;
    };

    if (credentials === undefined) {
        throw new Error('No credentials got returned!');
    }

    const params = credentials;
    const url = params.url!.replace(/\/$/, '') || null;
    const accessToken = params.accessToken! || null;

    const optionsFile: OptionsWithUri = {
        headers: {
            'Content-Type': 'application/json',
            Authorization: `Bearer ${accessToken}`
        },
        method,
        qs,
        uri: `${url}/files/${ID}`,
        json: true
    };

    const optionsAsset: OptionsWithUri = {
        headers: {
            'Content-Type': 'application/json',
            Authorization: `Bearer ${accessToken}`
        },
        method,
        qs,
        uri: `${url}/${path.replace(/^\//, '')}`,
        json: true,
        encoding: null, //"arrayBuffer",
    };

    try {
        const resFile = await this.helpers.request!(optionsFile);
        const file = resFile.data;

        const res: any = await this.helpers.request!(optionsAsset);
        const binaryData = Buffer.from(res);

        const binary: IBinaryKeyData = {};
        binary![dataPropertyName] = await this.helpers.prepareBinaryData(
            binaryData,
            file.filename_download,
            file.type
        );

        const json = { file };
        const result: INodeExecutionData = {
            json,
            binary
        };
        return result;
    } catch (error) {
        throw new Error(error);
    }
}