I’m exploring n8n with great interest and one of the first thing I want to try is reading local files and process them in my workflow.
After searching around I’ve realized, with my surprise, there is no existing node that let me read recursively a directory (and filter out the results)
I thought it was a perfect opportunity to create my first custom node
It use recursive-readdir-files utility and import fs from 'node:fs'; under the hood.
The node builds fine, and is linked in my local n8n (it was working fine till I imported recursive-readdir-files)) but it doesn’t register when I run n8n
What is the error message (if any)?
$ n8n
User settings loaded from: /home/es/.n8n/config
Initializing n8n process
Error loading node "FileList" from: "/home/es/.n8n/nodes/node_modules/n8n-nodes-file-list/dist/nodes/FileList/FileList.node.js" - require() of ES Module /home/es/data/LLM/n8n/custom_nodes/n8n-nodes-file-list/node_modules/.pnpm/[email protected]/node_modules/recursive-readdir-files/lib/index.js from /home/es/data/LLM/n8n/custom_nodes/n8n-nodes-file-list/dist/nodes/FileList/FileList.node.js not supported.
Instead change the require of index.js in /home/es/data/LLM/n8n/custom_nodes/n8n-nodes-file-list/dist/nodes/FileList/FileList.node.js to a dynamic import() which is available in all CommonJS modules.
I’ve also tried to run with NODE_FUNCTION_ALLOW_EXTERNAL=* but looking at the generated dist code I think I’ve spot the real issue, the code get translated in const recursive_readdir_files_1 = __importDefault(require("recursive-readdir-files"));
Please share your workflow
(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)
I’ve already made a python script that accepts as input a json with the directory to search, exclude and include directory/files, and slice for batch operations, it’s send back a json. (I’ve spent 3 hours trying to find the right syntax for find)
However I think this is a basic functionality that should work and be present in n8n, my use case is to index local files, but potentially there are hundreds of workflows that could benefit a node that list recursively with a given set of criteria.
So please don’t skip this issue
Ehm… Actually, n8n is not intended to store anything locally
Moreover, you have very limited access to the local storage on the n8n cloud instances (e.g. you can’t execute any command on the instance and have basic capabilities to read/write files)
Best practice is to use external storage for that (AWS S3, Azure blobs, OneDrive, gDrive, etc).
This is for imports to use in the code nodes. You can import npm packages into custom nodes without any extra env setup
So it was registered in n8n before you imported recursive-readdir-files but then stopped working after? Or it never showed up in n8n?
If the import is what caused the issue then I don’t know how to help off the top of my head, I would have to go in and try what you’re doing to actually help. If you PM me a github link i can try it out on my local machine, but that probably won’t be today.
@Jon or @BramKn might know what’s goin on off the top of their head
Actually I’m not storing, just reading btw on premise installations let n8n be used as any other software in your local environment, so this open up a lot of potential use case for n8n…
Yes, I’ve started by setting up my dev enviroment with the node-starter, customized the package.json, build it ok, linked to ~/.n8n/nodes, showed up in the workspace, then by developing it as soon as I’ve imported the library with the ES module n8n would not register it anymore.
You are doing an import inside the custom node?
Not sure if that would just work. When I have used external packages in my nodes I always add them to the dependencies in the package.json. This has always worked, so please try that if you haven’t done that yet.
If you have then I do not know directly why it would not work. If so, we need to wait for mr Pineapple @jon to reply
Note: Also make sure to run npm install to install this extra package.
Just had a minute to look into it and I couldn’t get it working
It looks like it isn’t working because the lib recursive-readdir-files is a ESM only package which n8n doesn’t seem to support yet.
Even though the package included docs for how to use with common js, it still gave an error, I believe because these modules are unsupported.
I’m not sure though, this is the first time im running into this, maybe Jon can tell us more. I won’t tag him for a 3rd time I’m afraid to get blocked
While it may be possible to get it working somehow, if i were you I would find another package to try, use the execute command node like @barn4k suggested, or make your own node based on the execute command code to do what you need.
Yes this is the conclusion I’ve reached as you can see from my first post, but I wonder why there is such a limitation also with custom nodes. I mean, I understand the sandboxing for function nodes but this limitation for custom nodes is probably unintended since there are other nodes that require('fs'); directly or indirectly (chokidar).
Unfortunately I’m not good enough with typescript to fix it, maybe is just a tsconfig settings…
I’ve solved my personal need with a python script, as you can read above, but I think this ESM issue should be solved broadly in n8n, don’t you think?
