Small but hanging my n8n instance - how to process it better?

[ivancarlos]

Hi masters,

I am running this workflow in my n8n, it’s quite simple, but demands a txt process that can take some time. What it does:

It get a small list of domains (2 domains) and download a TXT file with 3MB/150k lines. The idea is:
If any of my listed domains is listed on file, report to me, so I can auction bid it to purchase te wanted domain.

When I run this workflow, it takes sometime to download it, no problem, but when it process to parse the lines and let me merge the results to find if some line is equal of my list, it hangs my server for 2 minutes. I am not a dev, so maybe I could do a better job or have other strategy to have it done. Someone have any tip of how could I do this better?

Thanks a lot and regards

Information on your n8n setup

• n8n version: 1.78.1
• Database (default: SQLite): Postgres
• n8n EXECUTIONS_PROCESS setting (default: own, main): default
• Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
• Operating system: Ubuntu 24.04

[Jim_Le]

Hey @ivancarlos

Probably just a lot of work for the merge node to get through the list. I was thinking you could just do the merge in another code node.

Alternatively, if you’re self-hosting and comfortable using the “execute command” node, you could saves both lists to disk and run the following linux command to do the comparison. Linux commands are likely going to be faster than doing it in code.

sort domains-1.txt -o domains-1.txt
sort domains-2.txt -o domains-2.txt

comm -12 domains-1.txt domains-2.txt > common_domains.txt

Hope this helps!

[Exnav29]

My thoughts exactly!