Automating Lead Data Filtering by Category

Questions
1

Hey everyone, I need some help with an automation project!

I’m working with a large dataset of about 8 million random leads, which includes a mix of categories like dentists, entrepreneurs, real estate, etc. My goal is to filter these leads by category (e.g., dentists) and save each filtered list into separate CSV files on my local computer.

I’m using MySQL to store the data and looking for the best way to automate this process, ideally with n8n. Has anyone tackled something similar? I’d love some advice on how to set this up, or if there are better tools or workflows I should consider.

Any insights or suggestions would be much appreciated! Thanks in advance! :blush:

2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

@Roxester , certainly n8n should be able to handle that. It would probably be best if you run SQL queries on MySQL in a way that the returned records (their size) are reduced to a minimum - only the data you need excluding any unnecessary properties. Perhaps even run filtering in separate workflows to ensure you do not hit the memory overload when running n8n.

4

Hi,

I am actually doing a similar thing with an agent connected to Google. Its good :))

You can use pagination on the database query and process everything in small batches.

It would take some time but it can be done very easily.

Let me know if you have any other questions.

5

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.