Workflow freezes with large dataset (22k rows) – how to optimize grouping and export to SharePoint?

[Dominika_Kaminska]

Hi

I’m building a workflow that should:

1. Fetch sales data (around 22,000 rows) from a file (Excel/SharePoint).

2. Group the data by the column “Partner Name” (~100 unique partners).

3. For each partner, generate a separate Excel file and upload it to SharePoint.

Current approach

• Get Rows → fetch all 22k rows.

• Code node → group data by partner.

• Loop Over Items → for each partner, create a file (Convert to File) and upload it (Upload to SharePoint).

Problem: the workflow hangs and freezes (especially in the Code node and Loop Over Items).
Sometimes it runs for a very long time, other times it never finishes.

Questions

1. What’s the best way to handle this use case in n8n?
2. Has anyone else faced a similar issue with large datasets (20k+) and SharePoint export?

Expected outcome

A workflow that can reliably:

• Fetch all rows,

• Extract unique partners,

• Generate and upload one file per partner,

• Without freezing or stalling after a few thousand rows.

Information on your n8n setup

• n8n version: 1.107.4
• Running n8n via ( n8n cloud )

[jabbson]

Hey @Dominika_Kaminska hope all is good. Welcome to the community.

22k rows is many rows, storing and processing this amount of data is what makes the instance ill. You can see the limits for the n8n cloud instance per plan here:

Cloud data management | n8n Docs

That being said, you will have to come up with a better way to deal with large amounts of data. One things that immediately comes to mind is having 22k rows in a spreadsheet is … well… not ideal. To store a lot of tabular information, it is better to have it in the database. Your best bet would be to either come up with a way to chunk up your input into much smaller portions, OR, better yet, ingest your data into a database first and then manipulate this data using specialized database optimized methods for sorting, grouping and filtering.