Enable data pruning

Hey hi, Nico here :smiley:

I did a quick test, cause data were not pruned before.

We are running N8N in Queue Mode, with ECS, each component in a Task Definition, and Postgres on a RDS.

I set these ENV VAR:
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_MAX_AGE=1

I delete Tasks that were running, and also forced a Workflow Manually, and we reduced size from 1.5 GB to 153 MB.

Still I find it weird that with 2212 rows, we have such number, any idea? Maybe storing logs?

Also another thing that I notice, is that I have to trigger a manual dummy Workflow in N8N, to start the prune job. Is it expected? For instance I’ve just triggered it, and number reduced to 29 rows.

Last question, withing those 29 rows, all of them are manual or trigger, these one are not deleted?

Regards,

Nico

Hey @Nicolas_Andres_Calvo.

Depending on your log option for the workflows it is likely logging all of the data items the workflow is processing. The more data you are processing the larger your database is likely to be, It may be worth only logging data from failed executions unless you need to keep everything.

Good point!

We are setting info in N8N_LOG_LEVEL in this environment. But we saw that in PROD we have debug level, and storage grow exponentially.

Regarding, EXECUTIONS_DATA_SAVE_ON_ERROR and EXECUTIONS_DATA_SAVE_ON_SUCCESS both are in all mode. I will do a POC regarding EXECUTIONS_DATA_SAVE_ON_SUCCESS changing to none.

Another question. in Queue Mode, does prune job works? Cause Only works when I trigger it manually from console, and runs on Main Container.

Last question, are manual or trigger rows deleted?

Regards,

Nico

Hey @Nicolas_Andres_Calvo,

It won’t be the N8N_LOG_LEVEL that is just for file or console output, It is will be the EXECUTIONS_DATA_SAVE_ON_X options that are populating the database. With the prune option from what I remember before it is not ran in queue mode, We have an internal ticket for this though which we are tracking as N8N-3765.

From what I understand prune will remove all records and it shouldn’t matter what starts the workflow.

Thank you @Jon

Really apprecitate it! :slight_smile:

Quick update!

I run VACUUM FULL execution_entity; and Rel Size was reduced

Regards,

Nico

Just to make sure I understand. You run n8n in queue mode with SQLite?

Hey hi @jan !

Nope we run it RDS Postgres 13.7

What we observed is that although rows are deleted, disk space are not recovered. In DEV, I run VACUUM FULL execution_entity; to get it back.

Also, trigger and manual executions are not deleted :frowning:

BTW, we are using 0.189.1 version.

Regards,

Nico