How can I store result of database queries and use them for later use, avoiding repeating queries to database

Questions
1

Hi,

I have a workflow (see attached image) triggered by a Schedule Trigger. It has two branches:

  • Upper branch: Fetches emails from Gmail and processes them.
  • Lower branch: Fetches a list of emails from Postgres, processes the data, and sets an email_list.

Both branches meet at an IF node, where I want to compare the extracted email from Gmail with the list from Postgres.

My goal:
I want the lower (Postgres) branch to run only once a week, cache its results, and then use that cached data for the daily runs of the upper branch (Gmail). This would avoid querying the database every day.

My questions:

  1. How can I cache the results from the Postgres branch and access them in the IF node during daily runs?
  2. What is the best way to structure this workflow so that the IF node can compare the Gmail email with the cached email list, even though the database branch is not executed every day?
  3. Are there built-in n8n nodes or best practices for this kind of in-memory or persistent caching between workflow runs?

WORKFLOW:

1 Like
2

Architecturally your best solution would to be store the cached items in a Redis db. Redis is an in memory key/value store widely used for caching purposes.

So your workflow will then be as you described. Build a workfow which runs once a week and queries data from postgres and ten stores it in redis. Then in your regular workflow you currenly have, you just load from redis using the redis nodes. You just need to cater for cases where the server maybe rebooted and needs to rehydrate your cache.

If you’re self hosting n8n, you simly just need an instance of redis running along side n8n and then connect to it

PS: for persisted caching, you can also configure redis to store its records to disk, but it will promarily run from memory

2 Likes
3

Hello @Himanshu_Rana

There are a few options, feel free to check out the workflow, but I think as mentioned redis is a good option perfectly acceptable here in this use case, quick fast in mem plus ability to restore and scale.

You have kafka too if you want Kafka it would acts as a high-throughput message buffer, allowing the workflows to push email data instantly without waiting for database writes. A separate consumer workflow can then process and store the data in the database asynchronously, near no data loss.

Also maybe a less obvious choice, but you could use a code node and write to disk but I’d push toward redis.

if speed and concurrent access matter Redis.
If durability Kafka.
If simplicity and occasional reads/writes disk write file is good enough.

Hope this helps further,

oh and there is execution data, I’ve not used much but here is link to it

Kind regards,

Samuel

2 Likes