I have a workflow that runs every 30 minutes, processes, and filters to keep only those from the last 48 hours. After that, I want to send the results to Telegram, but I need to use Data Tables first to check the unique IDs and filter out any duplicates from previous runs. Since Data Tables have a 50MB limit, how can I automatically delete records older than 48 hours to keep the table ‘infinite’ and avoid hitting the storage limit? Also, where is the best place to insert the new IDs into the table? Sorry for the basic questions, I’m quite new to n8n and still learning how everything works. Thanks for the help!
This is what i have right now…
welcome to the n8n community @ElGarretGG
I recommend use the Data Table as a small deduplication ledger, delete rows older than 48 hours at the start of each run, then insert only the IDs that were successfully sent to Telegram, together with a timestamp. if Data Tables become limiting, move the deduplication store to PostgreSQL/MySQL/Supabase and run a scheduled cleanup query like DELETE FROM sent_ids WHERE sent_at < NOW() - INTERVAL ‘48 hours’;
Welcome to n8n, @ElGarretGG! Great questions for someone just getting started - you’re already thinking about this in exactly the right way.
The others covered the flow well. Let me add one practical detail on how to do the duplicate check step inside n8n with Data Tables:
When you read from the Data Table to find duplicates, you’ll want to use a Code node to filter your current items against the stored IDs. Something like:
const storedIDs = $('Get Data Table').all().map(row => row.json.id);
return $input.all().filter(item => !storedIDs.includes(item.json.id));
This keeps only the items that haven’t been sent yet.
For the cleanup step, use the Data Table node with a Delete operation and set the filter to timestamp < {{ $now.minus({hours: 48}).toISO() }}. Run this at the very start of each workflow execution so your table stays lean.
A quick summary of the full flow:
- Delete rows older than 48h from Data Table
- Fetch your data
- Filter to last 48h only
- Read all IDs from Data Table
- Filter out IDs already in the table (dedup)
- Send new items to Telegram
- Insert new IDs + current timestamp into Data Table
This pattern stays well under 50MB for most use cases. If you start running thousands of unique items per day though, switching to an external DB (Supabase free tier works great) is the right long-term move.
Thanks a lot for the solution! It works perfectly and really helps me keep my workflow clean.
On a side note, I’m becoming really fascinated by this world of AI and automation. I’ve always loved tech, but now I’m seriously considering pursuing this professionally. Since you clearly know your way around n8n, I wanted to ask: How did you learn all of this?
Do you recommend any specific courses or projects/automations for a beginner to practice? Currently, I try to build things on my own with AI for guidance, but I’d love to know what the best learning path is to master these tools. Thanks again for your time and help!
@ElGarretGG Glad the solution worked for you!
Honestly, the best way to learn is through real templates and solving simple problems first - that’s where you’ll gain the most practical experience.
You can check out my Creator page for lots of free templates. I’ve explained how they work in detail and tried to make them easy to follow, hope they help you out!
