Clear last execution's data from deduplication filter after error?

Questions
1

Describe the problem/error/question

I have a workflow that keeps a list of RSS URLs in a persistent dedupe filter (that way I’m never shown the same thing twice). This works great, except if the workflow errors out downstream from the filter, the next time I run it, the URLs are already in the dedupe filter, so they don’t make it through.

So, is there a way to remove the last execution’s worth of data from the deduplication filter’s persistent storage? If I disable the entire filter, ALL the posts make it through. I’d really like to just rewind the filter a few days.

Thanks!

2

I don’t think there’s a native solution for this case, Once the items pass through the “Remove Duplicates” node, their job is basically done, even if an error occurs afterward..

The only available option in this node is to clear all history, You could maybe use that in an error workflow, when an error occurs clear the history, but this would still allow some duplicates to pass through..

You’ll probably need to implement your own remove duplicates logic using data tables.