hi, here you can see my current RAG workflow.
this is so get a huge amount of sharepoint data in supabase for the first time.
problem:
- i want to add a ‘file id check’ to prevent duplicates. in case the flow fails and i wouldn’t be able to see what was already added. but i can’t figure out how to intelligently do this.
what I tried:
- i went with the supabase ‘get rows’ node before the 'Set File Information
node, filtered by the file_id in the metadata, and set up a filter that if it exists, the file should not be included. but when i did that, the rest of the workflow used the supabase info, not the info we actually need for extraction etc.
- i experimented with the merge node but that was just messy and didn’t help
basically i just want to use it as a criterium, but currently in replaces the actual data
how would you guys do this? thank you
