Hello everyone! I have a bot that collects a client database by adding data to a Google Sheet. Then I created a workflow where this Google Sheet is the trigger: the file is downloaded, data is extracted from it, and then loaded into a vector store. The main agent, which directly communicates with the clients, uses this vector store to determine whether the client is new or already exists in our database.
However, there is a problem: each time the trigger fires, the entire spreadsheet is downloaded again and completely loaded into the database, causing duplicate entries.
Is there a way to only add new rows or remove duplicates from the vector store?
I tried using the Limit node, but it didn’t help because it’s still just one file being downloaded, and the information inside that file is updated.
I also tried adding another Google Sheets trigger so that, upon updating the file, old entries could be deleted from the vector store using a Superbase Delete node. But that didn’t work either, since those records don’t get assigned an ID in Superbase.
Could someone suggest a solution?
