Hey guys,
Hope you are well! I have an issue with uploading my documents to a pinecone vector store. The issue is that it grabs the 13 files in my google drive folder but then only loops through 4 items and then when I go into the pinecone index I have 10 vectors? Could anyone explain?
Hi there
can you provide the image of the execution? the one that shows how many items are from each node
it will definitely help to identify your issue
the way vector works is that from your file, they split it into a more smaller chunks, hence why you have the default data loaded and the recursive char text splitter, their job is to break your file into smaller chunks and then embed it using openai, then store it in pinecone
so depending on how your text splitter work, it make sense if you got more vectors then your file quantity, because your file got broken down into more smaller manageable pieces
So in this run I grabbed 76 items from Drive but it only looped through 4 and added 10 vectors to pinecone
hi, just to confirm, your loop over items batch size is 1?
Here is the automation…
Yeah for sure, however Im getting 4 items in the default data loader and I want 76 items in it. It would make sense if I got 76+ items looped but in this case its way less
Im seriously lost as to why this is not working does anyone have any clue?
Same issue happening to me, could you find any solution ?
