Describe the problem/error/question
Here is my use case. I’m crawling a large website and storing all the data in a vector db (pinecone). I then want to utilize an Open AI agent on the dataset. For purposes of this example, let’s assume I want to ask the Open AI agent to create certain technical how-to steps based on reading the dataset.
If I utilize a an agent with the pinecone db as a tool, it works, except for the size limitation:
The challenge is I can only pass a certain amount of data to the open AI agent.
In the vector store node, I specify a limit of results to return. If you make that number to high, it exceeds the amount of data you can pass to the open AI agent.
What is the error message (if any)?
Too much data is passed to the open AI agent node if I set the limit too high in the pinecone vector node.
If I don’t set a high limit, then the open AI agent doesn’t process the entire data set.
How would one be able to instruct the openAI agent to iterate through the dataset? So I can pass the max limit in every iteration, ensuring that the agent goes through the entire dataset to complete its task? Right now it’s a single execution based on the limit of data sent from the vector store node.