Summarizing and Q&A AI agent

Questions
1

Hello,

We are trying to create an AI agent that can provide a summary of a book that we upload and provide Q&A. But we found that it’s not working to the level that we’re expecting that it cannot do a summary and the answers it returns seems to be only partial.

Here’s more details about our workflows:

Objective

Upload any document (100 pages or more) and have the AI workflows to learn about the book so that it’s able to summarize the book and answer questions related to the book content.

Workflow 1:

  1. Download the document from Google Drive (https://drive.google.com/file/d/1iAi-bCNpde2BdoEhNzZ3gTW038wubauI/view?usp=sharing)
  2. Using Google Gemini to process the document and upload to Pinecone Vector Store
  3. Recursive Character Text Splitter Settings: Chunk Size: 500 and Chunk Overlap: 20

Workflow 2:

  1. When asked a question, using Google Gemini to process the question
  2. Then look up relevant information from the Pinecone Vector Store and provide an answer

Steps:

  1. Upload Dot Com Profits (https://drive.google.com/file/d/1iAi-bCNpde2BdoEhNzZ3gTW038wubauI/view?usp=sharing) in Workflow 1
  2. Ask questions to the chat agent in Workflow 2 related to the document uploaded.

Problems:

  1. When asked to provide a summary of the book, the agent appears to just trying to find if a summary is already provided in the document. It doesn’t generate a summary based on the book and responded that “no summary was found”.
  2. When asked what are the steps of generating wealth online, the agent is not able to describe the steps (chapters) mentioned in the book. Rather, it responds that the book mentioned about creating multiple income streams, but not the specific steps which can be found in the book.
  3. When asked “how many steps are there?”, the agent is able to return that there are 8 steps. But when asked “what are the eight steps?”, it’s not able to list it.

Questions:

  1. It appears that the agent only attempts to find information that’s explicitly provided and it’s not able to aggregate information or provide a summary. Are there any extra processing required to prepare the document to able to do that?
  2. Are the settings of uploading the document able to allow the agent to process the document properly? Is it better to have larger chunk size and chunk overlaps to allow continuous context?
  3. Sometimes, the Q&A agent seems to have the correct answer like when asked “how do I sell a high-priced product?”, it responded “The provided text states that selling high-priced products depends on perceived value. Effectively communicating the value of your product, including the time and money invested in its creation, and highlighting your knowledge and experience are key. A well-built marketing funnel is also important.?”
  4. What can be the reasons that it can get some answers right, but generally, others incorrectly?
2

Hey @Elizabeth_Knutzen,

Thank you for sharing everything in detail!

You’ve done things correctly, but when it comes to RAG , there are a few more things we need to look at… like:

  • Chunking strategy
  • embeddings
  • Whether we are chunking the text properly or not… etc.

The reason your agent answers some questions correctly and misses others is mostly because of the quality of chunks it’s using.

and yes there is a lot of things to discuss, but I can’t share it all here…

So I am sharing some really helpful YouTube videos that explain these concepts in a simple and clear way. They’ll help you build a strong and smart agent that can answer even the most complex questions from the book.

Here’s the list of those YouTube videos:

I have myself watched all these videos and learned a lot from them, so I truly believe they’ll help you too. and you’ll get the answer of all your questions here.

Happy Flowgramming : )

1 Like