I’m facing problems whereby many of my docs (in PDF) contains tables and columns. Hence my chatbot couldn’t retrieve the correct answers. Only truncated answers are given. I need to improve the accuracy of the extraction of data to “flatten” it out. As I’m not IT trained, with no coding or programming knowledge, I’ve relied on Manus to teach me step-by-step on what to add/edit the nodes. But it’s a spiral with more and more repeating steps and new nodes, but not addressing the problem.
Hi @Landy_Phang Welcome!
LlamaParse is already returning markdown, so the tables leave the PDF intact. They break at the Recursive Character Text Splitter. With no split option set it falls back to splitting on paragraphs, newlines and spaces, so a table gets cut mid-row and the chunk stored in Pinecone loses its header row, which is why the answers come back partial.
Open the Recursive Character Text Splitter node, click “Add Option” and set “Split Code” to “Markdown”. Set “Chunk Size” to 2000 and “Chunk Overlap” to 200 so a full table fits inside one chunk.
Clear the Pinecone index and re-run the ingestion after that, the vectors already in there were embedded with the old chunking and will not update on their own.
Anshul’s chunking point is the part I’d fix first.
One extra thing I’d add: after you change the splitter settings and rebuild the Pinecone index, test with one PDF/table where you already know the correct answer. That gives you a clean before/after check instead of changing five nodes at once and guessing what helped.
For tables, I’d also try to preserve row and header context as much as possible. A table row without its column names is easy for the chatbot to misunderstand later.
A simple test path would be:
1. Pick one PDF with a known table answer
2. Re-ingest it after changing the markdown chunking settings
3. Ask one specific question where the answer lives in the table
4. Check whether the answer includes the right row and column context
5. Only then scale it across the rest of the documents
That keeps the workflow beginner-friendly and avoids the spiral of adding more nodes before you know where the extraction is breaking.