[Challenge] Why LLMs hallucinate on grid extraction and how we parsed a handwritten scorecard in n8n

[easybits]

Hey n8n Community,

Document extraction pipelines handle invoices and CVs beautifully. But during a game night, we fed our AI something that looked simple but completely broke it: a handwritten Kniffel (Yahtzee) scorecard.

The grid trap

The goal: Take a photo, find players with a handwritten asterisk (*) above their name, extract their scores, calculate the 35-point bonus (if top $\ge$ 63), total it up, and push to Sheets and Telegram.

Look at the attached photo:

It’s obvious to humans. But the LLM hallucinated names (“Ivan” became “ban”), merged columns, and botched the math.

Here is why spatial data breaks AI, and the prompt architecture we used to fix it:

Breaking the “Left-to-Right” bias

LLMs read like a book (left-to-right, row-by-row). A scorecard demands the opposite. To stop “row bleeding” across columns, we forced an Array of Objects output ([{player_name, sum_top, sum_bottom}]). This forces the AI to finish reading one vertical column before moving to the next.

Deterministic math, zero LLM judgment

We already know the Extractor we are using is incredibly reliable at classifying documents and pulling structured data, so we decided to push its limits and test if it could handle calculations, too. Big mistake. LLMs are text-prediction engines, not calculators. The fix: We instructed it to only extract raw numbers, and used a pure JavaScript Code node downstream to handle the math and conditional bonus.

Explicit empty states stop hallucinations

Players wrote dashes (-) for empty boxes. Because the LLM couldn’t fit a string dash into a Number schema, it panicked and outputted -1, breaking our JavaScript. Adding a strict constraint—“CRITICAL: If a cell is a dash or empty, you MUST output 0”, fixed the pipeline instantly.

Anchoring the eyes with labels

To stop the AI from drifting across columns, we gave it visual coordinates. We instructed it to look at the printed labels on the far left (e.g., “Dreierpasch”), trace an invisible horizontal line to the starred column, and extract only that cell.

One honest caveat

Offloading math to JS guarantees perfect logic, but handwriting recognition still depends on photo quality. If you need flawless data from messy writing, human-in-the-loop validation is still the ceiling.

How to run it

I used the easybits Extractor because it handles JSON-schema enforcement natively without wrestling with HTTP node prompts. Both cloud and self-hosted fit in the free tier.

• n8n Cloud: Search easybits in the nodes panel.

• Self-hosted: Install @easybits/n8n-nodes-extractor from Community Nodes.

Feedback welcome

I’ve attached the raw image. I’m highly curious: what happens when you run this exact image through your current OCR or LLM setup? Have you found a cleaner way to extract vertical columns from dense horizontal grids?

Best,
Felix

[nguyenthieutoan]

The math-offloading approach is spot on - keeping all arithmetic in a Code node instead of trusting the LLM makes the pipeline deterministic regardless of model. The column-first JSON schema constraint ([{player_name, sum_top, sum_bottom}]) is a clean way to fight row bleeding.

One thing worth trying if accuracy still slips on poor-quality photos: crop each column as a separate image region before passing to the vision model. Even a rough coordinate-based crop in a Code node (using the image dimensions you know from the scorecard layout) can dramatically reduce positional hallucinations by giving the model a simpler, single-column context.

[easybits]

Hey @nguyenthieutoan, thank you for the feedback!

The idea of cropping each column into a separate image is definitely an interesting one. My only concern is that it would require the photo to be taken with fairly consistent framing every time. Otherwise, there’s a risk of cutting off important information during the cropping process.

That said, it’s a great suggestion, and I’ll definitely give it a try to see how practical it is in a real-world setup. Thanks for sharing the idea!

[nguyenthieutoan]

The framing concern is valid, but you can work around it by adding a generous overlap to each crop - say 10-15% buffer on both sides of each column boundary rather than cutting at the exact edge. The LLM will still focus on the correct column because your schema + prompt locks it to one player at a time, and a bit of overlap is far less damaging than cutting into the numbers. For printed scorecards like Kniffel where the column headers are fixed-width, you can also derive the crop boundaries once from a reference image and reuse the same pixel offsets reliably across all photos taken in similar conditions (same table, same phone distance).

[easybits]

Hey @nguyenthieutoan, sounds valid! I’ll definitely give this approach a try.

It’s just funny to see how capable AI can be in general, but then a single edge-case idea comes along and suddenly you realize it can break a lot of solutions. I actually really enjoy discovering those kinds of situations, they tend to be the most insightful ones.