Multimodal RAG: When the Answer Lives in the Table

Run a spec sheet through a text extractor and read the output. The numbers survive. The column headers that told you what those numbers meant do not. Meaning in real documents lives in structure — the row a value sits in, the label positioned next to an arrow on a diagram, the caption under a screenshot, the footnote that qualifies the whole table.

Flattening a PDF into a bag of sentences is fast and cheap, and it discards precisely the evidence a user needs to trust an answer.

Preserve structure at ingestion, because you cannot recover it later

Every downstream capability depends on decisions made during parsing. Keep page boundaries, heading hierarchy, coordinates, table relationships, image references, and document metadata alongside the text. Generate representations that support both semantic search and exact keyword matching — part numbers, statute references, and error codes need literal matching, and embeddings alone will lose them.

Above all, keep a path back to the original page. If you cannot show the user where an answer came from, they have to take it on faith, and in most enterprise contexts they will not.

• Use layout-aware parsing for multi-column pages, tables, forms, and footnotes
• Create image or page-level representations when the visual carries the meaning
• Attach source, version, page, and section metadata to every retrievable unit
• Rerank candidates against the full question, not just the embedded fragment
• Return a citation that opens the exact page or region the answer used

Retrieve evidence bundles, not chunks

A question about a chart may need three things at once: the chart itself, its legend, and the paragraph describing how the data was collected. Systems that return a single best-matching chunk will find one of the three and answer confidently from a third of the picture.

Design retrieval to assemble a coherent bundle — the primary hit plus its structural neighbors. Page context is usually the right unit for documents that were laid out for human reading, because whoever formatted the page already grouped the related material for you.

Version-aware retrieval prevents a whole class of quiet errors

Policy manuals, contracts, and specs get revised, and the old version usually stays in the corpus. A retriever that has no concept of recency or supersession will happily cite a rate table from two years ago with total confidence. Store effective dates and version numbers, and filter on them before the model ever sees a candidate.

Evaluate by document type, not in aggregate

Invoices, manuals, research reports, slide decks, and dashboard exports fail in completely different ways. An aggregate accuracy number averages those failures into something that looks acceptable and tells you nothing about what to fix. Build a retrieval test per source class, and score citation accuracy — whether the cited page actually supports the claim — not just answer similarity.