Why your RAG pipeline fails on tables — and the chunking fix

Every RAG project we've been called in to rescue has had the same shape of bug report: "it works on most questions but it's confidently wrong on the ones that matter." When we go looking, the failures cluster. And more often than not they cluster on documents containing tables.

Here's why, and what to do about it.

The failure, concretely

Take a clinical dosing table — four columns, twelve rows, sitting halfway down page 6 of a protocol document. The header row says Age band, Weight, Dose, Max daily. The rows underneath are numbers.

Now run it through the default chunker almost everyone starts with: split every 1,000 characters, overlap 200. The table is 1,400 characters. So it becomes two chunks. Chunk A gets the header and the first seven rows. Chunk B gets the last five rows and, thanks to the overlap, a fragment of row seven.

Chunk B, in its entirety, is now something like this:

| 9–11  | 30–38 kg | 250 mg | 750 mg |
| 12–14 | 38–50 kg | 375 mg | 1125 mg |
| 15+   | 50+ kg   | 500 mg | 1500 mg |

Review after 48 hours if symptoms persist.

There is no indication anywhere in that chunk of what the numbers are, which drug they refer to, or that the whole table applies only to patients under sixteen. The information that made those numbers meaningful is in a different chunk, which may not be retrieved.

Two things then go wrong, and it's worth separating them because they have different fixes.

1. Retrieval can't find it

Embed that chunk and you get a vector that means, roughly, "a table of numbers about milligrams". A query like "what's the paediatric dose of amoxicillin for a 40kg child" won't be near it, because the words "paediatric" and "amoxicillin" appear nowhere in the chunk. The relationship that would have made this findable was in the header, and the header is elsewhere.

2. If it is retrieved, generation still goes wrong

Suppose your retriever gets lucky — the query mentioned "375 mg" and lexical search hit it. Now the model has a table fragment with no header. A well-behaved model will say it can't tell what these numbers mean. Most models, given a helpful-assistant system prompt and a plausible-looking table, will infer column meanings from position and answer anyway.

The model isn't hallucinating. It's doing exactly what you'd do if someone handed you a fragment of a table and asked what the third column meant. The bug is upstream.

Why the usual fixes don't fix it

Three things people reach for first, and why each disappoints:

Bigger chunks. Going from 1,000 to 4,000 characters means fewer tables get split, and it feels like an improvement. But now every chunk contains four thousand characters of mostly-irrelevant surrounding text, so its embedding is an average of several topics and matches nothing precisely. You've traded a sharp failure for a diffuse one, which is harder to notice and harder to debug.

More overlap. Overlap of 200 characters doesn't help when the header is 1,200 characters from the rows that need it. Pushing overlap high enough to fix tables means duplicating most of your corpus, which inflates your index and puts near-identical chunks in competition with each other during retrieval.

A better embedding model. No embedding model can recover information that isn't in the text it was given. This is a data preparation problem wearing an ML costume.

The fix: chunk along structure, then repair meaning

Three changes, in order of how much they'll move your numbers.

Parse structure before you chunk anything

Get the document into a structured representation first — headings, paragraphs, tables, lists — using something like Unstructured, Docling, or the PDF library of your choice. Then chunk within those boundaries rather than across them. A table is an atomic unit. A list is an atomic unit. A section heading belongs with the content beneath it.

The rule we use: a table is never split. If a table is larger than the chunk budget, it stays whole and becomes an oversized chunk. An oversized chunk is a minor cost; a meaningless one is a correctness bug.

Prepend the heading path to the embedded text

This is the cheapest change on the list and it consistently gives the largest single improvement. Every chunk carries the chain of headings above it, and that chain goes into the text you embed — not merely into metadata sitting next to it.

@dataclass(frozen=True)
class Chunk:
    text: str
    heading_path: tuple[str, ...]   # ("Paediatric", "Respiratory", "Dosing")
    kind: Literal["prose", "table", "list"]
    doc_title: str
    page: int

    def for_embedding(self) -> str:
        """What actually gets vectorised.

        Metadata that only lives in a metadata column cannot influence
        similarity. If you want the retriever to know this fragment is
        about paediatric respiratory dosing, the words have to be in
        the embedded string.
        """
        context = " > ".join((self.doc_title, *self.heading_path))
        return f"{context}\n\n{self.text}"

Chunk B from earlier now embeds as Protocol 14: Amoxicillin > Paediatric > Dosing by weight followed by the rows. The paediatric query now lands close to it.

Give every table a generated summary, and embed that

A table's raw text is a bad thing to embed even when it's complete. Pipe characters and digits carry little semantic signal, and a table of financial results and a table of dosages look more similar to an embedding model than either does to a paragraph describing it.

So at ingest time, generate a one- or two-sentence description of each table and embed that, while storing the full table as the payload retrieved and passed to the model. You search over the description; the model reads the real thing.

TABLE_SUMMARY_PROMPT = """\
Describe what this table contains in one or two sentences.
State what the rows represent, what the columns measure, the units,
and any population or period the table is restricted to.
Do not restate individual values.

Document: {doc_title}
Section: {heading_path}

{table_markdown}
"""

async def build_table_chunk(table: ParsedTable, ctx: DocContext) -> Chunk:
    summary = await llm.complete(
        TABLE_SUMMARY_PROMPT.format(
            doc_title=ctx.title,
            heading_path=" > ".join(ctx.heading_path),
            table_markdown=table.to_markdown(),
        ),
        max_tokens=120,
    )
    return Chunk(
        # Embedded: the summary. Retrieved and shown to the model: the table.
        text=table.to_markdown(),
        embed_text=f"{ctx.title} > {' > '.join(ctx.heading_path)}\n\n{summary}",
        kind="table",
        heading_path=ctx.heading_path,
        doc_title=ctx.title,
        page=table.page,
    )

This costs one cheap model call per table at ingest, once. On a 12,000-document corpus with around 9,000 tables that was a few dollars and a couple of hours of wall clock.

What it was worth

Below is a worked example of how the four changes typically stack up on a table-heavy corpus, measured as recall@5 — the share of questions where the passage containing the answer appears in the top five results. Treat the figures as an illustration of the ordering and rough magnitude of each change, not as a benchmark of your corpus. Run your own eval set; that is the entire point of the last section.

Note where the movement is. Overall recall improves respectably. Recall on table-answerable questions roughly doubles. If your corpus is mostly prose with occasional tables, the headline number will understate how much this matters to the subset of users asking the questions tables answer — and those are frequently the highest-value questions in the system.

The part people skip

You cannot do any of this without an evaluation set. Every number in that table exists because there were 400 labelled questions with known correct source passages to measure against.

Without that, "we improved chunking" is a claim, and you'll find out whether it was true when a user reports something wrong. Build the eval set first. It's two weeks of unglamorous work that makes every subsequent decision checkable, and it's the difference between engineering and redecorating.

Quick checklist

• Parse to structure before chunking; never split a table or a list
• Put the heading path into the embedded string, not just a metadata column
• Embed a generated summary for tables; retrieve and show the real table
• Keep a kind field on every chunk so you can measure table questions separately
• Measure recall@k on a labelled set before and after every change