Chunking Strategy for Production RAG

We spent two weeks tuning prompts to fix hallucinations before realizing the LLM wasn't the problem — it was faithfully generating answers from the wrong chunks. The most common root cause of production RAG failures isn't the model. It's what you feed it.

This post is a deep dive from our production RAG case study, where we built a document intelligence system that validates property title cleanliness across over a thousand properties in six Indian metros. The chunking problems described here nearly derailed the project.

The Symptom: Confident Wrong Answers

Here's what production RAG failure looks like from the outside: the system returns a fluent, well-structured, completely wrong answer. The user sees "the AI is hallucinating." The engineer opens the prompt, tweaks the system message, adds "only use information from the provided context," deploys, and the same wrong answers keep coming.

The disconnect is that most teams debug generation when the problem is upstream in retrieval. The retrieval layer decides what context the model sees. If you feed the model the wrong chunks, it will generate a wrong answer. Confidently, fluently, and with enough plausibility that nobody catches it until someone checks against the source document.

We hit this wall processing tens of thousands of Indian property documents (sale deeds, encumbrance certificates, revenue records, mortgage deeds) across six languages. Our extraction pipeline was pulling wrong values. Our chain-of-title analysis was flagging false issues. Every prompt we rewrote produced a differently-worded version of the same wrong answer, which in hindsight was the tell: the model was doing its job on bad input.

Why Sentence-Boundary Chunking Broke Immediately

We started with sentence-boundary detection. It's a reasonable baseline. Most RAG tutorials recommend it, and it works well enough on narrative text like blog posts, documentation, or support articles.

Indian legal documents broke it on the first batch.

Sentence boundaries don't map to semantic boundaries in legal drafting. A single sentence in a sale deed can span half a page. A typical conveyance clause reads something like: "The Vendor hereby conveys, transfers, and assigns unto the Purchaser, ALL THAT piece and parcel of land bearing Survey Number 45/3, Block B, situated at..." and continues for 200+ words through boundary descriptions, measurements, conditions, and exceptions, all within a single grammatical sentence. Splitting this at sentence boundaries meant the property description was torn apart across chunks, with the survey number in one chunk and the boundary description in another.

Meanwhile, a critical clause might be a three-word numbered sub-item ("Subject to Clause 9.4") with no sentence terminator. It's semantically complete but syntactically invisible to a sentence-boundary detector.

This isn't unique to Indian legal documents. Any structured document (contracts, financial filings, technical specifications, regulatory submissions) has semantic boundaries that don't align with sentence boundaries. If your documents have numbered sections, cross-references, or tabular data, sentence-level chunking will fail.

Clause-Level Detection: The First Real Fix

We moved to clause-level detection, parsing the numbering schemes that structure legal documents (Section 4.2.1(a)(iii), Schedule A, Part II) and splitting at clause boundaries instead of sentence boundaries. This kept semantically complete units together: a full conveyance clause, a complete condition precedent, an entire schedule description.

For Indian property documents specifically, we had to handle multiple numbering conventions: the British-style hierarchical numbering (1, 1.1, 1.1.1) common in sale deeds, the Roman numeral sections in older documents, the tabular format of revenue records, and the entirely different structures of encumbrance certificates across states. An encumbrance certificate from Karnataka looks nothing like one from Maharashtra. Different layouts, different field structures, different conventions.

Clause detection solved the most obvious problem. But it revealed three deeper ones that were far more damaging.

Failure Mode 1: Table Headers Lost Across Pages

Property tax payment schedules are tables that span multiple pages. Headers (columns like "Assessment Year," "Property Tax Amount," "Date of Payment," "Receipt Number") appear on page 1. The data rows continue through pages 2, 3, and 4.

With our clause-level chunking, each page became its own chunk. Pages 2-4 had rows of numbers with no column labels. The chunk was syntactically valid text. It embedded fine. But it was meaningless without context.

What the user asked: "What was the property tax paid for assessment year 2018-19?"

What the system retrieved: A chunk containing the row 2018-19 | 14,250 | 18-Mar-2019 | RCT/2019/4521, but without headers, the system couldn't distinguish the tax amount from the receipt number. The LLM guessed. Sometimes it got lucky. Often it didn't.

What the user saw: A confident, specific answer citing the correct document, but with the receipt number where the tax amount should be.

This pattern appears everywhere structured data spans pages: financial statements, compliance tables, loan amortization schedules, inventory lists. Any multi-page table is a chunking landmine.

The fix: Tables detected as single chunks with headers repeated at the top of every page-continuation chunk. We used AWS Textract's table extraction to identify table boundaries, then ensured headers were prepended to every table-continuation chunk regardless of page breaks. This added roughly 5-10% to chunk sizes for table-heavy documents but eliminated the header-loss problem entirely.

Failure Mode 2: Cross-References Became Dead Links

Indian sale deeds are full of internal cross-references: "Insurance obligations as per Clause 9.4," "The property more particularly described in Schedule A hereto," "Subject to the conditions set forth in Part III below."

With clause-level chunking, "Insurance obligations as per Clause 9.4" landed in one chunk. Clause 9.4 itself lived in a completely different chunk. When the system retrieved the first chunk, it had a reference to information it couldn't access. The LLM either hallucinated the content of Clause 9.4 or returned a vague non-answer.

This was especially damaging for extraction schemas. Our sale deed schema asked the LLM to extract property descriptions, but in many documents, the property description is in Schedule A, referenced from the main body by "the property more particularly described in Schedule A hereto." If the Schedule A chunk wasn't retrieved alongside the main body chunk, the extraction returned empty or fabricated values.

Early schemas broke specifically on property descriptions split across schedules. The sale deed schema alone went through 15-20 iterations before extraction consistency stabilized, and cross-reference handling drove most of those iterations.

The fix: Cross-reference metadata attached to chunks. When the parser detected a cross-reference pattern (Clause X, Schedule Y, Part Z, Annexure N), it recorded the reference target as chunk metadata. At retrieval time, if a chunk with outbound cross-references was selected, the system automatically retrieved the referenced chunks alongside it. This wasn't a RAG "trick." It was a document-graph relationship built into the chunking layer.

Failure Mode 3: OCR Garbage Fed to the Embedding Model

This one was insidious because it looked like the system was working.

Sub-registrar stamps, handwritten boundary descriptions, and faded typewritten text from 1990s-era documents produced OCR output with confidence scores below 60%. The text was garbled: wrong characters, missing words, invented punctuation. But it was still text. It chunked normally. It embedded into the vector space. It showed up in search results.

What happened: The system would retrieve a chunk that was 40% garbage OCR from a registrar's stamp mixed with 60% legitimate document text. The LLM, seeing what looked like context, would extract values from the garbage portion: a misread boundary measurement, a garbled party name, a hallucinated registration number that was actually an OCR error on a stamp date.

Why it was hard to catch: The retrieved chunk was from the correct document. The citation was accurate (it really was from page 3 of the 2005 sale deed). The answer looked legitimate. Only checking against the original scan revealed the extraction was from a misread stamp, not from the deed text.

Older properties from the 1980s-90s had 3-4x the OCR error rate of post-2010 properties. In our pipeline, roughly 12-13% of all pages ultimately needed human review: handwritten sale deeds from the 1970s-80s, damaged or faded documents, registrar stamps misread as deed content.

The fix: OCR confidence scoring at the chunk level, not just the page level. Each chunk received a confidence score based on the OCR confidence of its constituent text. Chunks below the threshold were flagged for human review rather than being fed to the embedding model. This meant some content was missing from the vector store until a human reviewer transcribed it, but the content that was in the store was reliable.

The tradeoff was real: accepting gaps in coverage to avoid poisoning the retrieval pipeline with garbage. In a legal context, "I don't have confident data for this section" is vastly better than "here's a confidently wrong answer."

The Extraction Schema Interaction

Chunking doesn't exist in isolation. It directly impacts the LLM extraction pipeline that runs on top of retrieved content.

Our extraction pipeline was structured: classify the document type, select the matching JSON schema, extract against that schema, validate against its constraints, then flag low-confidence fields. Each document type had its own schema: sale deeds (parties, consideration, property description, conditions, encumbrances), encumbrance certificates (registration numbers, entries, time period), mortgage deeds (lender, borrower, amount, security, conditions), and so on.

State-specific variants added complexity. For sale deeds, the majority of fields were universal (parties, consideration, dates, and basic property description are structurally similar across Indian states), with state-specific overlays for the different standard clauses, stamp duty structures, and registration endorsement formats between Karnataka and Maharashtra. Revenue records were even more fragmented: a Karnataka RTC extract and a Maharashtra 7/12 extract don't share a layout, a field structure, or even a conceptual model.

Bad chunking amplified schema failures. When a multi-party deed (seller, buyer, confirming party, witness, power-of-attorney holder) was split across chunks, the schema couldn't determine party roles because the relationship context was in a different chunk. When conditional consideration structures ("₹50 lakhs upon execution, balance ₹25 lakhs upon registration") were split, the extraction returned partial amounts.

The schemas stabilized after processing 300-400 documents with a lawyer feedback loop: extract, lawyer reviews, incorrect extractions fed back as training signal for schema refinement. A notable share of extraction errors that surfaced in this feedback loop were traceable to chunking problems, not prompt or schema problems.

Measuring Retrieval Quality: You Can't Fix What You Can't Measure

Without measurement, chunking strategy is a vibes exercise. We tracked several metrics that made chunking quality visible:

Cross-encoder re-ranker scores. The median re-ranker score per query gave us a running measure of retrieval relevance. When we changed chunking strategies, this metric moved immediately. Sentence-boundary to clause-level detection: +18% median re-ranker score. Adding table-header repetition: +7% on queries involving tabular data. Adding cross-reference metadata: +12% on queries involving cross-referenced clauses.

Extraction accuracy on sampled comparisons. Weekly comparison of system-extracted values against lawyer-verified values. After the full chunking improvements, the extraction error rate on parties dropped from roughly 8% to 3%. Property description accuracy improved from roughly 72% to 89%.

OCR confidence distribution. The percentage of chunks falling below the confidence threshold. This metric directly measured how much potentially garbage content was in the vector store.

Flag overturn rate. How often lawyers overturned the system's title defect flags. This was the downstream quality metric. A rising overturn rate meant something upstream (usually chunking or OCR) was degrading.

The observability stack (CloudWatch for infrastructure, Langfuse for AI-specific traces and cost tracking) let us correlate chunking changes with downstream quality changes within hours, not weeks.

Re-Chunking a Live System

If you've already deployed with a chunking strategy that's producing bad results, here's what worked for us:

We migrated chunking on a live system without downtime. The key was keeping the old index live alongside the new one, routing a percentage of queries to the new index, and validating retrieval quality before cutting over. We started with the most problematic document types (revenue records with tables, old sale deeds with cross-references) and expanded incrementally, with a per-document-type rollback path if regressions appeared.

The full re-chunking from sentence boundaries to the final strategy took about two weeks of engineering time, not because the parsing was complex, but because validating that retrieval quality improved across all document types and query patterns required systematic testing.

When Chunking Isn't the Problem

Not every RAG failure is a chunking failure. Before you rebuild your chunking pipeline, rule out these other causes:

Embedding model mismatch. If your documents use domain-specific terminology (Indian legal terms like "khata," "mutation," "encumbrance"), a general-purpose embedding model may place these terms in the wrong region of vector space. This looks like a chunking problem (the right document isn't being retrieved) but it's an embedding problem.

Missing hybrid search. If your queries include exact identifiers (survey numbers, registration numbers, case numbers, account numbers) pure semantic search will fail regardless of chunking quality. These identifiers don't embed meaningfully. You need keyword search (BM25) alongside semantic search.

Retrieval window too narrow. If you're retrieving top-3 chunks and the answer requires context from 5-6 chunks across multiple documents, the problem isn't chunking. It's that you're not retrieving enough context.

The debug hierarchy we learned: when quality drops, investigate data preparation first (OCR, chunking), then retrieval (search, re-ranking), then generation (prompts, model choice) last. 60% of our debugging time was spent upstream of the LLM. Most teams do it in reverse: they tweak prompts for weeks before realizing the model never had the right context to begin with.

Start at the retrieval layer: log what chunks the system retrieves for failing queries. Read them. Are they the right chunks? If yes, the problem is downstream (prompt, model). If no, the problem is upstream (chunking, embedding, search). Fix one layer at a time, measure after each change, and resist the urge to change everything at once.

Chunking Strategy for Production RAG: What We Changed After Real Documents Broke Ours