Guardrails for Production AI: Citation Checking, Confidence Thresholds, and Human Escalation
Part of Anatomy of a Production RAG System: AI-Powered Title Validation for Indian Real Estate
The model cited the right page, quoted a real section, and returned a value that was close but wrong - "Suresh Patil" instead of "Suresh Patel." Prompt instructions caught 97% of cases. In legal title validation, the other 3% is where liability lives.
This post is a deep dive from our production RAG case study, where we built a document intelligence system that validates property title cleanliness across over a thousand properties in six Indian metros. The guardrails described here were built after we realized that prompt-level instructions ("always cite your sources," "never fabricate information") were failing silently on roughly 2-3% of extractions.
Why Prompt-Level Guardrails Are Not Enough
Our system prompt said: "Only extract information that appears in the provided document. Always cite the specific page and section. If the information is not present, return null."
It worked on 97-98% of extractions. On tens of thousands of documents with multiple extraction passes each, that remaining 2-3% meant hundreds of incorrect extractions. In a legal context where a single wrong value can invalidate a title opinion, that's not acceptable.
The failures were subtle. Nothing came back as obviously fabricated text. A real page from the real document, cited correctly, with one value quietly off: "Suresh Patil" for "Suresh Patel." "₹45,00,000" for "₹54,00,000". A registration date of "12.03.2015" when the deed said "12.03.2016." Plausible, cited, wrong. The LLM had "corrected" the surname based on regional naming patterns in its training data.
We stopped treating prompt instructions as reliable controls after that.
Citation Verification: Does the Answer Match the Source?
Every extraction included a citation: chunk ID, page number, and the text span the value came from. This wasn't just for the lawyer - it was the input to an automated verification step.
After the LLM extracts a value (say, "Consideration: ₹45,00,000"), the system checks whether that string (or a normalized variant) appears in the cited chunk. Numeric fields are normalized (removing commas, converting lakh/crore notation (Indian number grouping: 1 lakh = 100,000; 1 crore = 10 million), handling "Rs." vs "₹"). Party names are checked verbatim.
Where deterministic matching worked and where it didn't. Exact string matching (with normalization) resolved citation verification for roughly 60-65% of extractions cleanly - party names and dates, primarily. The remaining 35-40% required progressively fuzzier approaches or fell back to human review entirely:
OCR noise pushed consideration amounts into the fuzzy tier. Textract occasionally misreads characters plausibly: 0/O, 1/I, 5/S. A deed says "₹54,00,000" but the OCR'd chunk contains "₹S4,00,000." The extraction is correct (the LLM interpreted context), but citation match fails. We added OCR-aware fuzzy matching for numeric fields: try common substitution patterns before flagging.
Multi-line values and table extraction broke matching on property descriptions and tax amounts. Descriptions span multiple lines with inconsistent OCR breaks. Table values (tax receipts, payment schedules) exist within pipe- or space-separated rows, not as standalone text. Matching "₹14,250" against "2018-19 | 14,250 | 18-Mar-2019" required table-row-aware matching.
Something that broke again after we fixed it: in week 9, a batch of Pune municipal tax receipts used spaces instead of pipes as table separators. The normalization silently failed - extractions that should have been flagged weren't. Caught through weekly extraction accuracy sampling, not through the guardrail itself. Guardrail normalization rules are themselves a maintenance surface that drifts as new document formats enter the pipeline.
For the fields where deterministic matching broke down entirely - property descriptions (multi-line, reformatted by the LLM) and any complex table extraction - we fell back to human review rather than over-engineering fuzzy logic that would itself need guardrails. The boundary between "automatable verification" and "needs a human" was field-type-specific and document-era-specific, and we drew it conservatively.
What citation verification catches: extracted values that aren't in the cited text - the "Suresh Patil" case where the LLM hallucinated a name correction.
What it doesn't catch (the important taxonomy):
- Wrong value from correct chunk. A chunk contains two consideration amounts (one for the subject property, one from a referenced prior transaction). The model extracts the wrong one. Both are in the text. Citation verification passes.
- Semantic ambiguity. "S. Raghavan" appears twice in a deed - once as the seller, once as a witness. The model assigns the wrong role. The name is in the text. Citation verification passes.
- Multi-value confusion in tables. A tax table has amounts for multiple years in one chunk. The model extracts 2019-20's amount when the query asked for 2018-19. Both values are in the text.
- OCR phantom matches. A garbled OCR character coincidentally creates a valid-looking value that matches the extraction - under 0.5% of verifications, but it happens.
These are where confidence thresholds and human review handle the remaining risk.
False positive trajectory: initially 5-7% of flagged extractions were actually correct, failing due to unhandled formatting differences. After tuning normalization (Rs./₹, OCR substitutions, whitespace, table rows), false positives dropped to roughly 2-3% by week 4. At that level, lawyers verified a flagged extraction in seconds by checking the cited page, versus minutes to find an unflagged error.
Confidence Thresholds: When the System Should Say "I Don't Know"
An honest caveat: cross-encoder re-ranker scores are not calibrated probabilities. A score of 0.7 means "more similar than 0.6," not "70% likely relevant." LLM self-assessed confidence was even noisier - the model said "high" on roughly 80% of extractions, including some that were wrong. It's most confident on clear text (where it's also correct) and on ambiguous text (where it guesses confidently). We used both as signals among several, never as sole decision criteria.
How we set thresholds. We used a validation set from schema stabilization. For each field: system extraction, lawyer-verified value, re-ranker score. We plotted accuracy at different score thresholds - not formal ROC analysis, a spreadsheet with scatter plots. But it showed where the accuracy cliffs were: party names had a clear drop-off (error rate roughly tripled below a certain score), consideration amounts had a lower cliff. Property descriptions had no clean cliff because errors there were about chunking quality, not retrieval relevance.
The thresholds: field-specific confidence thresholds calibrated against a validation set, with stricter cutoffs on critical fields (parties, consideration, dates) and looser ones elsewhere. LLM "low" confidence combined with moderate re-ranker scores also triggered a flag. OCR confidence below threshold on the source chunk flagged regardless.
On the validation subset, these caught roughly 85% of extraction errors. The 15% that slipped through were predominantly the "right value from wrong context" failure modes listed above - citation verification passes, confidence is high, but the model chose the wrong value from a multi-value chunk.
The measurement gaps we're honest about. We don't have a clean recall number for the full corpus because ground truth only exists for the validation sample and the extractions lawyers happened to check. We can't know what the lawyers didn't catch. The validation set itself has selection bias: it skewed toward documents processed early (mostly cleaner, more recent properties). Recall on the harder tail of the corpus - older documents, multi-language revenue records - was likely lower, but we don't have the ground truth to measure it. This is not enterprise-grade statistical rigor. It was a team of four on a deadline. The thresholds were good enough to catch most errors and route them to humans. They were not mathematically optimal.
They also weren't stable. Thresholds drifted as document mix changed (modern apartments vs 1990s commercial properties have different score distributions) and when Anthropic updated Claude mid-project. We revalidated roughly every two weeks. Twice we re-tightened thresholds that had been relaxed based on earlier batches that turned out to be unrepresentatively clean.
Human Escalation: The Safety Net
Escalation triggers: citation verification failure, re-ranker below threshold on critical fields, OCR confidence below threshold, LLM "low" confidence with moderate re-ranker, schema validation failure (value doesn't match expected format/range), and cross-document inconsistency flags (area mismatch, party name mismatch between documents).
The handoff mattered more than the trigger. Escalated extractions arrived with: flagged fields highlighted, cited chunk text with the relevant section marked, a link to the original scan, the escalation reason, and the LLM's value alongside raw text. This context meant most escalations resolved in 30-60 seconds. Without it, reviewers searched through multi-page deeds for 5-10 minutes. Handoff quality directly determined review queue throughput.
Escalation rate trajectory (party name extraction): week 1: ~15% (thresholds set aggressively on purpose). Week 4: about 10% (relaxed after validating false positive patterns). Week 6: spiked to 13% when we hit 1990s Hyderabad commercial properties with Telugu-English mixed deeds - thresholds tuned on Bengaluru apartments were too loose. Re-tightened selectively for revenue records (government land ownership registers - the Indian equivalent of county land records in the US or Land Registry extracts in the UK). Week 10: close to 6% (stabilized, ±1-2% weekly variance). Revenue records stayed persistently higher (12% even at week 10) due to multi-language formatting. Post-2010 sale deeds ran under 3%.
Feedback loop. Every human resolution was recorded: system extraction, human correction, which trigger fired. This fed back into threshold calibration - too many false escalations, relax; uncaught errors surfacing in lawyer review, tighten.
Caching Verified Extractions
Once verified (by citation checking or human review), caching extraction results reduced both cost and risk on re-processing. Same extraction query against the same chunk (by hash) → return cached result. This mattered during schema iteration: when we updated the sale deed schema (15-20 iterations), unchanged fields served cached results without new LLM calls.
Cache invalidation: source document re-OCR'd, schema changed for the affected field, human reviewer corrected the value, or model version changed. That last one bit us: when Anthropic updated Claude 3.5 Sonnet mid-project, we kept serving v1 cached extractions alongside v2 new extractions. Formatting differences (the model started including more context in boundary descriptions) made extraction behavior inconsistent across the corpus for a few days. Caught through accuracy monitoring, invalidated affected caches.
We ran periodic revalidation: every two weeks, re-ran citation verification on a random 10% sample of cached extractions using the latest normalization rules. Caught a handful of cached wrong values that had passed earlier, less-complete normalization. Cache hit rate after schema stabilization: ~43% during re-processing runs (35-55% depending on how many fields changed per iteration).
What's Missing: Adversarial Resistance
These guardrails were designed for accidental failures, not adversarial ones. The security post covers prompt injection and document-level attacks, but the guardrails themselves aren't hardened against deliberate gaming.
Someone who understood the confidence thresholds could craft queries producing high re-ranker scores while targeting wrong values. Citation verification catches values not in the text, but not an adversary steering extraction toward a specific value that IS in the text. More subtly: if an attacker controls which documents are in the corpus (not unrealistic when sellers provide documents for due diligence), they can plant documents designed to produce high re-ranker scores, pushing extraction toward attacker-chosen values that pass all guardrails. That's semantic poisoning at the corpus level, and the only defense is upstream document provenance verification - a business process, not an AI guardrail.
In our context (internal legal team, documents from official government sources), these gaps were acceptable. For external users or untrusted document sources, they wouldn't be.
Monitoring Guardrail Effectiveness
Guardrails need their own monitoring. Too strict wastes human time. Too loose lets errors through.
What we tracked: false positive rate per trigger and per document type (citation verification highest on revenue records, confidence thresholds highest on older documents with OCR noise). Recall on the validation subset (~85% of errors caught - but with the selection bias caveat that this set skewed toward cleaner, earlier-processed documents; recall on the harder tail was likely lower and we don't have the ground truth to know by how much). Escalation rate by document type and metro - the aggregate rate is misleading because post-2010 Bengaluru apartments (3-4%) and 1990s Telugu revenue records (12-15%) are completely different populations. Resolution time per escalation - dropped from ~85 seconds in week 1 to ~35 seconds by week 6; climbing resolution time usually meant genuinely harder cases or missing handoff context.
The guardrails weren't static. They were tuned continuously as the corpus moved from clean modern documents to older, messier ones. The goal: enough escalations to catch real errors, not so many that reviewers drowned in false alarms. That balance shifted weekly, and the thresholds shifted with it.
Content on this page may not be reproduced, distributed, or republished without prior written permission. Sharing links is encouraged. See our Terms of Use for details.