Debugging Production AI: The Observability Stack That Tells You Why Your System Broke
Part of Anatomy of a Production RAG System: AI-Powered Title Validation for Indian Real Estate
Every dashboard was green. Latency was normal. Error rates were flat. And the legal review team was getting confidently wrong answers - because traditional monitoring measures whether a request succeeded, not whether the answer was right.
This post is a deep dive from our production RAG case study, where we built a document intelligence system that validates property title cleanliness across over a thousand properties in six Indian metros. The observability stack described here was built after we spent two weeks debugging extraction errors by reading logs manually.
Why Traditional APM Is Not Enough
ECS tasks healthy, memory and CPU well inside limits, Step Functions executions succeeding end to end. By every infrastructure signal we had, the system was working.
And the legal review team was getting wrong answers.
The system returned HTTP 200 with a fluent, well-cited, completely incorrect extraction. "Consideration amount: ₹45,00,000" when the deed said ₹54,00,000. The response included a citation to the correct page. The format was perfect. The number was wrong because the retrieved chunk contained a different transaction's consideration amount from the same document, and the LLM picked the wrong one.
Traditional APM tells you the request succeeded. It did. The answer was just wrong. You need a different kind of monitoring for that.
The Three-Layer Stack
We ended up with three layers, each answering a different question:
CloudWatch: "Is the system running?" Infrastructure metrics. ECS task health, memory and CPU utilization, Lambda invocation counts and errors, S3 upload rates, Step Functions execution status. This catches outages, resource exhaustion, and infrastructure failures. It does not catch wrong answers.
Langfuse: "Is the system correct?" AI-specific observability. Per-request traces showing every step of the pipeline: which chunks were retrieved, what re-ranking scores they received, what prompt was constructed, what the LLM returned, what cost was incurred. This is where quality debugging happens. Every extraction, every retrieval, every generation call is a trace with metadata we can filter, search, and aggregate.
Grafana: "What's the trend?" Dashboards combining metrics from both layers into views that show quality over time, not just point-in-time. Retrieval relevance trends, extraction accuracy trends, cost per property trends, human review queue depth. The dashboard that tells you something is degrading before anyone complains.
These layers don't duplicate each other. CloudWatch never tells you the answer was wrong. Langfuse never tells you the ECS task ran out of memory. Grafana shows patterns that neither raw metric source reveals on its own.
What We Tracked
Not everything is worth monitoring. We started with too many metrics, then cut to the ones that actually drove action.
The single most useful metric we tracked was the median cross-encoder re-ranker score per query - a 0-1 score for how well each retrieved chunk matches the query. When the median dropped, the retrieval pipeline was serving lower-quality chunks. We set alerts on sustained drops over 24 hours. The Hyderabad Telugu records incident (described in the case study) was caught by this metric two days before anyone on the legal team noticed degraded answers.
Extraction accuracy: sampled comparison against lawyer-verified values. Every week we pulled a sample of extractions and compared the system-extracted values (parties, consideration amounts, dates, property descriptions) against what the lawyer verified. This gave us a running accuracy rate per field. Party name extraction ran at roughly 97% accuracy. Consideration amounts at roughly 95%. Property descriptions at roughly 89% (the lowest, because descriptions are the most structurally complex). When any field's weekly accuracy dropped noticeably, we investigated.
OCR confidence distribution. The percentage of chunks in the vector store with OCR confidence below threshold. A batch of particularly old properties would spike this metric, growing the human review queue. We tracked this to predict human reviewer workload and catch batches that would degrade downstream quality if not reviewed first.
Latency by stage. Ingestion, OCR, retrieval, re-ranking, generation, each tracked independently. Not for user experience (lawyers weren't waiting in real-time) but for bottleneck identification. When retrieval latency spiked, it usually meant the pgvector index needed a vacuum or the Elasticsearch index needed optimization. When generation latency spiked, it usually meant we were hitting rate limits on the LLM provider.
We also tracked how many pages got routed to human-in-the-loop review. A rising rate was the earliest signal of timeline slip - it meant either OCR quality was degrading (bad) or our confidence thresholds were too aggressive (fixable). Either way, it directly predicted cost.
Flag overturn rate. How often lawyers overturned the system's title defect flags. This was the downstream quality signal. If the system started flagging more false issues, the overturn rate climbed. A rising overturn rate over two or more weeks meant extraction or reasoning quality was degrading somewhere upstream.
Tracing a Failure: From Wrong Answer to Root Cause
Here's what a real debugging session looked like.
The report: A lawyer flagged that the system extracted the wrong seller name for a Pune property. The system said "Suresh Patil" sold the property. The deed said "Suresh Patel." Close, but wrong. This matters because party name mismatches break the chain-of-title analysis.
Step 1: Find the trace in Langfuse. Every extraction generates a trace with the property ID, document ID, and extraction output. We searched for the property ID and pulled the trace.
Step 2: Check what was retrieved. The trace showed which chunks were retrieved and their re-ranker scores. The top-ranked chunk was from the correct deed, correct page. The retrieval was fine.
Step 3: Check what was sent to the LLM. The prompt included the chunk text. Reading the chunk, we could see "Suresh Patel" in the original text. So the retrieval was right and the source text was right.
Step 4: Check what the LLM returned. The extraction output said "Suresh Patil." The LLM had changed "Patel" to "Patil." This wasn't a retrieval problem or a chunking problem. It was a generation problem: the model was "correcting" the name based on its training data, where "Patil" is a more common Marathi surname and the property was in Pune (Maharashtra).
The fix: Post-extraction citation verification - comparing extracted values against source chunk text. The full implementation and its failure taxonomy are in the guardrails deep dive.
Without Langfuse traces, this would have been a day of reading logs, re-running extractions, and guessing.
Drift Detection: Catching Problems Before Users Complain
Quality drift happens silently. The system doesn't crash. It doesn't throw errors. It just starts returning slightly worse answers, and nobody notices until the cumulative effect becomes obvious.
Three sources of drift we encountered:
Document batch composition changes. When the pipeline moved from processing mostly post-2010 Bengaluru apartments (clean digital registrations, English-heavy) to 1990s-era Hyderabad properties (Telugu-language revenue records - government land ownership documents, similar to county land records - with older scan quality), retrieval quality dropped. The embedding model handled modern English documents better than mixed-language older documents. The re-ranker score metric caught this within two days. The fix wasn't model-related; it was re-tuning the OCR confidence thresholds for the different document profile.
Classifier drift on non-standard formats. The Hyderabad Telugu records incident from the case study: a specific sub-registrar office (the local property registration authority - similar to a county recorder's office) used a non-standard format for revenue records. The classifier was misidentifying them, which meant wrong chunking boundaries, which meant retrieval quality dropped for those properties. Caught by the re-ranker score metric before the legal team reported issues. Fix: added the format variant to the classifier training set and re-processed the affected batch.
LLM behavior changes after provider updates. During the project, Anthropic updated Claude 3.5 Sonnet. The update was minor, but extraction consistency on one schema field (property boundary descriptions) changed subtly: the model started including more context around the boundary description instead of extracting just the description. This inflated the extracted field and broke downstream comparison logic. We caught it through the weekly extraction accuracy sampling. The field accuracy dropped from 89% to 81% in one week. Fix: tightened the schema prompt for that field and added a character-length validation rule.
All three were caught by automated metrics before manual reports. That's the point of the observability stack: you learn about problems from dashboards, not from angry emails.
Cost Observability
AI systems have a cost dimension that traditional software doesn't. Every LLM call, every Textract page, every embedding generation has a per-unit cost that scales with usage.
We tracked cost alongside quality in the same Langfuse traces. Every trace includes token counts and estimated cost. Aggregated per property, per document type, per processing stage.
What this revealed: Cost varied by an order of magnitude across property types. Complex properties (long chains, many documents, older scans requiring multimodal OCR) cost far more to process than simple apartments. This wasn't obvious until we had per-property cost attribution. The most expensive properties in the portfolio were driven by human review time on handwritten documents and multiple rounds of LLM extraction on ambiguous clauses.
The Anthropic batch API optimization (described in the case study economics section) was identified through cost observability. We noticed that extraction calls dominated the LLM cost. Since extraction isn't latency-sensitive (lawyers don't wait for real-time extraction), we moved to batch processing, roughly halving LLM costs.
What We'd Build Differently
Automated quality regression tests. We tracked metrics and investigated when they dropped. What we should have built: a nightly regression suite that re-runs a fixed set of 50-100 queries against the current pipeline and compares results to known-good baselines. This would have caught the model update issue immediately instead of waiting for the weekly sampling cycle.
Per-document-type dashboards. Our Grafana dashboards showed aggregate metrics. We should have had per-document-type views from the start. Sale deed extraction accuracy and revenue record extraction accuracy are different metrics with different baselines. Aggregating them hid problems in less common document types until the lawyer team flagged them.
Real-time alerting on extraction field failures. We had threshold alerts on aggregate metrics. We should have had per-field alerts: if party name extraction accuracy drops below 95% in any rolling 48-hour window, alert immediately. The aggregate metric can stay green while a single critical field degrades, which is exactly what happened with property boundary descriptions after the model update.
The debug hierarchy holds: investigate data preparation first (OCR, chunking), then retrieval (search, re-ranking), then generation (prompts, model choice) last. But the observability stack needs to surface problems at each layer independently. Aggregate metrics are a starting point. Per-layer, per-field, per-document-type metrics are what actually let you diagnose.
Content on this page may not be reproduced, distributed, or republished without prior written permission. Sharing links is encouraged. See our Terms of Use for details.