Testing Conversational AI in Production: DeepEval, Prompt Regression, and the Evaluation Stack
Part of LangChain Agent to Production: WhatsApp Hotel Booking
For the first few weeks, "testing" meant someone typing messages at the bot and eyeballing what came back. It caught the obvious failures and created false confidence about everything else, including an intent-classification drift on Hindi-English queries that ran undetected for days.
This post is a deep dive from our WhatsApp hotel booking case study, where we shipped a production conversational AI for hundreds of properties. Here's the testing infrastructure that made iteration possible.
Why Manual Prompt Testing Fails Silently
Eyeballing responses catches the glaring stuff: a hallucinated room type, a check-in date that makes no sense. It catches nothing subtle.
The trap isn't the bugs you miss. It's how sure you feel afterward. You type ten messages, the bot handles them, you ship. What you don't see: a model swap last Tuesday shifted intent classification accuracy by 3%, enough that "early check-in" queries now land in the wrong state 1 in 20 times. A prompt tweak that improved date extraction for English quietly degraded Hindi-English code-switched messages. A retrieval change that boosted relevancy for beach properties worsened results for hill stations.
These failures spread across hundreds of conversations, unevenly distributed by property and language. By the time an owner complains that "the bot keeps getting confused," you've been shipping a degraded experience for days.
Conversational AI has a combinatorial surface area. Eight states in our LangGraph. Dozens of intent categories. Multiple languages. Hundreds of properties, each with a different inventory shape. Manual testing covers a vanishingly small fraction of that space. We needed automated evaluation that could run across it on every change and give us a number, not a gut feeling, when we asked whether a change was safe to ship.
Component-Level Testing with DeepEval
The first layer tested individual LangGraph nodes in isolation. Before worrying about end-to-end booking flows, we wanted confidence that each piece (intent classifier, constraint extractor, inventory ranker, response generator) performed above a baseline.
DeepEval made this practical because it plugs into pytest. Evaluation runs looked like any other test suite: pytest tests/eval/ in CI, pass/fail in the same format as everything else.
Setting Up Metrics
AnswerRelevancyMetric checked whether a node's output actually addressed the input. We ran it on the response generator to catch a common failure mode: fluent, confident text answering a slightly different question than the one posed.
HallucinationMetric compared outputs against provided context. For the inventory ranker this was the critical one. Did property explanations match actual data, or did the model invent amenities? We'd seen it live: the model confidently stated "complimentary breakfast included" for a property that charged extra.
ContextualRecallMetric measured whether retrieval surfaced the right information. Low contextual recall meant the downstream model was working from incomplete data, no matter how good its generation looked.
Custom G-Eval Criteria
The built-ins covered general quality. The most valuable checks were custom G-Eval criteria: task-specific evaluation prompts that scored domain correctness.
For the intent classifier, the criteria covered primary intent detection, date extraction, guest count and room configuration, hard constraints like budget and amenities, and message language, with explicit scoring rules that penalized missed constraints.
The thing we learned about G-Eval: vague criteria produce vague scores. "Is the output good?" tells you nothing. "Does the output correctly extract dates in YYYY-MM-DD format when the guest says 'next weekend'?" catches real bugs. It took about two weeks of iterating on criteria specificity before they became reliable regression detectors.
Pass Thresholds
We set pass thresholds at 0.85, not 1.0. LLM-based evaluation is itself non-deterministic; the evaluator model sometimes scores the same output differently across runs. At 1.0 you get constant flaky failures and a test suite nobody trusts. At 0.85, genuine regressions reliably trip the threshold while normal variance stays inside it.
Component evals ran on every PR. A developer changing a system prompt got immediate feedback: your tweak dropped intent accuracy by over 10 points on the Hindi code-switching test set. No ambiguity, no waiting for production complaints.
Unit Testing LangGraph Nodes
DeepEval scored output quality. The deterministic parts of each node (state transitions, tool calls, structured data handling) got traditional unit tests. Plain pytest assertions against mocked inputs.
Our booking system had eight LangGraph states: Inbox, CollectStayConstraints, SearchInventory, RankAndExplainOptions, CommitBooking, PaymentAndReceipts, PostBookingOps, and HumanHandoff.
What We Tested Per Node
Intent classifier (Inbox node). Given a raw WhatsApp message, did the classifier produce the correct intent label, the detected language, and a flag for whether the message referenced an existing booking? We built multilingual fixtures with ground-truth labels covering English, Hindi, and code-switched inputs. Pure classification accuracy: assert output.intent == expected_intent, no LLM judging needed.
Constraint extraction (CollectStayConstraints). Given a classified message, did the node populate the structured state fields: check_in_date, check_out_date, guest_count, room_config, budget_max, required_amenities? We tested explicit constraints ("2 adults, 1 child, under 5k/night") and implicit ones ("next weekend" should resolve to the right Friday–Sunday). The current date was mocked so tests stayed deterministic.
Tool call correctness (SearchInventory, CommitBooking). Given a populated state, did the node call the right tool with the right parameters? We mocked the PMS API and asserted on call arguments: date range, occupancy, amenity filters. For CommitBooking we also verified the node passed the correct booking_intent_ref for idempotency and included all required guest details.
State transition logic. Given a node's output, did the graph route to the correct next state? A low-confidence classification should route to HumanHandoff. A successful booking should route to PaymentAndReceipts. A PMS timeout should retry with backoff, then route to HumanHandoff once retries run out.
Mocking Strategy
Every external dependency (PMS APIs, payment gateway, WhatsApp sending) was mocked at the tool boundary. Nodes received a tool registry; tests swapped real tools for mocks returning controlled responses. That let us test scenarios hard to reproduce live: the PMS returning stale inventory, the payment gateway timing out, zero availability for the requested dates.
The mocks also let us replay failure modes that had caused real incidents. What happens when the PMS returns a malformed response? When the inventory cache is 45 minutes stale? When a guest sends an image instead of text? Each one became a permanent test case.
65 Test Cases for One Booking Flow
Component tests verified individual nodes. Integration tests verified that the full conversational flow (multiple turns, state transitions, tool calls, response generation) produced correct end-to-end outcomes. We built 65+ of them.
Happy Paths
Complete booking flows from first message to confirmation, across English, Hindi, and code-switched queries, from simple ("room in Goa this weekend") to gnarly ("3 rooms, 7 adults, 2 kids, breakfast included, pool required, under 6k/night per room"). Each case defined the full multi-turn conversation as input and checked the final booking state, the properties surfaced at each stage, and the tool calls made.
Edge Cases
The scenarios that broke the system in production, now immortalized as regression tests. Code-switched Hindi queries that confused the language detector. PMS timeouts during booking confirmation. Guests changing their dates mid-conversation after seeing options. Referential queries: "that cottage you showed me earlier." Budgets expressed sideways ("not too expensive" vs. "under 5k"). A guest who gets results, goes silent for 6 hours, then picks the conversation back up. Properties with zero availability returning empty results.
Production Failure Reproductions
Every incident became a test case. The guest who was promised "complimentary breakfast" at a property that charged for it. The Hindi expletive classified as a booking intent. The multi-room query where the agent quoted a per-room price but displayed a total assuming single occupancy. Each test carried the actual guest messages that triggered the failure, the expected correct behavior, and assertions on the specific failure point.
Handling Non-Determinism
Running 65 multi-turn conversations through an LLM-based system means living with non-determinism. Three mechanisms kept it manageable.
Threshold bands, not exact matches. The 0.85 pass threshold applied across the suite, not per case. A single case scoring 0.80 didn't fail the build; the aggregate had to stay above the bar.
Single retry on failure. This absorbed the 5-7% flake rate from evaluation non-determinism. Failed twice? Genuine regression.
Structural assertions alongside LLM scoring. Was the right property booked? Was the correct price quoted? Did the tool call include the right dates? Those checks were deterministic assertions. LLM evaluation covered quality and relevance; structural assertions covered facts.
Runtime and Gating
The full suite was too slow for every commit and too important to skip, so it gated release-candidate builds rather than individual merges. Everything else got by on component-level DeepEval tests.
Picking Between DeepEval, promptfoo, LangSmith, and TruLens
DeepEval was our primary framework, but we tried the neighbors. Each occupies a different niche.
DeepEval
Component-level LLM evaluation with tight CI integration. G-Eval custom criteria is the standout feature: define evaluation rubrics as natural-language prompts and an evaluator LLM scores outputs against them. pytest integration means evals run alongside regular tests with no separate infrastructure. The limitation is scope: it's a testing tool, not an observability platform. It says pass or fail; it won't help you understand production behavior over time.
promptfoo
A/B prompt testing and model comparison. If the question is "which of these three prompt versions performs better on this golden dataset?", promptfoo is the most direct path. We used it during prompt iteration: five variants of the intent classification prompt against a curated dataset, winner picked on classification accuracy instead of intuition.
LangSmith
Worth it when you're already in the LangChain/LangGraph ecosystem and want evaluation wired into tracing. Dataset-based evaluations let you build test sets from production traces, annotate expected outputs, and evaluate against them. Its LLM-as-judge works much like G-Eval but sits deeper in LangGraph's execution model. We used LangSmith for graph-level evaluation during development while DeepEval handled CI gating.
TruLens
Built for multi-step agent behavior: tool selection accuracy, reasoning chains, feedback loops. For us that would have meant checking the agent picked the right tool at each step. We explored it and passed. DeepEval for component quality, plus structural assertions for tool-call correctness, covered the same ground with less setup.
When to Use Which
In practice most teams need at most two: one tool gating CI and one for deeper analysis. Ours were DeepEval and promptfoo, with LangSmith during development. TruLens earns its keep when agent tool-selection itself is the risky part.
Monitoring the Live System
Pre-deploy testing catches regressions. Production traffic is stranger than any test suite imagines.
We sampled live Langfuse traces, stratified by property for coverage, and fed them into batched DeepEval runs using the same metrics and G-Eval criteria as the integration tests. The output was a weekly quality scorecard measured on real guest conversations: answer relevancy, hallucination rate, contextual recall, intent classification accuracy.
Drift Detection Triggers
We defined three alert thresholds.
Per-metric regression. Any metric dropping below its trailing average got flagged for investigation. A dip in contextual recall meant something shifted in retrieval: a property updated its listing in a way our embeddings didn't capture, or a new guest segment brought query patterns retrieval wasn't tuned for.
Per-property anomalies. Langfuse's per-property tracing showed not just that quality was drifting but where. A Goa property whose retrieval recall dropped 10 points while every other Goa property held steady points to a property-specific data issue, not a systemic one. That granularity is the difference between "something is wrong with search" and "property X updated their amenities list and our cache hasn't refreshed."
Stage-level degradation. If one LangGraph stage (say, CollectStayConstraints) declined while the others held, the problem lived in that node's prompt or logic. That saved hours compared to staring at an aggregate quality drop and binary-searching the whole pipeline.
When to Investigate vs. When to Auto-Remediate
Not every alert needed a human. A recall drop traced to a stale property cache triggered an automatic refresh. A hallucination spike on one property after a PMS data update triggered automatic re-indexing of that property's inventory. Known failure modes, known fixes.
Humans took the novel patterns: a query type the system hadn't seen, a quality drop correlated with a model provider's API update, degradation spanning properties with no obvious data cause. The point was to keep the alert-to-action ratio high enough that alerts stayed meaningful instead of becoming noise.
When to Gate, When to Just Report
The three evaluation layers mapped to different points in the deployment pipeline. Getting the gating strategy wrong costs almost as much as having no tests.
The Pipeline
PR merge: component gate. Every pull request ran the DeepEval component tests. Runtime 2-4 minutes. Fail meant no merge. This caught the most common regression: a prompt change that improved one dimension and degraded another.
RC build: integration suite. PRs touching prompts, LangGraph node logic, or retrieval configuration triggered the full 65-case suite on merge. It gated the release, not the merge. That distinction matters for development velocity.
Deploy: production sampling. Post-deployment, the monitoring pipeline ran its first evaluation within 24 hours. Not a gate, a verification step. If quality dropped we could roll back in minutes. In six months we rolled back twice: once for a retrieval regression component tests didn't cover, once for a model provider API change that altered output formatting.
What Triggered Each Layer
Prompt changes gated separately from code changes. Retrieval and embedding configuration changes triggered additional targeted evaluation sets. Model provider changes ran all three layers plus a manual review before release. CI stayed fast for changes that didn't touch AI behavior and thorough for changes that did.
What All This Testing Actually Bought
Confidence to make changes. That mattered more than any single metric.
Before: every prompt tweak, model swap, or retrieval update was a leap of faith. Deploy, hold your breath for 48 hours, wait for complaints. Some changes we simply didn't make because the risk felt too high.
After: model swaps went from multi-day deliberations to run the suite, check the dashboard, ship if green. Prompt updates could be aggressive because regressions surfaced in minutes instead of days. Retrieval changes (the scariest category, since they hit every property differently) got validated against per-property baselines before any guest saw them.
None of it was glamorous. Component evals on every PR, the 65-case suite on every release candidate, a weekly scorecard on live traffic. But that's the infrastructure that turned "deploy and pray" into change, measure, ship. For a system handling real bookings across hundreds of properties, that confidence was worth more than any feature we built.
For the full production system this testing stack was built around: Shipping a LangChain Agent to Production: Conversational Hotel Booking Over WhatsApp. For observability and tracing: AI Observability in Production. For guardrails and confidence thresholds: Production Guardrails for AI Systems.
Content on this page may not be reproduced, distributed, or republished without prior written permission. Sharing links is encouraged. See our Terms of Use for details.