How Infrastructure Decisions Determined Our AI Margins
Part of LangChain Agent to Production: WhatsApp Hotel Booking
We built on what seemed like reasonable infrastructure defaults. Every one broke within the first month, not in spectacular ways, but in quiet, compounding ways that showed up as margins we couldn't explain.
This post is a deep dive from our WhatsApp hotel booking case study. For the pass/fail checklist version of these decisions, see The AWS Infrastructure Checklist.
The Infrastructure Didn't Come First. The Failures Did.
We built the WhatsApp booking agent on what seemed like reasonable defaults: standard Fargate tasks, basic auto-scaling, API keys in environment variables, no centralized LLM routing. Each of those broke in its own way inside the first month.
This isn't a post about AWS best practices. It's about the infrastructure decisions that directly determined whether we made or lost money on each of our 160+ hotel properties — and how each one was forced by a specific operational failure, not planned from a checklist.
The Gateway Changed Everything
The single decision that enabled everything else in this series: routing every LLM call through one gateway.
Before the gateway, calls happened wherever they happened: different LangGraph nodes, retry logic, fallback paths. Usage showed up as one line item from the model provider. Total monthly cost across all properties. A single number we couldn't decompose.
The gateway attached per-request metadata tagging for cost attribution before forwarding to the model provider. Every call, including retries and fallbacks, passed through this single point. The same metadata flowed to Langfuse for observability and Stigg for billing — one instrumentation point feeding both systems. What that unlocked, in order:
Metering became possible. Each call emitted a Stigg metering event from the same gateway metadata. Without the gateway, our billing system would have had no usage data to aggregate against tenant budgets. The entire entitlement architecture — soft caps, model downgrades, turn limits — depends on this one piece of infrastructure.
Cost attribution became possible. Langfuse received traces on every generation with full metadata. We could slice cost by property, by pipeline stage, by model. The per-property P&L that changed our pricing was a join across gateway-tagged Langfuse data and booking revenue. Without the gateway, that join doesn't exist.
Cost leaks became visible. A flaky PMS (property management system) integration on one property triggered LangGraph retries, each retry generating a fresh LLM call. Without per-call tagging, those retries looked like normal traffic. They were doubling LLM cost on 15% of that property's conversations. Separately, a fallback path — Flash returning a 503, call re-routed to a backup model at different pricing — meant a single integration bug was responsible for over 10% of that week's inference spend before anyone noticed.
In agentic systems, cost is not per request — it is per execution path. Retries, tool calls, and fallbacks multiply cost in ways that are invisible without instrumentation.
We added the gateway after the first month of production. That month of unattributed spend was gone permanently.
If you don't have a centralized LLM gateway, you don't have a billing system. You have a cost center you can't explain.
Auto-Scaling for AI Is Not Auto-Scaling for Web
Standard CPU-based auto-scaling is built for web traffic patterns. AI traffic doesn't follow them.
A WhatsApp property listing going viral on Instagram meant booking inquiries jumping 10x in minutes. CPU-based scaling was too slow — by the time new Fargate tasks spun up, conversations had already timed out. On WhatsApp, slow responses mean guests assume the bot is broken. They resend, which spawns parallel pipeline runs, which makes things worse. During one spike, we estimated 20-30% of conversations hit timeouts before new capacity was ready.
We switched to scaling on active conversation count — a custom CloudWatch metric pushed from the application — rather than CPU utilization. More aggressive scale-up, slower scale-down to avoid flapping. The latency budget was a sub-5-second P50 target. That number drove every compute decision: model selection, caching strategy, when to trade quality for speed.
The other compute lesson was simpler. We overprovisioned by 2x initially, then right-sized based on actual peak usage. Conversation state, embedding caches, and model client connections consumed far more than typical web services. Right-sizing cut compute costs by over a third.
Both mistakes had the same root: treating AI workloads like the web workloads we'd been running for years.
PII Infrastructure Was Forced, Not Planned
We didn't build the PII vault because a compliance checklist said to. We built it because multi-party WhatsApp conversations made it unavoidable.
Guests shared UPI IDs with property managers. Managers shared payment details with guests. Support agents accessed conversations. All of it passed through the LLM context window. "Tell the model not to leak PII" was our first instinct. It was wrong — the risk wasn't the model volunteering information. It was information flowing between parties who shouldn't see each other's details.
The infrastructure: ingress filter at the webhook stripped PII before anything reached the agent. An encrypted PII vault with conversation-scoped access, placeholder substitution, and audit logging ensured the LLM never saw raw values. Egress filter caught leaks before delivery.
The design principle: swapping frameworks, changing prompts, or refactoring the agent should never affect PII handling. Infrastructure, not application logic. Government ID images were never forwarded to the model; they were stored encrypted, with every access logged. Audit logs surfaced property managers repeatedly asking guests for direct phone numbers. Small numbers, but the kind of behavior that erodes platform trust.
This infrastructure also turned out to be the difference between passing and failing enterprise security conversations. Assessments ask about data handling, key management, access logging. Having answers grounded in actual architecture — not promises about prompt behavior — is what closes those conversations.
Caching Tied to Business Events
Redis semantic caching cut inference costs by close to 20% on high-traffic properties. The implementation detail that mattered was invalidation.
Cache invalidation was tied to PMS webhook events. When availability or pricing changed, the relevant cache entries were purged. Common queries like "do you have availability this weekend?" hit the cache instead of the LLM, but only when the underlying data hadn't changed since the last answer.
Without this tie, cached responses would serve stale availability. A guest would see "available" for dates already booked. The booking attempt would fail at the PMS, the guest would lose trust, and a second LLM call would happen anyway to explain the discrepancy. Stale caching was more expensive than no caching — it cost inference plus the recovery conversation.
Three Budget Categories, Not One
For the first month, all AI spending showed up as one number. We couldn't tell whether a cost spike came from Fargate scaling, LLM API calls, or storage growth.
We split into three budget categories in AWS Budgets: infrastructure (Fargate, RDS, ElastiCache), inference (LLM API calls via the gateway), and storage (S3 conversation logs, PII vault). Tiered budget alerts on monthly projection for each.
Inference was the largest and most volatile. It tracked with conversation volume, which spiked unpredictably. Infrastructure was steady enough to commit to reserved capacity: reserved instances cut database costs by roughly a third for always-on databases, Fargate Savings Plans for baseline compute. Batch workloads (embedding generation, re-indexing) on Spot saved over half.
The split made cost conversations productive. When total spend climbed, we could immediately see which layer was responsible — and whether the fix was an infrastructure change, a model routing decision, or a product decision about conversation limits.
The Cascade
The reason infrastructure decisions matter for AI margins isn't any single component. It's how they compound.
No centralized gateway means no per-call metadata. No metadata means no usage metering. No metering means no entitlement enforcement. No enforcement means your highest-usage tenants are your most expensive ones, and you won't know which ones until the margin is already gone.
No PII infrastructure means you can't pass enterprise security assessments. No enterprise deals means your addressable market is smaller. Or worse: you pass the assessment on promises and fail the audit on architecture.
No application-level auto-scaling metrics means dropped conversations during traffic spikes. Dropped conversations on pay-as-you-go properties mean zero revenue on conversations you already paid inference costs for.
Each decision looks like a standalone technical choice. Together, they determined whether the product was economically viable. We didn't learn this from a whitepaper. We learned it from a month of margins that didn't add up.
For the per-tenant economics these decisions enabled: Per-Customer AI Cost Attribution. For the billing system that depends on the gateway: How SaaS Companies Should Be Billing for AI Features. For the pass/fail checklist version: The AWS Infrastructure Checklist.
Content on this page may not be reproduced, distributed, or republished without prior written permission. Sharing links is encouraged. See our Terms of Use for details.