AWS Infrastructure Checklist for AI Workloads

Every item on this checklist either prevented a production failure or was added after one - across the conversational agents, document intelligence pipelines, and RAG systems we've run on AWS. Standard web-service defaults (512MB tasks, CPU-based scaling, single-region everything) break in expensive ways once AI traffic arrives.

When a production AI system fails, the post-mortem almost always lands on one of three findings: nobody can explain the cost, the system served a wrong answer confidently, or it fell over under real load. The sections below are organized by AWS service category, but keep those three in mind. Every item traces back to cost, correctness, or reliability.

Compute: Sizing, Scaling, and Orchestration

ECS/Fargate capacity. An AI service is hungrier than it looks. Conversation state, embedding caches, and model client connections push memory to 2–4x what a comparable web service needs. Most teams start at the 512MB default and scale up after the first crash. That works, but you pay for it in downtime. Profile after two weeks of real traffic and right-size from data.

Auto-scaling. CPU-based scaling reacts too slowly for AI traffic. A spike can hit 10x in minutes; by the time new tasks spin up on CPU metrics, requests have timed out, users have retried, and every retry is a duplicate LLM call. You pay twice for the same answer. Scale on application-level metrics instead - active sessions, queue depth, batch size - and put SQS in front to absorb the burst while capacity catches up. Give every request entering the LLM gateway an idempotency key, so retries don't trigger duplicate inference no matter where they come from.

Two things surprise teams here. First, the real scaling bottleneck is often the model provider's rate limit, not your compute. Adding containers doesn't help when the provider is throttling you, so set per-container concurrency with that ceiling in mind. Second, in agentic systems cost multiplies per execution path, not per request: retries, tool calls, and fallbacks all stack. And without per-tenant rate limiting, one tenant's burst starves everyone else.

Fargate vs EKS. Default to Fargate. EKS earns its keep when the architecture needs latency-critical sidecars: sub-10ms entitlement checks at the gateway, observability agents, service mesh proxies. For simpler sidecar setups, Fargate handles them fine (platform version 1.4.0+).

Lambda. Wrong abstraction for multi-turn or stateful AI. Right abstraction for batch and event-driven preprocessing: embedding generation, document classification, webhook processing.

Step Functions. Orchestration buried in application code hides retries and lets partial failures leak downstream - inconsistent embeddings, half-processed batches, data loss nobody notices for a week. Step Functions puts error handling, retries, and timeouts at the workflow level, with stage-level observability built in.

Textract. Document-heavy pipelines need OCR with confidence-based routing. Skip the confidence scoring and low-quality OCR flows into your embedding pipeline silently, degrading everything downstream. Route pages below threshold to a multimodal LLM or human review.

Bedrock vs direct API. This is a compliance-versus-flexibility call. Bedrock gets you VPC endpoints, IAM-based access, and coverage under AWS compliance programs - when a deal stalls on data residency or HIPAA, Bedrock usually removes the objection. The tradeoff: narrower model selection, limited batch and prompt-caching discounts, slower access to new models. Direct APIs also make dynamic routing possible - cheap models for classification, expensive ones for reasoning, confidence-based escalation, per-tenant model selection. A single Bedrock endpoint can't express that.

Storage and Data: Vector Database, Search, Caching

Vector database selection.

pgvector on RDS if you're already on PostgreSQL. Embeddings live next to relational data and joins are trivial. Plan for index tuning at scale.
OpenSearch for AWS-native hybrid search - BM25 and vector in one managed service. Most precision-critical domains need both.
Hybrid search stops being optional the moment your domain has precise identifiers: account numbers, registration numbers, case IDs. Pure semantic search returns plausible wrong answers on exact-match queries.
Neptune when the questions are about relationships - ownership chains, dependency graphs, knowledge graphs.

Session state. DynamoDB for sessions that span hours or days (payment flows, async handoffs): durable, no capacity management. Redis for sub-millisecond reads and ephemeral state that can expire on a TTL.

Object storage. S3 with lifecycle policies, and keep raw documents, processed chunks, and generated embeddings separate. Raw documents need long retention for audit. Chunks and embeddings can always be regenerated.

Content quality gates. Score inputs on the way in - OCR confidence, extraction confidence, language detection - and route anything below threshold to human review instead of the embedding pipeline. When answer quality drops, teams blame the model. In our experience the problem is usually upstream.

Caching. ElastiCache for response caching, with one hard rule: invalidation must be tied to upstream data-change events. A stale cached answer costs you the original inference plus the recovery conversation that follows it.

Networking: API Gateway, Load Balancing, and Latency

API Gateway. A multi-constraint AI request can take 8–12 seconds - closer to the default 29-second timeout than most teams expect. Set timeouts explicitly. Same goes for payload limits: the default 10MB runs out fast on document-upload endpoints.

Circuit breakers on the LLM gateway. When a provider starts returning 503s or latency blows past budget, the gateway should degrade on purpose: fall back to a cheaper model, serve a cached response, or skip enrichment and answer fast. Without that, one provider outage cascades into timeouts, retries, and a cost spike across every tenant.

ALB health checks. AI services take 15–20 seconds to start - model clients load, caches warm, vector database connections open. Default health-check settings will mark them unhealthy before they finish initializing.

Latency budget. Decide the target before you build: under 4 seconds P50 and under 8 seconds P95 for interactive AI, throughput-optimized for batch. Then split the budget across network, processing, inference, and delivery. That split ends up driving model selection, caching strategy, and every quality-versus-speed tradeoff.

Streaming responses. Streaming partial output cuts perceived latency for interactive AI, but it touches more than the UI. The gateway must support chunked transfer, timeouts split three ways (connection, first-byte, total), and the client has to handle partial responses.

EventBridge. Let AI services make decisions and let downstream systems handle side-effects. EventBridge fans events out without the inference pipeline waiting on anyone.

CloudFront. Useful for static assets. Keep it away from real-time inference - caching dynamic AI responses is how you end up serving stale answers.

Security: IAM, KMS, Secrets Manager, WAF for AI

Secrets Manager. Every LLM API key lives in Secrets Manager with rotation. Not in environment variables, not in code, not in CI config. On Bedrock, IAM roles replace keys entirely.

KMS encryption. Encrypt vector databases, document storage, and conversation logs at rest. PII gets its own encrypted store - contact details, financial identifiers, government IDs - keyed by session and user. The model only ever sees opaque placeholders; the real values resolve server-side when a tool needs them.

Pre-retrieval access control. Enforce access at query time, with metadata filters in the vector database restricting results to documents the user is authorized to see. Filtering after retrieval isn't access control - result counts and latency patterns still reveal what the user wasn't supposed to find.

WAF for AI endpoints. Per-user rate limiting (one user shouldn't be able to exhaust your inference budget), payload inspection for prompt-injection patterns, bot detection, and webhook signature verification before anything gets processed.

IAM and VPC. Least privilege, concretely: the inference service reads the retrieval corpus and writes to the application database, and never touches raw PII vault values. AI processing runs in private subnets, with NAT for outbound API calls - or Bedrock VPC endpoints to skip internet egress entirely.

Observability: CloudWatch, X-Ray, and AI-Specific Monitoring

Traditional APM tells you the request succeeded. In an AI system, the request can succeed and the answer can still be wrong.

LLM gateway as the anchor. Route every LLM call through one gateway that stamps tenant_id, session_id, pipeline_stage, model_id, and token counts onto each call. That metadata feeds cost attribution, quality monitoring, and billing. Skip it and you have a cost center you can't decompose.

CloudWatch custom metrics. Tokens per request by model tier, cost per session and pipeline stage, retrieval quality (hit rate @ k, MRR, answer grounding rate), and errors by category: tool failure, timeout, hallucination, PII leak prevented, confidence below threshold.

X-Ray tracing. Trace end to end, from API Gateway through AI processing to response. In our traces the LLM is rarely the latency bottleneck; the upstream data fetches usually are.

Alarms. Cost per hour (catches runaway sessions), error-rate spikes (usually a model API outage), P99 past SLA (catch degradation before users do), retrieval quality below threshold (embedding drift or stale data).

Dashboards. Put quality, cost, and latency on one screen, with per-tenant drill-down that shows which usage patterns expose weaknesses. This dashboard has a habit of becoming more important than any single AI feature.

Cost: Budget Alerts, Reserved Capacity, and Right-Sizing

AI cost has two layers: infrastructure (compute, storage, networking) and inference (the LLM calls). Most teams obsess over inference and never right-size infrastructure. One dependency worth stating plainly: everything below assumes inference flows through a centralized gateway. Without one, you can't split costs by tenant, stage, or model.

Budget alerts. Give AI its own budgets in AWS Budgets, separate from general infrastructure, with alerts at 50%, 80%, and 100%. Track three categories - infrastructure, inference, storage. Inference will be the largest and the most volatile.

Reserved capacity and Spot. RDS Reserved Instances for always-on databases (roughly a third off). Fargate Savings Plans for baseline compute. Spot for batch - embedding generation, re-indexing, document processing - where savings run past half.

Batch API pricing. Non-interactive LLM work gets close to half off with batch pricing. If it isn't latency-sensitive, batch should be the default, not the exception.

Right-sizing. Default task sizes are almost always over-provisioned. Profile after two weeks of production traffic; the gap is typically around a third of the compute bill.

The Checklist

Compute

Fargate/EKS task definitions sized from actual AI workload profiling, not defaults
Auto-scaling on application-level metrics with SQS absorbing burst traffic (CPU-based scaling leads to timeouts and duplicate LLM cost)
Health check thresholds account for 15–20 second AI service startup
Batch workloads on separate, cost-optimized compute
Pipeline orchestration (Step Functions) with stage-level error handling
Bedrock vs direct API decision made on compliance and model flexibility needs

Storage

Vector database provisioned with growth headroom
Hybrid search (vector + keyword) evaluated if domain requires exact identifiers
S3 lifecycle policies differentiate raw documents, processed chunks, and logs
Response caching with invalidation tied to upstream data changes (stale cache leads to wrong answers and recovery cost)
Input quality scoring with threshold-based routing
Vector database backup and restore tested
Session state store selected (DynamoDB for durable async, Redis for ephemeral real-time)

Networking

API Gateway timeout set explicitly for AI latency profile
End-to-end latency budget defined (under 4s P50 interactive, throughput-optimized for batch)
ALB health check intervals account for AI startup time
Circuit breaker on LLM gateway with graceful degradation
EventBridge decouples AI decisions from downstream side-effects

Security

LLM API keys in Secrets Manager with rotation (or Bedrock with IAM roles)
KMS encryption at rest for vector databases, document storage, conversation logs
PII handling: ingress filtering, encrypted vault, model boundary (placeholders only), egress filtering
Access control enforced at query time, not post-retrieval (latency patterns leak data)
WAF with rate limiting and prompt injection payload inspection
Least-privilege IAM for AI service data store access
AI processing in private subnets (NAT or Bedrock VPC endpoints)

Observability

All LLM calls through a single gateway with metadata tagging (no gateway means no cost attribution)
AI-specific CloudWatch metrics: tokens/request, cost/session, retrieval quality, error categories
End-to-end X-Ray tracing from ingress through AI processing to response
Alarms for cost spikes, quality degradation, latency past SLA
Per-tenant dashboard showing quality, cost, and latency together

Cost

AI-specific budget in AWS Budgets, separate from general infrastructure
Budget alerts at 50%, 80%, 100% across infrastructure, inference, and storage
Reserved capacity for predictable workloads, Spot for batch
Batch API pricing for all non-interactive LLM workloads
Task right-sizing reviewed after initial production traffic (defaults are almost always over-provisioned)

The AWS Infrastructure Checklist We Run Before Shipping AI