Running LLM workloads on AWS without surprises

Adding an AI feature to an existing product tends to go smoothly right up until it meets the infrastructure that was designed for a different kind of workload.

The problems are consistent and they are all downstream of three properties that ordinary requests do not have.

Requests are long, and your timeouts are not

A typical API call finishes in under a second. A model call can take thirty, and a multi-step agent call can take several minutes.

That collides with defaults everywhere. API Gateway caps at twenty-nine seconds. Application Load Balancer idle timeout is sixty by default. Lambda maxes out at fifteen minutes, which sounds generous until an agent loop runs long. Your frontend fetch probably has a timeout too, and so does whatever HTTP client sits between them.

The fix is architectural rather than a matter of raising numbers. Anything that can exceed a few seconds should be a job, not a request. Accept the work, return an identifier, stream or poll for the result. Streaming responses buy you a lot here because the connection stays active and the user sees progress, but the underlying work still belongs in something that can outlive an HTTP request.

Cost scales with tokens, not with traffic

Your existing capacity planning is built on requests per second. Model cost is built on tokens, and the relationship between the two is unstable.

One user pasting a long document can cost more than a thousand ordinary requests. An agent loop that fails to terminate can cost more than a thousand of those. Neither event registers as a traffic spike on any dashboard you currently have.

So instrument tokens as a first-class metric, per user and per feature, not just in aggregate. Set a hard ceiling per request and per user per day, enforced in code rather than monitored after the fact. Alert on cost per hour, not just on error rate. The failure mode you are guarding against is not an outage, it is a quiet four-figure bill accrued overnight by a loop nobody noticed.

Retries mean something different

Your HTTP client almost certainly retries on 5xx and on timeout. For an idempotent read that is correct behaviour. For a model call it is a duplicate charge, and for an agent with write access it is a duplicated action.

Turn off blind retries on anything that costs money or has side effects. Where you do retry, use idempotency keys so the second attempt is recognised as the same request. Put model calls behind a queue with visibility timeouts tuned to the real duration, because a queue consumer that processes a message twice is the same problem in different clothing.

Rate limits from the provider also want a different response than your usual backoff. A 429 is not a transient network blip, it is a capacity signal, and the right reaction is usually to queue rather than to hammer.

What we set up before launch

A small checklist that has saved us repeatedly. Token usage in CloudWatch with alarms on daily spend. A per-user quota enforced at the application layer. Long-running work moved onto SQS with a dead letter queue. Timeouts audited end to end so no layer is stricter than the one it wraps. Model responses logged with their token counts, which doubles as your evaluation corpus later.

None of this is exotic infrastructure. It is the ordinary discipline applied to a workload whose cost and duration characteristics happen to be unusual.