SLOs for features that answer differently every time

A classic SLO says: 99.9% of requests succeed, and 99% finish within 300 ms. Put a language model behind the feature and both halves stop making sense. Responses take seconds by design. The model provider is a dependency you don’t control. And an HTTP 200 can carry a confidently wrong answer.

The goal stays the same: decide what “working” means for the user, measure it, and agree on how much failure is acceptable. The measurements change.

Latency: measure what the user waits for

For streamed responses, total duration is the wrong number. The user waits for the first token, then reads at the speed the text arrives.

• Time to first token (TTFT) is the latency that feels like “is this broken?”. Set the SLO here, for example 95% under 2 seconds.
• Output speed (tokens per second after the first) decides whether reading keeps up. Track it, and alert on sustained drops.
• Total time still matters for non-streamed work like batch jobs and tool calls. Set those SLOs separately rather than averaging two very different shapes.

Availability: count the errors a status code hides

Count as failures:

• Provider errors and rate limits (HTTP 429) that reach the user, after your retries.
• Timeouts, including streams that start and then stall.
• Responses your own checks reject: empty output, invalid JSON when you asked for JSON, failed safety filters, or answers truncated by the token limit.

If a fallback model handled the request, count it as a success, but keep a separate metric for how often the fallback runs. A fallback that serves 30% of traffic is an incident that hasn’t been declared yet.

Quality: sample it, don’t guess it

Correctness can’t be fully measured in real time, but it can be sampled:

• Explicit feedback, such as thumbs down or a retry, as a rate per thousand responses.
• Automated checks on a sample of traffic: does the answer cite the provided sources, does it follow the required format, does it stay on topic. Score them offline and trend the results.
• A fixed test set run on every prompt or model change. It works like a regression suite and stops a “small prompt tweak” from quietly lowering quality.

Quality SLOs are usually looser and slower-moving than availability. A weekly target, such as negative feedback under 3%, is more honest than a per-minute alert.

Error budgets cover the provider too

Your SLO is what users experience, so the model provider’s outages spend your error budget. That’s fair: it’s how you find out whether one provider is enough. Know your provider’s limits, keep a fallback (a second region, a second model, or a smaller self-hosted model for degraded mode), and test the switch before you need it.

Instrument it the standard way

OpenTelemetry has semantic conventions for generative AI calls (gen_ai.* attributes for the model, token counts and operation type). They’re still marked as in development, but using them now means model calls show up next to the rest of your traces, and dashboards built on them survive a change of provider. Record input and output token counts on every call. They drive both latency and cost.

A feature is reliable when users can depend on it. For LLM features that means fast enough to start, available enough to trust, and right often enough that people don’t check every answer.