Your observability bill is a cardinality bill

Most teams find out what their monitoring costs when someone in finance asks why it now rivals the compute it monitors. Traffic usually didn’t triple. Three things did: metric cardinality, log volume and trace retention. Each one has a fix that doesn’t cost you visibility.

Metrics: every label multiplies

A time series is one unique combination of metric name and label values. Add a label and you multiply the series count by however many values that label can take:

http_requests_total{service, route, status}         50 × 40 × 5   =     10,000 series
                    + pod (200 values)                              2,000,000 series
                    + user_id                                       a very bad week

Most metric backends charge by active series, or slow down in proportion to them, so this is the number to watch. The usual offenders are labels that look harmless: pod names that change on every deploy, raw URL paths with IDs in them, customer or user IDs, and error messages used as labels.

What to do:

• Find the top series per metric. In Prometheus, topk(20, count by (__name__)({__name__=~".+"})) shows which metrics hold the most series. Managed backends have a usage view for the same thing.
• Template routes. /orders/{id}, not /orders/83712. Most HTTP instrumentation can do this for you if you configure it.
• Drop labels at the edge. Prometheus metric_relabel_configs and the OpenTelemetry Collector’s transform and filter processors can remove a label before it ever reaches the bill.
• Keep per-instance detail short-lived. Pod-level series help during an incident. Aggregate them by service for anything kept longer than a few days.

Logs: pay for the lines you read

Log bills scale with ingestion, and most of what gets ingested is never queried. Debug logging left on after an incident, health check access logs, and the same stack trace repeated thousands of times a minute.

• Sample the noise, keep the signal. Keep every error. Keep a sample of successful requests. Drop health checks entirely.
• Turn repeats into counts. A metric that counts an error is far cheaper than ten thousand copies of the same line.
• Use cheaper storage tiers for logs you keep for compliance. On AWS, CloudWatch Logs has an Infrequent Access class that costs less to ingest, for logs you rarely search. Archive to object storage when you only need to keep them, not query them.
• Set retention on purpose. “Never expire” is the default on more log groups than anyone would admit. Pick a number per log group and put it in code.

Traces: keep the interesting ones

Keeping 100% of traces is rarely worth it. Head sampling (deciding at the start of a request) is cheap but random, so it often throws away the one slow request you needed. Tail sampling decides after the trace finishes. The OpenTelemetry Collector’s tail_sampling processor can keep every trace with an error, every trace over your latency threshold and a small percentage of the rest.

That one change usually cuts trace volume by an order of magnitude while keeping almost every trace anyone would open.

Make it someone’s number

Monitoring spend spreads across every team and belongs to none of them. Break it down by service and put it in the same weekly report as compute. A team that sees its debug logs cost more than its database tends to fix that on its own.