Measure Kafka lag in seconds, not messages

The standard Kafka consumer lag metric is an offset count: how many messages the consumer group is behind the end of each partition. Put it on a dashboard and you’ll get a question in the first week: “Is 50,000 bad?”

Nobody can answer that from the number. On a topic that receives 100,000 messages a second, 50,000 messages is half a second behind. On a topic that receives ten a minute, it’s three and a half days. Offset lag measures the backlog, not the delay, and only the delay means anything to the people downstream.

Lag in time

What users of a stream care about is: how old is the newest data this consumer has processed? Two ways to measure it:

• Timestamp of the last processed record. Every Kafka record carries a timestamp. Have the consumer publish now - record.timestamp as a metric after processing. This is the most direct measure, and it’s the one to alert on.
• Estimate from offsets. Tools like Burrow, or exporters that track offset rates, convert offset lag into time using each partition’s recent production rate. Useful when you can’t change the consumer code.

Then the SLO writes itself in business terms: “Orders are visible in the warehouse within 2 minutes, 99% of the time.”

Partitions are your ceiling

When lag in seconds grows, the instinct is to add consumers. That only helps up to a point: within a group, each partition is consumed by at most one consumer. A topic with 12 partitions can’t use more than 12 consumers in a group. Consumer number 13 sits idle.

So partition count is a capacity decision made when the topic is created, and it’s awkward to change later because adding partitions changes which partition each key goes to. Size it for the throughput you expect at peak, plus headroom, divided by what one consumer can handle.

Rebalances are a hidden lag source

A consumer group pauses to rebalance whenever a member joins, leaves or times out. With the older “eager” protocol, every member stops consuming during every rebalance. Rolling deploys of 20 consumers could cause 20 rebalances, and each one shows up as a lag spike.

Things that help:

• Cooperative rebalancing (the cooperative-sticky assignor) moves only the partitions that need to move, so other consumers keep working.
• Static membership (group.instance.id) lets a consumer restart without triggering a rebalance, as long as it returns before the session timeout.
• The new consumer group protocol (KIP-848), generally available since Kafka 4.0, moves assignment to the broker and makes rebalances incremental by design.

Alert on the trend, not the spike

Lag in seconds will spike during deploys and catch up afterwards. That’s healthy. What isn’t healthy is lag that grows steadily, because it means consumers can’t keep up at all. Alert on lag above the SLO for several minutes, or lag rising continuously for, say, 15 minutes. That catches real problems without paging anyone for a rolling restart.

Offset lag is still useful for capacity planning. It just needs a unit, and the unit is time.