A dashboard should answer one question in five seconds
Open the dashboard for almost any service during an incident and you’ll see the same thing: forty panels, half of them flat lines, in no particular order, with titles like rate(http_server_requests_seconds_count[5m]). The person on call scrolls, squints, and eventually opens a terminal instead.
Dashboards fail for the same reason as most bad documents: nobody decided what question they answer.
One question per dashboard
Before adding a panel, write the dashboard’s question at the top:
- “Is checkout healthy right now?”
- “Is this cluster running out of capacity this month?”
- “Is the nightly pipeline on time?”
Each question has a different audience, time range and level of detail. A dashboard trying to answer all three answers none of them at 3 a.m.
Read it like a page
People read dashboards the way they read anything else: top-left first, then down. Use that.
- Top row: the answer. Three to five large stat panels: requests per second, error rate, latency at p95 or p99, and the SLO with its remaining error budget. Green or red, readable from across the room.
- Middle: the shape over time. The same signals as time series, so you can see when things changed and whether it’s getting better or worse. Mark deploys on these graphs as annotations. “It started at 14:02, right after the deploy” solves a remarkable number of incidents.
- Bottom: the why. Saturation and dependencies: CPU, memory, connection pools, queue depth, downstream latency. This is where investigation starts once the top rows have told you something is wrong.
For request-driven services, the RED method (rate, errors, duration) fills the top rows. For resources like hosts, disks and queues, USE (utilization, saturation, errors) fills the bottom.
Small rules that make a big difference
- Units and plain titles. “Checkout errors (% of requests)” instead of the raw query.
- Consistent colors. Errors are always the same color on every dashboard.
- Thresholds drawn on the graph where the SLO or alert threshold is, so “is this bad?” doesn’t require memory.
- Variables instead of copies. One dashboard with a
serviceandenvironmentselector beats twenty copied dashboards that slowly drift apart.
Dashboards as code, and dashboards that die
Store dashboards in Git and provision them, rather than editing them by hand in the UI. Then a dashboard is reviewed, versioned and reproducible, and the standard layout above can be a template every service starts from.
Equally important: delete dashboards nobody opens. Grafana can show how often each dashboard is viewed. A dashboard that hasn’t been opened in six months isn’t documentation. It’s clutter that makes the useful ones harder to find.
Why this matters more in 2026
AI assistants can now generate dashboards and answer questions about metrics in plain language. That’s genuinely useful for exploring. But during an incident, a well-designed dashboard is still the fastest way for several people to share the same picture at once. A generated dashboard with forty panels has the same problem as a hand-made one. Decide the question first, whoever builds the panels.