The GPU line item is the new EC2 line item

For years, the cost review started with EC2 at the top of Cost Explorer. In a lot of accounts it now starts with GPU instances, or with a model API invoice that didn’t exist two years ago. The order of operations from the EC2 playbook still works: turn off, rightsize, commit, make it boring. Each step just needs a different measurement.

1. Find GPUs doing nothing

GPU instances are often idle and rarely look like it. A notebook instance left running over a weekend, or an inference endpoint sized for a launch-day traffic spike that never came. CPU metrics won’t tell you. Collect GPU utilization directly (NVIDIA’s DCGM exporter puts it into Prometheus) and look for instances under 10% utilization for days at a time.

Development GPUs get the same treatment as dev EC2: scheduled shutdown, and an owner tag that’s required to launch them.

2. Rightsize the model before the instance

With inference, the cheapest change is usually upstream of the hardware. A smaller model, a quantized model or a shorter prompt can cut cost more than any instance change. Measure quality against your own test set, not a leaderboard, and only then pick the smallest instance that holds your latency target.

3. Match the purchase to the workload

• Training and batch jobs that checkpoint can tolerate interruption. Spot capacity is a good fit when the job can resume from its last checkpoint.
• Steady inference looks like any other always-on service. Compute Savings Plans cover GPU instance families too, so commit to the baseline you measured in step 1, not the peak.
• Planned bursts, like a training run next month, can use reserved capacity for a fixed window (on AWS, Capacity Blocks for ML) instead of a one-year commitment.

Same rule as before: commit last. Buying a commitment before steps 1 and 2 locks in a discount on idle hardware.

4. Make the unit cost visible

Total spend is the wrong number for AI features because it should grow when usage grows. What you want to know is whether each unit of work is getting cheaper. Pick a unit that matches the product, such as cost per thousand requests, per document processed, or per million tokens. Then chart it weekly next to the total.

If you pull billing from several providers, the FinOps Open Cost and Usage Specification (FOCUS) gives you one schema for all of them. AWS, Azure and Google Cloud can all export in it, so cloud GPUs and other usage end up in the same table.

The weekly report goes to the same people as before: whoever owns the tag. A number with a name next to it still moves faster than any dashboard.