Abhinav C

Terraform state is a production database

Teams back up their databases, restrict who can read them, and would never let two writers change the same row at the same moment without a lock. The Terraform state file that describes that database often gets none of this. It sits in a bucket with broad access, versioning switched off, and a lock table someone created by hand in 2021.

State is the map from your code to real resources. Lose it and Terraform no longer knows it owns anything. Corrupt it and the next apply can try to recreate production.

Lock with the bucket itself

For years the S3 backend needed a DynamoDB table for locking. Since Terraform 1.10, S3 can do it on its own with a lock file stored next to the state:

terraform {
  backend "s3" {
    bucket       = "acme-terraform-state"
    key          = "network/prod.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

DynamoDB-based locking is now deprecated, so this is one less resource to manage. Migrate one state at a time, and keep both locking methods enabled during the move so no run goes unlocked.

Protect the bucket like a database

  • Versioning on. It’s your undo button when an apply or a manual edit goes wrong.
  • Encryption, and tight access. State can contain secrets in plain text: database passwords, generated keys, anything a provider returns. Only CI and a few administrators should be able to read it.
  • Deletion protection. Block public access, deny deletes in the bucket policy, and consider Object Lock for state that would be painful to lose.

Keep each state small

One state per environment per component, not one state per company. Small states plan faster, lock less of the team out at once, and limit what a bad change can touch. A typical split is network, data and application, with each one reading the others’ outputs rather than everything living in a single 4,000-resource plan.

Change state with code, not commands

terraform state mv and terraform state rm work, but they leave no record and can’t be reviewed. Terraform now has blocks for the same jobs, and they go through a normal pull request:

  • moved renames or restructures a resource without destroying it.
  • import adopts an existing resource and shows the change in the plan before anything happens.
  • removed stops managing a resource without deleting it.

A reviewer can see exactly what will change, and the next person to read the code can see why.

Catch drift before it catches you

Someone will change something in the console. Run terraform plan -detailed-exitcode on a schedule against every state. Exit code 2 means the real world and the code disagree. Open a ticket for it, the same way you would for any other alert. Drift found on a Tuesday afternoon is a small fix. Drift found during an emergency apply turns into a second incident.

Filed under: files that are scarier than they look.

More writing