S3 quietly became a coordination service

For most of its life S3 was a place to put files. You could not safely use it to coordinate anything: reads could be stale, and two writers to the same key simply overwrote each other. If you needed a lock, a counter or “only one of us gets to do this,” you added DynamoDB or a database next to it.

That has changed in three steps, and the third one is easy to miss.

Step 1: strong consistency (2020)

Since December 2020, S3 provides strong read-after-write consistency for every request. A successful write is immediately visible to every subsequent read and list. That removed a whole category of “wait and retry until it shows up” code.

Step 2: conditional writes (2024)

In 2024, S3 added conditional writes:

• If-None-Match: * makes a write succeed only if the key doesn’t exist yet. Two writers race, and exactly one wins. The other gets a 412 Precondition Failed.
• If-Match: <etag> makes a write succeed only if the object hasn’t changed since you read it. That’s compare-and-swap, the building block of optimistic concurrency.

Those two headers turn S3 into a coordination primitive. Some things that no longer need a separate service:

• Locks. Create a lock object with If-None-Match. Whoever creates it holds the lock. This is how Terraform’s S3 backend now locks state with use_lockfile, without a DynamoDB table.
• Exactly-once job claims. Workers claim a job by creating claims/<job-id>. Only one claim can succeed.
• Safe manifest updates. Read manifest.json, change it, write it back with If-Match. If someone else changed it first, re-read and try again instead of silently losing their update.

Step 3: S3 as a table store (2024 onwards)

S3 Tables, launched at the end of 2024, store Apache Iceberg tables with S3 managing compaction and maintenance. Iceberg itself depends on atomic commits, which is exactly what conditional writes provide. The result is that “a bucket of files” and “a table you can query with transactions” are now the same storage, with no database server in between.

What this changes

Two practical consequences:

1. Check what your extra infrastructure is for. If a DynamoDB table or a small database exists only to hold locks, leader election or job claims for data that already lives in S3, it may no longer be needed.
2. Design for contention. S3 coordination works well for low to moderate contention: a handful of writers, decisions made in seconds. It isn’t a replacement for a database handling thousands of conflicting updates per second. Every conditional write that fails is still a billed request, and retries need backoff.

The caveat that matters

Locks need expiry. A process that takes a lock and crashes leaves the lock behind. Write an expiry time into the lock object, have other workers break locks that are clearly stale, and use S3 lifecycle rules to clean up anything older than a safe limit.

S3 is still the cheapest, most durable storage most teams have. It’s now also a surprisingly capable place to agree on things.