torana: what a 5 MB reverse proxy taught me about latency

torana is a reverse proxy written in Rust for the places where a full proxy is more than you need: the edge of a small service, a sidecar, an embedded device. It ships as a single static binary of about 5 MB, starts in milliseconds and uses rustls for TLS, so there’s no OpenSSL and no runtime dependencies.

The name comes from the torana (तोरण), the ceremonial gateway arch in Indian architecture: a small, sturdy structure that everything passes through.

It’s early (v0.6) and hasn’t been hardened by production traffic, and the README says so in its first paragraph. But building it taught me more about where latency actually comes from than years of tuning other people’s proxies.

The benchmark journey

The benchmark compares torana, Caddy and stock nginx, all proxying the same static backend inside one Docker network, using fortio as the load generator.

For comparison, Caddy on the same setup measured 4,495–4,704 requests a second. The proxy is roughly 5 MiB of memory at idle against Caddy’s 21 MiB, in a 4.9 MB image against 60 MB.

Two lessons:

1. Connection reuse is most of the performance. Version 0.1 opened a new TCP connection to the upstream for every request. Every request paid for a handshake before doing any work. Pooling keep-alive connections roughly quadrupled throughput. No clever parsing, no unsafe code, no custom allocator: just not redoing work.

2. The tail was Nagle’s algorithm. After pooling, throughput was fine but the p99 was poor. Nagle’s algorithm delays small writes so it can combine them into fewer packets, which is great for bulk transfers and bad for request/response traffic. Setting TCP_NODELAY on both the accepted client socket and the upstream socket brought the p99 down to about 31 ms. One socket option was worth more than any amount of tuning.

Correct before clever

Most of the work in a proxy isn’t speed. It’s getting the dull details right, because a proxy that’s subtly wrong is worse than a slow one.

• Retries only when they’re safe. torana retries a different upstream on connection failure, but only for GET, HEAD and OPTIONS requests with no body. Anything that might have been partly sent is never retried, because retrying a POST can charge a card twice.
• Forwarded headers have a trust boundary. torana appends to an incoming X-Forwarded-For rather than replacing it, the same convention as nginx and Caddy. An upstream that uses this header for access control has to read the last hop, not the first, because a client can put anything it likes at the start. The docs say this explicitly, because it’s the kind of thing that becomes a security bug when nobody writes it down.
• mTLS identity that can’t be spoofed. With client certificates required, torana forwards the verified certificate’s fingerprint to the upstream and removes any client-supplied value for that header.
• Reloads that can’t break the running config. On SIGHUP, the new configuration is validated first and swapped in atomically only if it’s valid. Routes, upstreams, health checks and plugins rebuild together.
• Bounded everything. A per-route total timeout (30 seconds by default) returns 504 instead of letting a hung upstream hold a client forever. Shutdown drains in-flight requests for up to 15 seconds and then exits.

Plugins without trust

torana can run request filters compiled to WebAssembly. They run in a real sandbox: no filesystem, no network, no processes, and a fuel limit, so a plugin with an infinite loop runs out of fuel instead of taking the proxy down. It’s opt-in at compile time, so the default binary doesn’t carry the runtime.

Saying what it doesn’t do

The README has a section called “What it doesn’t do (yet)”: no HTTP/3, no HTTP/2 or TLS to upstreams, no circuit breaker, and no mTLS combined with ACME on the same listener. It ends with “If you need these today, use Caddy, nginx, or Envoy.”

That section matters more than the feature list. The benchmark notes include two consecutive runs instead of the better one, and warn that Docker Desktop on macOS runs in a VM, so only the relative numbers mean anything. A small tool earns trust by being precise about its limits. That’s true of proxies, and of most infrastructure.