The bug that scared me most never reached a single user

The bug that scared me most never reached a single user.

It was the very beginning of tunyt. We were testing internally, someone opened an event dashboard, and it showed another host's data.

We caught it ourselves, and no user ever saw it. But for a few seconds that didn't matter, because the only question in my head was how many other places this was happening and for how long.

Decision 1: cache at the edge

Not long before, we'd been working on caching, and part of that work was in nginx, in front of the app.

The reasoning was sound. A public event page looks exactly the same for everyone. If nginx can answer it from memory, the request never touches the app or the database at all. It's about the cheapest response you can serve, and during a ticket sale, when everyone is refreshing the same event page, it's exactly the win you want.

So I wrote a rule to cache the event pages. Remember that.

Decision 2: go to the code

When private data shows up in the wrong place, every instinct says the application is leaking it. So that's where I went, and I went all the way down: how the dashboard loads its data, how every query is scoped to the logged-in host, how sessions work. All of it was correct.

So I went back to the browser and watched the network calls, again and again. One thing kept bothering me. The dashboard response was fast. Suspiciously fast, for a page that should have been doing real work for one specific host.

I put debuggers in and ran it locally, and the right host saw the right data every single time. Of course it did. There was no nginx on my machine.

The click

I was doing the same checks for what felt like the tenth time when it clicked: I had done caching on nginx too.

I opened the config and there it was. The regex I'd written to match public event pages also matched the event dashboard pages. Both were being cached as public. Something like this, simplified and not our real config:

# Written to cache public event pages.
# Also matched /events/<id>/dashboard, which is different for every host.
location ~* ^/events/ {
    proxy_cache events;
    proxy_cache_valid 200 5m;
    add_header Cache-Control "public, max-age=300";
    proxy_pass http://app;
}

Why it leaks

The mechanism catches a lot of people. By default, nginx decides that two requests are the same if they have the same scheme, host and URL. That's the cache key. Your cookies aren't in it, and neither is who you're logged in as.

So two different hosts opening the same dashboard URL look identical to the cache. Whoever loads it first fills it, and the next person gets their copy. The app never even sees the second request, which is exactly why it was so fast.

Cache-Control: public makes it worse. It tells every cache along the way, not just nginx, that the response is safe to store and share with anyone.

Decision 3: the fix, and the rule behind it

The immediate fix was to narrow the rule so it matched only the public pages it was written for.

The rule I took away is bigger than the regex. Anything behind a login never goes into a shared cache. There's no "be careful with it". It's never. A personalised response should say private or no-store, and if a cache rule could ever match it, the rule is wrong.

I made it absolute because the failure is lopsided. Get a public page's caching wrong and it's a little slower. Get a private page's caching wrong and you've handed one customer's data to another. Rules with that shape of risk don't get exceptions.

What it left me with

A response that's too fast is a clue. I noticed it early and didn't trust it. Real work takes time, so when a personalised page comes back instantly, something in front of your app answered for it.

When local and production disagree, suspect the difference first. The bug lived in a layer that didn't exist on my laptop, and I could have debugged the application for a week and found nothing, because nothing was wrong with it.

A caching rule is also a security rule. It decides who gets to see a response, and when the rule is a regex, one pattern decides whether a private page is public. That deserves the same review as an auth check. I'd stopped thinking of nginx as part of the system because I wasn't writing it every day, and config is code that just doesn't come with tests.

If I could add one thing to every proxy config I've ever written, it would be a header saying whether the cache answered. One X-Cache-Status: HIT in the network tab would have pointed straight at nginx in seconds.

The regex did exactly what I wrote. I wrote it for the pages I was thinking about, and I never thought about every page that happened to match.