The obvious fix would have worked. That's why I didn't take it.

In July 2026 tunyt's database hit its ceiling, and there was an obvious fix sitting right there.

Find the slowest query. Add an index. Watch the graph drop. Everyone relaxes.

It probably would have worked, for a while, and that's why I didn't take it. The fix wasn't wrong. It was going to work just well enough to stop us asking the real question.

What saturation looks like

Every connection in the pool was busy, and every request that needed one waited in line behind them. Pages that had nothing to do with the cause got slow too, because they were queueing for the same connections.

That detail matters most. When unrelated pages slow down together, you're not looking at a slow query. You're looking at a shared resource that ran out.

Decision 1: don't chase the query

Speed up the slowest query and a different one becomes the slowest. Traffic grows, the pool saturates again, and you're back adding another index. It's whack-a-mole, and every round feels like progress.

So instead of asking which query was slow, I asked why one part of the product could take the whole database down with it.

A little maths

Every connection pool obeys one law:

connections in use ≈ requests per second × seconds each request holds a connection

That's Little's law, and it's the most useful equation I know for backend work. A page that loads 20 times a second and holds a connection for a quarter of a second each time keeps about 5 connections busy, permanently.

Now make that page run fourteen queries at once.

What was actually happening

Our organiser dashboard needed fourteen pieces of data, and the first version fetched all fourteen in parallel. One page load took fourteen connections.

The results were cached, so most loads should never have reached Postgres. But caches expire. When ours did, every request in that window missed it, and every one of them ran all fourteen queries itself. We saw two identical waves of the same queries hit the database together. That's a cache stampede: the cache protects you right up until it expires, and then everyone rushes the database at once.

Meanwhile the analytics worker was running its own jobs against the same database, five at a time.

No single query was the problem. Nothing in the system said how much of the database any one part of it was allowed to use.

Decision 2: a budget, not a bigger database

The other easy answer was a bigger database, with more connections and more headroom.

Look at the equation again. Demand grows with traffic. A bigger pool moves the ceiling up, but nothing stops the next fan-out or the next stampede from filling it too. And Postgres connections aren't free, because each one is a whole server process, so you can't keep adding them.

So we set one number for how many connections the whole deployment may hold, and split it. The API gets one slice and the workers get a smaller one. A busy worker can't starve the API any more, and the API can't starve the workers. "The database is down" becomes, at worst, "the analytics are a bit slow", which is a much better day.

Decision 3: put a ceiling on fan-out

No more Promise.all over an open-ended list of queries. How many run at once is capped, and no longer depends on how much data a page happens to want.

// Run fn over items with at most `limit` in flight at once.
async function mapLimited<T, R>(items: T[], limit: number, fn: (item: T) => Promise<R>) {
  const out = new Array<R>(items.length);
  let next = 0;
  const lanes = Array.from({ length: Math.min(limit, items.length) }, async () => {
    while (next < items.length) {
      const i = next++;
      out[i] = await fn(items[i]);
    }
  });
  await Promise.all(lanes);
  return out;
}

Decision 4: one request rebuilds, everyone else waits politely

When the cache misses, the first request takes a short Redis lock and rebuilds the value. Everyone else keeps getting the last good copy meanwhile. There's also a guard inside each process, so a server doesn't race itself for the lock.

// Simplified. The real one also stamps an expiry so stale values can be detected.
async function getDashboard(eventId: string) {
  const key = `cache:dashboard:${eventId}`;
  const cached = await redis.get(key);
  if (cached && !isStale(cached)) return JSON.parse(cached);

  // Only one request rebuilds. The rest serve the last good copy.
  const won = await redis.set(`${key}:lock`, "1", "PX", 10_000, "NX");
  if (!won && cached) return JSON.parse(cached);

  const fresh = await buildDashboard(eventId);
  await redis.set(key, JSON.stringify(fresh));
  return fresh;
}

The trade-off is that some people see data a few seconds old while it rebuilds. For an analytics dashboard that's an easy trade. Nobody's decision changes because a chart is five seconds behind, and everyone's day changes when the database is down.

Decisions 5 to 7: the small ones

Stop recomputing the past. A finished day of analytics never changes, so completed days now come from a daily rollup table. Only today is computed live.

The background never wins. The analytics worker went from five concurrent jobs to three. That's slower analytics, on purpose. Background work shouldn't be able to win a fight with the people actually using the product.

Indexes that don't block writes. When we do add an index to a busy table, it's built with CREATE INDEX CONCURRENTLY. It's slower to build, but it doesn't lock writes while it runs. A fix for slow reads shouldn't take down the writes.

Why I'd make the slow call again

The index would probably have made the graph go green, and it would have taught the system nothing. The next stampede, or the next dashboard with fourteen queries, would have found the same hole.

Budgets feel like paperwork until the day they're the reason the rest of the app keeps working while one page misbehaves. Every one of these decisions is now a written rule that the team checks before shipping a list endpoint, a cache or a worker, and our coding agents read the same rules. The budgets live in one table, and changing a number needs a reason in the pull request, because some of those numbers were paid for with this incident.

The database had no budget, so one page spent all of it.