The Day One Render Took Down Every Site

On June 10, a single page render wedged a worker in our shared V8 pool. Within seconds, every site on the server stopped responding — not the one app with the bad render, all of them. Here's what happened, why one render could take down the whole box, and what we changed so it can't cascade the same way again.

This is a blameless post-mortem. The goal isn't to find who shipped the slow render — it's to explain why the system let one render become everyone's outage.

The setup, and why it was a single point of failure

The pool is eight worker subprocesses, shared across every tenant. A render borrows a worker, runs, and hands it back. The whole economic argument for the pool is that capacity is shared — but shared capacity is also a shared failure domain. A worker pinned by one site's render is a worker that no other site can use.

We knew that in the abstract, and we had a watchdog for it: if a worker didn't return within thirty seconds, we'd SIGKILL it and spawn a fresh one. We thought that was enough. It wasn't.

What actually happened

Three things lined up.

A global cache invalidation. A routine source change on one site woke the live-reload watcher, which — first bug — invalidated the compiled-bundle cache for every site, not just the one that changed. The next request to each site now had to recompile its routes from cold.

A recompile storm. Compilation is CPU-heavy. All at once, every site's next request was queued behind a cold compile, and those compiles were all fighting for the same cores. Throughput on the box collapsed — not because rendering was slow, but because almost nothing could get to rendering.

A wedged render. In the middle of that, one render hit a pathological input and went into a tight CPU loop — the kind that never yields and never returns. It pinned its worker.

Now watch the watchdog fail. It was built to catch a worker that hangs — one stuck waiting on something. It fired on a thirty-second timeout and killed the worker. But thirty seconds is an eternity once the queue is backing up, and the watchdog was blind to why a worker was busy: it couldn't tell a slow compile from a wedged render from a genuinely hung call. So it waited the full thirty seconds, killed a worker, and the next queued request — another cold compile, or another hit of the same bad render — took the freed slot and stalled it again.

flowchart TB
  edit["Source edit on one site"] --> inval["Bundle cache invalidated for EVERY site"]
  inval --> storm["Recompile storm: every next request cold-compiles"]
  wedge["Pathological input"] --> pin["Render wedges, pins a worker"]
  storm --> drain["Pool drains, one slot at a time"]
  pin --> drain
  drain --> queue["No backpressure: request queue grows unbounded"]
  queue --> down["Every site times out"]
  classDef bad fill:#fff1f2,stroke:#f43f5e,color:#0c0c0c,stroke-width:1.5px
  class drain,down bad

One slot at a time, the pool drained. With no worker free and no backpressure to reject incoming work, the request queue grew without bound, and every site sharing the pool — which is every site — started timing out.

Why the blast radius was the whole server

The honest answer is that we'd built density without bounding the failure that density enables. Every individual decision was reasonable:

• One shared pool, because per-app pools waste memory.
• A coarse watchdog, because a thirty-second SIGKILL is simple and hard to get wrong.
• A conservative cache invalidation — evict broadly, never risk serving stale code.

Each was fine on its own. Together, they meant a single bad render plus a routine source edit could cascade into a total outage. The watchdog wasn't wrong; it was just the only line of defense, and it operated at the wrong altitude — on whole workers and thirty-second timescales, when the failure was a single render on a sub-second one.

What we changed

We didn't fix this with one change. A shared pool needs defense in depth: bound every render, scope every invalidation, and shed load before it piles up. Each of these shipped behind its own flag so we could enable, measure, and roll it back independently¹.

The keystone is the render deadline. The old watchdog asked "is this worker stuck?" and could only answer with a thirty-second sledgehammer. The deadline asks "has this render run longer than any render should?" and answers inside V8 by terminating just that execution — fast, surgical, and safe to run while other work shares the process. The render that started all this would now be cut off long before the queue even noticed.

The circuit breaker is the backstop: if a slot keeps timing out anyway, we stop sending it traffic instead of letting it absorb-and-fail request after request. And scoped invalidation removes the trigger entirely — a source edit on one site no longer touches its neighbors, so there's no recompile storm to amplify in the first place.

What we took away

Two things.

First, a shared resource needs admission control, not just cleanup. Our only defense was a janitor that swept up wedged workers after the fact. What we were missing was a bouncer — something that bounds each unit of work up front and turns away new work when there's no capacity to do it. Cleanup alone always runs a step behind the cascade.

Second, mitigations have to operate at the altitude of the failure. The bug was a single render on a sub-second timescale; the defense was a whole-worker kill on a thirty-second one. They never had a chance to meet. The fixes that worked are the ones that act on the same thing that fails: a render deadline for a runaway render, a per-slot breaker for a bad slot, per-site invalidation for a per-site edit.

The shared pool is still the right design — it's what lets one box host every site we run. But "the failure mode you inherit" with a shared pool isn't a footnote. It's the thing you have to engineer against hardest, because when it goes, it doesn't take one site down with it. It takes all of them.

---

1. Each mitigation is gated behind its own environment flag — BEXT_RENDER_DEADLINE_MS, BEXT_V8_POOL_CIRCUIT_BREAKER, BEXT_TURBOPACK_SCOPED_INVALIDATION, and the rest — so it can be turned on, watched, and rolled back on its own rather than as one big-bang change. Reliability work you can't reverse is its own kind of risk. ↩