What happened
On 18 November 2025, from 11:20 UTC, Cloudflare returned widespread 5xx errors across much of its network. Per its post-mortem, a database permissions change deployed at 11:05 UTC made table access explicit, which caused a Bot Management 'feature file' query to return duplicate rows and more than double in size. That file exceeded a hardcoded runtime limit — the Bot Management system caps machine-learning features at 200 — and the proxy panicked ('thread fl2_worker_thread panicked: called Result::unwrap() on an Err value'). Because the feature file is auto-published network-wide every few minutes, the bad file propagated globally. Cloudflare stopped the bad config at 14:24 UTC, restored core traffic with a known-good file by 14:30, and had all systems normal by 17:06. It states the outage was not caused by any cyber attack.
Why it matters
This is a canonical modern outage: a small internal data change — not an attacker — turned a routine config refresh into a global failure. The blast radius came from three ordinary decisions stacked together: a machine-generated file published to the whole fleet, a hardcoded size assumption, and an `unwrap()` that panicked instead of degrading.
The lesson isn't 'raise the limit to 300.' It's that config and data are part of your deploy surface: anything auto-published fleet-wide needs the same validation, staged rollout, and blast-radius controls as code, and hot-path parsers should fail closed — skip, clamp, or serve last-known-good — never panic on unexpected input. The Staff move is to hunt your own systems for the pattern: an `unwrap()`/assert on data you don't fully control, a global config push with no canary, a buffer sized to today's data.
A machine-generated config file, auto-published to your whole fleet, suddenly violates a size assumption in a hot-path parser. How do you keep that from taking down production?
Probes fail-closed design, config-as-deploy discipline, staged rollout, and blast-radius thinking.