INCIDENT REPLAY · 2021-10-13

OVHcloud backbone outage from router misconfiguration during maintenance

At about 07:20 UTC OVH's global backbone failed for roughly an hour, taking customer services offline in all regions. The company was increasing DDoS-scrubbing capacity at its Vint Hill (US-East) data centre when a router was misconfigured; OVH isolated the device, corrected the settings and cancelled the change. The precise mechanism (my unverified recollection of OVH's later post-mortem is that Internet routes were injected into the IGP) was not confirmed in the sources opened for this entry. Slice note: OVHcloud is a hosting/cloud provider with its own global backbone rather than a telecom carrier; kept here because the failure was a backbone routing change, but it may overlap a cloud-provider slice.

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: OVHcloud (France), AS16276

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

At about 07:20 UTC OVH's global backbone failed for roughly an hour, taking customer services offline in all regions. The company was increasing DDoS-scrubbing capacity at its Vint Hill (US-East) data centre when a router was misconfigured; OVH isolated the device, corrected the settings and cancelled the change. The precise mechanism (my unverified recollection of OVH's later post-mortem is that Internet routes were injected into the IGP) was not confirmed in the sources opened for this entry. Slice note: OVHcloud is a hosting/cloud provider with its own global backbone rather than a telecom carrier; kept here because the failure was a backbone routing change, but it may overlap a cloud-provider slice.
Who: OVHcloud (France), AS16276

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

The mechanism is not confirmed. The sources say a router was misconfigured during maintenance at Vint Hill and the backbone failed. We assume BGP routes were redistributed into the IGP, which is the recollection noted in the catalog and the shape of Rogers 2022 and KT 2021. If OVH's own account says otherwise this replay is of the wrong fault.
One edge, two cores, a transit sending 120 prefixes.
Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict: pass. No flow is lost when a table multiplies.
Verdict now: fail: both cores would hold 129 routes where they hold 8 (the route-table growth check added earlier in this batch).

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

the edge, two cores and a transit

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

route-table growth: 2 device(s) would hold more than twice the routes they hold now

Pinned in tests/gauntlet/replays-individual.yaml.

Limits

The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
Why the catalog counts this outage as in scope: A single router config change during planned maintenance that collapses backbone routing is the canonical case a pre-push simulation of route-table growth and lost reachability is meant to catch.

Not fixed: The mechanism behind OVH 2021

Not confirmed by the sources. The replay assumes BGP into the IGP and says so.

Recorded against: OVH

Imported from docs/replays/ovh-2021/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.