HomeIncident library
Incident library

INCIDENT REPLAY · 2026-07-23

Azure West US: maintenance automation withdrew routes from every WAN-egress device at once

A break-fix repair on one optical network device was expanded by a defective blast-radius analysis system to include all devices carrying egress from a West US datacenter. Routes were withdrawn from all of them simultaneously, severing the datacenter from Microsoft's WAN for about 5 hours; traffic inside the region was unaffected. Automated recovery could not run because connectivity was gone, so rollback was manual.

Fits no familyNetwork automation pushed bad configCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Microsoft Azure (tracking ID ZJV6-SGG)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • A break-fix repair on one optical network device was expanded by a defective blast-radius analysis system to include all devices carrying egress from a West US datacenter. Routes were withdrawn from all of them simultaneously, severing the datacenter from Microsoft's WAN for about 5 hours; traffic inside the region was unaffected. Automated recovery could not run because connectivity was gone, so rollback was manual.
  • Who: Microsoft Azure (tracking ID ZJV6-SGG)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • This incident rests on one or two write-ups (see the catalog's note on 2025 and 2026 dates).
  • Draining is modelled as an outbound deny-all prefix-list on the egress device. Three egress devices; the repair as intended drains one, the push drains all three.
  • Both runs carry --allow-flow-changes, the waiver an operator doing a planned drain passes, so that the intended change passes and the verdict on the push is not the generic tripwire.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
pass for the intended drain, fail for the push (the datacenter subnet is lost from the WAN).
Verdict now
unchanged. The intended drain is also the control for the new idle-router check: the other two egress devices are ECMP members today, not idle.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

the repair as intended: one egress device drained

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned pass.

Pinned in tests/gauntlet/replays-individual.yaml.

what the automation pushed: every egress device drained

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --allow-flow-changes returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • connected subnet(s) reachable before are not reachable after
  • 198.18.0.0/24 on DC-FABRIC-1

Pinned in tests/gauntlet/replays-individual.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: Evaluating the aggregate candidate (all egress devices drained together) instead of each device alone shows the datacenter loses every path to the WAN; this is exactly a lost-redundancy / reachability-regression gate.

Imported from docs/replays/azure-westus-2026/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.