HomeIncident library
Incident library

INCIDENT REPLAY · 2001-01-23

Microsoft.com / MSN / Hotmail DNS unreachable after edge-router change

At 18:30 PST on 23 Jan 2001 a technician changed the configuration of the routers in front of Microsoft's DNS servers. Microsoft.com, MSN, Hotmail, Expedia and other properties became unresolvable for a large share of the Internet for about 22.5 hours, although the web servers themselves were up. All four authoritative name servers sat behind the same router pair.

Fits no familyOtherCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Microsoft

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • At 18:30 PST on 23 Jan 2001 a technician changed the configuration of the routers in front of Microsoft's DNS servers. Microsoft.com, MSN, Hotmail, Expedia and other properties became unresolvable for a large share of the Internet for about 22.5 hours, although the web servers themselves were up. All four authoritative name servers sat behind the same router pair.
  • Who: Microsoft

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • Microsoft said a technician changed the configuration of the routers in front of its DNS servers and did not say what the change was. We assume the DNS subnet fell out of the routing process on both routers.
  • The lesson usually drawn, that all four name servers sat behind one pair of routers, is a design property and not a change; nothing here audits for it.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
fail, for the cause: the DNS subnet is reachable before and not after from the core, the edge and the upstream.
Verdict now
unchanged.

Quoted from docs/replays/families/individual.md. “First” is the tool before the fixes this replay prompted.

the DNS routers, the core, the edge and an upstream

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • connected subnet(s) reachable before are not reachable after
  • 198.18.0.0/24 on MS-DNS-RTR-1

Pinned in tests/gauntlet/replays-individual.yaml.

the DNS routers, the core, the edge and an upstream (potential microloop)

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail.

This run repeats another pinned run (same snapshots, flags and verdict) to record the potential microloops it finds. A caveat never moves the verdict.

Caveats pinned (some caveat contains each line; a caveat never moves the verdict)

  • MS-CORE-1 -> MS-EDGE-1 -> MS-CORE-1, if MS-CORE-1 has its new next hop while MS-EDGE-1 still has its old one

Pinned in tests/gauntlet/replays-transients.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: A router change that removes outside-to-DNS-server reachability is exactly a before/after reachability regression, and all name servers sharing one router pair is a config-visible single point of failure.

Imported from docs/replays/microsoft-dns-2001/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.