HomeIncident library
Incident library

INCIDENT REPLAY · 2023-08-22

AT&T 911 outage during network testing (four states)

During testing of portions of AT&T's 911 network, a contractor's technician inadvertently disabled part of the network and AT&T's systems did not automatically adjust to route around the disabled portion. 911 service failed in parts of Illinois, Kansas, Texas and Wisconsin for 1 hour and 14 minutes, with over 400 failed 911 calls. AT&T paid $950,000 and accepted a three-year compliance plan in an August 2024 FCC settlement.

Required flowOtherCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: AT&T (AS7018)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • During testing of portions of AT&T's 911 network, a contractor's technician inadvertently disabled part of the network and AT&T's systems did not automatically adjust to route around the disabled portion. 911 service failed in parts of Illinois, Kansas, Texas and Wisconsin for 1 hour and 14 minutes, with over 400 failed 911 calls. AT&T paid $950,000 and accepted a three-year compliance plan in an August 2024 FCC settlement.
  • Who: AT&T (AS7018)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_policy.py): PSAP-LINK-1 (the PSAP side) reaches E911-GW-1 (the 911 gateway) through TEST-FW-1 (the portion of the network disabled for testing), which filters inbound on its outside interface. The intent file names the flow: 911 signalling reaches the gateway on udp/5060.
  • A portion of the network was disabled for a test and traffic did not fail over. Modelled as the disabled firewall's ACL closing while the second path's firewall has no permit for the flow; the intended test has the second path carrying it.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

the change as intended

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot with --policy @att_911_testing_2023/policy.yaml --allow-flow-changes returned pass.

First run, before the fixes this batch prompted

first: fail
  • policy gate: high/critical failure(s) on the 'after' network: 911 signalling reaches the gateway

Pinned in tests/gauntlet/replays-policy.yaml.

the change as pushed

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --policy @att_911_testing_2023/policy.yaml --allow-flow-changes returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • policy gate: high/critical failure(s) on the 'after' network: 911 signalling reaches the gateway
  • NOT reachable (expected reachable)

First run, before the fixes this batch prompted

first: fail
  • policy gate: high/critical failure(s) on the 'after' network: 911 signalling reaches the gateway

Pinned in tests/gauntlet/replays-policy.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

A flow rule names its protocol and port

Every reachability rule probed TCP/80, so an intent about SIP could not tell the push from the intended change: 7 of 11 intended changes failed on main and the one that dropped only the named port passed. protocol: and dst_port: on the rule; the policy gate's reason now carries the violation, not only the policy's name.

Found by: Required-flow family

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: Disabling one portion of a network and discovering that traffic does not fail over is a what-if failure simulation finding, but the public record does not say whether the disabled portion or the missing failover was IP routing or 911 voice-application configuration.

Imported from docs/replays/att-911-testing-2023/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.