INCIDENT REPLAY · 2022-12-21

Verizon Wireless six-state VoLTE 911 outage (flawed security policy update re-applied)

A Verizon Wireless employee re-applied a security policy update file that the company already knew had caused a similar outage in October 2022. The update disrupted 911 VoLTE traffic in Alabama, Florida, Georgia, North Carolina, South Carolina and Tennessee for one hour and forty-four minutes and prevented hundreds of 911 calls from completing. Verizon paid a $1,050,000 civil penalty and accepted a compliance plan in a June 2024 FCC consent decree.

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Verizon Wireless (Cellco Partnership; Verizon AS701 / AS6167)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

A Verizon Wireless employee re-applied a security policy update file that the company already knew had caused a similar outage in October 2022. The update disrupted 911 VoLTE traffic in Alabama, Florida, Georgia, North Carolina, South Carolina and Tennessee for one hour and forty-four minutes and prevented hundreds of 911 calls from completing. Verizon paid a $1,050,000 civil penalty and accepted a compliance plan in a June 2024 FCC consent decree.
Who: Verizon Wireless (Cellco Partnership; Verizon AS701 / AS6167)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

The family template (tools/replays/gen_family_policy.py): PARTNER-1 (the 911 routing partner) reaches E911-GW-1 (the 911 gateway) through CORE-FW-1 (the firewall the security policy was pushed to), which filters inbound on its outside interface. The intent file names the flow: 911 signalling from the partner on udp/5060.
A known-bad security policy was pushed a second time. It is modelled as a deny for SIP ahead of the partner's permit; the intended policy denies SIP from one untrusted range only.
Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

the change as intended

now: pass

On the reconstruction, hammerhead preflight on a before and an after snapshot with --policy @verizon_volte_911_2022/policy.yaml --allow-flow-changes returned pass.

First run, before the fixes this batch prompted

first: fail

policy gate: high/critical failure(s) on the 'after' network: 911 signalling from the partner

Pinned in tests/gauntlet/replays-policy.yaml.

the change as pushed

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --policy @verizon_volte_911_2022/policy.yaml --allow-flow-changes returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

policy gate: high/critical failure(s) on the 'after' network: 911 signalling from the partner
NOT reachable (expected reachable)

First run, before the fixes this batch prompted

first: fail

policy gate: high/critical failure(s) on the 'after' network: 911 signalling from the partner

Pinned in tests/gauntlet/replays-policy.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

A flow rule names its protocol and port

Every reachability rule probed TCP/80, so an intent about SIP could not tell the push from the intended change: 7 of 11 intended changes failed on main and the one that dropped only the named port passed. protocol: and dst_port: on the rule; the policy gate's reason now carries the violation, not only the policy's name.

Found by: Required-flow family

Limits

The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
Why the catalog counts this outage as partly in scope: A known-bad security policy being pushed again is what a pre-push gate that simulates the policy against required flows (here 911 VoLTE signalling) should stop, but the consent decree does not say what kind of device enforces the policy, so whether it is a firewall config the tool could parse is unknown.

PUT THE IDEA TO WORK

Green links. Broken checkout.

Chicago payments network. Four routed metro nodes form a ring; the service gateway has a separate handoff. Route checks and ICMP succeed. The primary service transport fails even though an allow rule appears in the service access list. Investigate the two service networks from R2: 10.201.130.0/24 to 10.201.136.0/24 (TCP/443), and 10.201.131.0/24 to 10.201.137.0/24 (TCP/8443).

Physical links · Service health not shown

Link addressing (5)

R1 R210.201.144.0/30
R2 R310.201.145.0/30
R3 R410.201.146.0/30
R4 R110.201.147.0/30
R4 R510.201.148.0/30

Free to practice. No account required.

Imported from docs/replays/verizon-volte-911-2022/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.