HomeIncident library
Incident library

INCIDENT REPLAY · 2021-10-25

KT nationwide wired and wireless outage

At about 11:20 KST on a Monday, KT's wired and wireless internet, and services riding on it (card payments, stock trading, online classes, delivery apps), failed nationwide. Korea Herald reported disruption of up to 85 minutes (about 89 minutes is also widely reported); BleepingComputer reported roughly 40 minutes for the core of it and that all of KT's ~16.5 million customers were exposed. KT first blamed a DDoS attack and then corrected itself to a routing error. The Ministry of Science and ICT (MSIT) investigated and published its finding on 29 October 2021.

Hand-built replaysIGP / MPLS core configCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: KT Corp (Korea Telecom, AS4766), South Korea; work performed by a subcontractor

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • KT's wired and wireless networks failed nationwide for 89 minutes from 11:16 on 25 October 2021.
  • The ministry found the cause was an incorrect command entered during maintenance on router equipment in Busan: the word exit was left out of the routing configuration script. Routing information meant for BGP was sent into IS-IS, the protocol KT uses internally, and spread across the country in under a minute.
  • The work was approved for 01:00 to 06:00 and was done in the daytime, by a subcontractor's staff, with no KT manager present.
  • The ministry also found that KT had no safeguard in place to stop wrong routing data spreading through the protocol.

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The missing exit is modelled by its effect. On Cisco IOS, configuration lines meant for router bgp that are typed while still inside router isis mostly fail to parse; the one that does something is redistribute, and "BGP routes into IS-IS" is what the ministry described. after adds one line, redistribute bgp 4766, under router isis on the Busan router.
  • Five routers: an upstream that sends a BGP table, the Busan edge, two cores and a Seoul edge with subscribers. Thirty-two /20s stand in for the table. AS 4766 is KT's real number; the upstream's AS and every address are from the documentation and benchmarking ranges. capacity.yaml declares 20 routes for each core, scaled to the corpus; KT's real limits are not public.
  • Cisco IOS syntax. KT's vendors are not on the record.

tools/replays/gen_kt_2021.py writes the corpus.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

Hammerhead's own first-versus-now record

First verdict
pass, 100% parse coverage. IS-IS redistribution was parsed and never simulated.
Verdict now
fail: "route-table growth: 2 device(s) would hold more than twice the routes they hold now — CORE-1 40 routes (was 7), …" (3 devices and 36 (was 4) until 2026-09, when the PE loopbacks made passive under router isis were put into the LSPs and the iBGP session between the PEs came up); with a declared limit, the capacity gate names it instead.
Fixed?
Yes, two fixes. IS-IS redistribution is simulated, and a table that at least doubles is a finding with NOTHING declared (--allow-table-growth). Rogers 2022 at defaults moved from pass to fail with it.

Quoted from docs/replays/README.md. “First” is the tool before the fixes this replay prompted.

core capacity declared

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot with --capacity @kt_2021/capacity.yaml returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • route-table capacity
  • CORE-1
  • CORE-2

Pinned in tests/gauntlet/replays.yaml.

at defaults, with nothing declared

now: fail

On the reconstruction, hammerhead preflight on a before and an after snapshot returned fail, for the reasons below.

Reason text pinned (some reason contains each line)

  • route-table growth: 2 device(s) would hold more than twice the routes they hold now
  • CORE-1 40 routes (was 7)

Pinned in tests/gauntlet/replays.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

IS-IS redistribution is simulated

redistribute static|connected|bgp under router isis was parsed and never read. Through a route-map it is still not applied (registry entry isis-redistribute-route-map).

Found by: KT

Route-table growth with nothing declared (--allow-table-growth)

A table that at least doubles and gains 25 routes on a device present in both snapshots. A declared limit replaces it for the device it names.

Found by: KT, Rogers

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as in scope: A pre-push simulation of the candidate router config would show external routes landing in the internal routing domain and flooding nationwide, which is exactly a routes-flooding-a-table finding.

Imported from docs/replays/kt-2021/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.