Incident library
134 documented outages, each rebuilt as a small network from the public record and run through Hammerhead. Read what happened, what the record says, what we had to assume, and what the tool reported on the reconstruction.
How to read these
- Every replay is a reconstruction from the public record, not the operator's configuration. No operator has published theirs; device names, vendors and most addresses are stand-ins.
- A verdict is what Hammerhead reported on that reconstruction. It is not a claim that the tool would have prevented the outage.
- Summaries are paraphrased from the cited sources. Each incident shows the confidence the Hammerhead catalog gives its summary.
- A change that trips a device-count threshold on a network this small is not counted as a catch: every preflight run uses
--max-blast-radius 100, so a verdict reflects a gate that fired for a reason. - Passes stay listed as passes. Hammerhead's replay notes say all 134 are flagged for their cause: 133 from the configuration change, and Microsoft WAN, January 2023 from the command that was run, with its vendor and command assumed because Microsoft published neither.
Softaculous / Virtualizor hijack of a Hetzner /24
From 20:57 UTC a /24 more-specific of Hetzner space hosting Softaculous update, client and billing sites was announced with path ... 6204 62390 24940, forging Hetzner as origin. Hetzner's ROA for the /16 allowed lengths to /24, so the route validated. The attacker obtained a valid TLS certificate and delivered a malicious Virtualizor update to some installations.
audit finding × 12026-07-23Fits no familyAzure West US: maintenance automation withdrew routes from every WAN-egress device at once
A break-fix repair on one optical network device was expanded by a defective blast-radius analysis system to include all devices carrying egress from a West US datacenter. Routes were withdrawn from all of them simultaneously, severing the datacenter from Microsoft's WAN for about 5 hours; traffic inside the region was unaffected. Automated recovery could not run because connectivity was gone, so rollback was manual.
fail × 1pass × 12026-06-16Mis-originationRcom (India) Telegram block leaks globally
After India ordered a nationwide Telegram block (16 to 22 June 2026), AS18101 began originating Telegram prefixes such as 91.108.4.0/22 at 07:17 UTC, and the routes escaped to the global table through its transit. When Telegram responded with more-specifics, AS18101 originated more-specifics too at 16:14 UTC. Kentik says only about 1.6 percent of BGP sources carried the bogus route because RPKI origin validation filtered it, but a small share of global Telegram traffic was pulled to India and dropped.
fail × 3pass × 1audit finding × 12026-02-20Fits no familyCloudflare BYOIP prefixes withdrawn by buggy cleanup task
A scheduled sub-task written to automate removal of BYOIP prefixes queried an API without a required value, got every prefix back, and treated them all as queued for deletion. About 1,100 customer prefixes (roughly 25% of BYOIP) were withdrawn from BGP, making those customers' services unreachable; full restoration took about 6 hours.
fail × 12026-01-22Fits no familyCloudflare Miami route leak (automated policy change)
An automated routing-policy change meant to remove Bogota prefixes deleted prefix-list filters from several BGP export policies on a Miami Junos router. The remaining terms matched every route of type 'internal', so the router advertised IBGP-learned IPv6 prefixes to transits and peers for 25 minutes. External traffic was pulled through Miami, congesting the Miami-Atlanta backbone, and about 12 Gbps was discarded by firewall filters.
fail × 22026-01-02Route leakCANTV (AS8048) provider-to-peer leak, Venezuela
Between 15:30 and 17:45 UTC AS8048 re-advertised routes learned from a transit provider to a peer, breaking valley-free routing. It drew attention because of political events the same day; Cloudflare counted eleven similar leaks by the same AS since December.
fail × 2audit finding × 12025-09-18Required flowOptus Triple Zero outage during firewall upgrade
From about 00:30 to 13:30 AEST on 18 September 2025 (almost 14 hours) emergency calls from Optus customers in South Australia, Western Australia, the Northern Territory and parts of NSW intermittently failed while ordinary calls kept working. Optus first said about 600 calls were affected; the independent Schott review found about 75 percent of 605 attempts failed (The Register: 455 calls did not go through). The outage was initially linked to three and later four deaths; accounts of the review tie two deaths to failed calls. Optus's monitoring excluded Triple Zero calls so no alarm fired, and five customer warnings to the call centre were not escalated. A Senate inquiry hearing was held on 3 November 2025, and Wikipedia reports the ACCC began Federal Court action in July 2026.
fail × 1pass × 12025-09-05Required flowBART systemwide shutdown after overnight network upgrade
All service was suspended on a Friday morning until about 11:45, affecting roughly 40,000 commuters, after an overnight upgrade to the communications and computer network left the control centre unable to monitor trains.
fail × 1pass × 12025-07-14Fits no familyCloudflare 1.1.1.1 global outage: resolver prefixes attached to a non-production service topology
On June 6 the 1.1.1.1 resolver prefixes were mistakenly included in the service topology of a new, not-yet-live Data Localization Suite service. On July 14 a change to that topology triggered a global refresh, and the resolver prefixes were withdrawn from Cloudflare's routers worldwide. 1.1.1.1 was unreachable for most users for 62 minutes.
fail × 1pass × 12025-04-01Route leakVoxility (AS3223) DDoS-mitigation leak through Lumen
From 07:35 UTC for about 20 minutes AS3223 leaked 31,258 routes learned from peers to its transit provider Lumen. Traffic was drawn to Bucharest, where it was misdirected or dropped for lack of capacity.
fail × 2audit finding × 12025-03-18RPKI and driftNorth Korea knocked offline by its own ROA
At 09:30 UTC a ROA was published with a maximum length of /22 while the country's four routes are announced as /24s, making them RPKI-invalid. Carriers doing origin validation rejected them within minutes and reachability fell for over 16 hours until corrected ROAs appeared.
fail × 1pass × 12025-02-11Route leakUPX Technologies (AS52863) path leak
From 18:16 UTC for about 15 minutes AS52863 passed 21,119 routes learned from a provider to its other upstreams through a sibling AS. Kentik found more traffic was dropped than misdirected.
fail × 2audit finding × 12025 (Qrator write-up dated 2025-12-15; event dates not given)RedundancyMalformed OTC attribute causes session resets
Qrator traced unexplained eBGP session resets to updates whose OTC attribute had an Extended Length flag inconsistent with its length encoding. Implementations differed: some discarded the route, some rewrote the attribute, some reset the session as RFC 4271 originally required.
pass × 1audit finding × 12024-09-26Route leakUztelecom route leak through Rostelecom
Starting 06:25 UTC and lasting about 40 minutes, Uztelecom leaked 3,144 routes (371 seen by 50 or more RouteViews peers) to its transit provider Rostelecom, inserting itself as an illegitimate intermediary. Traffic for prefixes of Amazon, Cloudflare and others was misdirected through Russia and Central Asia. No new more-specifics were created.
fail × 2audit finding × 12024-07-30Required flowAzure Front Door / DDoS: latent network config sent non-European traffic to the European scrubbing system
After a routine DDoS mitigation, a latent network configuration issue caused traffic from outside Europe to be routed to the DDoS protection system inside Europe, congesting it. Azure Front Door, CDN, the Azure portal and Microsoft 365 services saw errors and latency for roughly 2 hours (with a longer tail). Microsoft's fix was to add missing configuration on network devices so an issue in one geography cannot spread.
fail × 1pass × 12024-07-18Required flowAzure Central US: incomplete storage 'Allow List' blocked VM hosts from their disks
An update workflow regenerated the Allow List of network address ranges permitted to reach Storage scale units in Central US, but the list omitted address information for many VM hosts. Those hosts lost access to their virtual disks, VMs went down, and dependent services including Microsoft 365 were impaired for about 14 hours.
fail × 1pass × 12024-06-27Mis-originationCloudflare 1.1.1.1 hijack and route leak (Eletronet, Nova Rede)
AS267613 originated 1.1.1.1/32 and, separately, AS262504 leaked 1.1.1.0/24 upstream where AS1031 propagated it. At least one Tier 1 treated the /32 as a blackhole route. Over 300 networks in 70 countries had 1.1.1.1 unreachable or very slow for about 7.5 hours.
fail × 3pass × 1audit finding × 12024-02-22Required flowAT&T Mobility nationwide wireless outage
At 2:42 AM an employee placed a new, misconfigured network element into production during a routine night maintenance window to expand capacity. Downstream elements propagated the error and three minutes later the network entered a protect mode that disconnected all devices from voice and 5G data. The rollback took close to two hours, but full restoration took at least 12 hours because device re-registration overwhelmed the registration systems. The FCC counted more than 125 million devices affected, more than 92 million voice calls blocked and more than 25,000 calls to 911 call centers prevented.
fail × 1pass × 12023-11-22Layer 2AMS-IX 'LACP leakage' outage
The Amsterdam peering platform was degraded for more than four hours; traffic fell from roughly 10 Tbps to about 2.1 Tbps. AMS-IX presented the post-mortem at RIPE 87.
audit finding × 22023-11-08Hand-built replaysOptus: a partner's table grew tenfold past configured prefix limits
A partner's table grew tenfold; about 90 PE routers hit their prefix limits and cut themselves off.
fail2023-08-22Required flowAT&T 911 outage during network testing (four states)
During testing of portions of AT&T's 911 network, a contractor's technician inadvertently disabled part of the network and AT&T's systems did not automatically adjust to route around the disabled portion. 911 service failed in parts of Illinois, Kansas, Texas and Wisconsin for 1 hour and 14 minutes, with over 400 failed 911 calls. AT&T paid $950,000 and accepted a three-year compliance plan in an August 2024 FCC settlement.
fail × 1pass × 12023-06-29Fits no familyGitHub: secondary edge path misconfiguration exposed by failover test
GitHub ran a live failover test of a second Internet edge facility. A network path configuration problem on the secondary side meant traffic could not be carried properly, and GitHub was unavailable for much of North and South America for about 32 minutes.
fail × 12023-06-02RedundancyUnknown-attribute update resets Juniper sessions (COLT and others)
A route carried a path attribute for a not-yet-standardised feature. Routers that did not understand it passed it on untouched; Junos routers that partly understood it treated it as malformed and reset the whole session, repeatedly, cutting off distant networks.
pass × 1audit finding × 12023-05-25Route leakAngola Cables leak slows Australia to US-East traffic
From about 10:30 UTC for roughly three hours, Angola Cables leaked hundreds of routes, many IPv6, through an exchange route server with short AS paths, so they were preferred. Australian access to AWS and Akamai on the US east coast was badly degraded.
fail × 5audit finding × 12023-03-12Route leakIran TIC leaks 20,000+ routes learned from Telecom Armenia
At 13:36 UTC, TIC leaked more than 20,000 prefixes it had learned from Telecom Armenia to its transit provider. The leak lasted about 20 minutes and misdirected traffic from several countries including Russia, Australia, Azerbaijan, Great Britain and Germany. Kentik notes it coincided with, and may have caused, a large internet disruption inside Iran.
fail × 2audit finding × 12023-02-05RedundancyTIM Italy international interconnection failure
On Sunday 5 February 2023 TIM fixed-line customers from Milan to Palermo lost or had degraded internet access for several hours; Cloudflare Radar showed nationwide traffic loss on AS3269 and AS16232. An Internet Society Pulse analysis (returned by search; the page itself refused my fetch) noted that TIM depends on Sparkle alone for international connectivity, so a fault on that interconnection had no alternative path.
pass × 1audit finding × 12023-01-25Hand-built replaysMicrosoft WAN: an IP change made every WAN router recompute
A command during an IP change made every WAN router recompute at once. The end state was fine.
fail × 1unknown × 1pass × 2caught for the command, vendor assumed2022-12-21Required flowVerizon Wireless six-state VoLTE 911 outage (flawed security policy update re-applied)
A Verizon Wireless employee re-applied a security policy update file that the company already knew had caused a similar outage in October 2022. The update disrupted 911 VoLTE traffic in Alabama, Florida, Georgia, North Carolina, South Carolina and Tennessee for one hour and forty-four minutes and prevented hundreds of 911 calls from completing. Verizon paid a $1,050,000 civil penalty and accepted a compliance plan in a June 2024 FCC consent decree.
fail × 1pass × 12022-08-17Accepting networkCeler Bridge hijack via forged IRR and forged AS path
The attacker created AltDB route objects so QuickhostUK's upstream would build filters permitting the Amazon /24, then announced it with Amazon's AS14618 forged as origin so origin validation passed. The route was up from 19:39 UTC with several re-announcements until 22:07 UTC; about $235,000 was stolen.
audit finding × 12022-07-26 (21:25 UTC, to 2022-07-27 09:39 UTC)Mis-originationRostelecom originates an Apple /19
For 12 hours 14 minutes AS12389 originated part of Apple's 17.0.0.0/8. Apple had no ROA for the space and responded about five hours in by announcing a more-specific /21.
fail × 2pass × 1audit finding × 12022-07-08Hand-built replaysRogers: a deleted route filter flooded the core
A deleted route filter let full BGP tables flood the core; routers ran out of memory. No flow is lost in the model.
fail2022-07-02Required flowKDDI (au) 61-hour nationwide outage
From 01:35 JST on 2 July to 15:00 on 4 July 2022 (61 hours 25 minutes) KDDI's voice and data services were degraded nationwide. KDDI's own report counts about 22.78 million voice (VoLTE) users and 7.65 million or more data users; MIC's guidance cites about 23.16 million voice and 7.75 million data users. ATMs, logistics, connected cars, weather data collection and emergency calling were affected; press reports put exposed lines at up to 39 million.
fail × 1pass × 12022-06-23 (publication; pattern ongoing for years)Fits no familyMikroTik prepend misconfiguration makes single-digit ASNs appear as hijackers
Operators typed a prepend count where MikroTik RouterOS expects an AS number, so paths appeared to contain or originate from AS2, AS3 and so on. Hijack detectors flagged the single-digit ASNs, and paths with the bogus AS could be dropped by networks filtering on it.
fail × 2pass × 12022-06-21Hand-built replaysCloudflare: policy terms reordered in 19 data centres
Policy terms reordered; a reject that matches everything moved ahead of the term advertising site-local prefixes.
fail2022-04-05Layer 2AMS-IX loop from wrong port configuration before planned maintenance (2022 statement)
AMS-IX published a statement dated 6 April 2022 describing a short outage in which 74.5% of all BGP sessions on the Amsterdam platform were interrupted between 10:20 and 10:45. During preparation for a routing test ahead of planned maintenance, an engineer's action put the wrong configuration on a port and created a loop on the production network; it was cleared by disabling the looping ports. Date caveat: the page does not state the incident day (it says members were told the root cause 'yesterday'), so 5 April 2022 is my best estimate; a search summary blended this statement with the very similar 13 May 2015 loop, and a DCD report returned by search (expected 7.3 Tb/s falling to just over 3 Tb/s) could not be opened to confirm which event it covers.
audit finding × 12022-03-28Mis-originationRTComm hijacks Twitter prefix (Russia block leak)
From about 12:05 to 12:50 UTC AS8342 originated Twitter's /24, apparently to enforce the Russian block, and the route leaked to peers and upstreams. Twitter had published a ROA after the Myanmar event, so networks doing origin validation dropped it and spread was limited.
fail × 3pass × 1audit finding × 12022-02-03RPKI and driftKLAYswap / Kakao SDK hijack
Announcements from AS9457 covering Kakao prefixes redirected requests for a JavaScript SDK that KLAYswap's site loads. The attacker obtained a valid TLS certificate for the hostname while holding the route and served modified code that diverted users' transfers, about $1.9 million, over roughly two hours.
exit-code check × 22021-10-25Hand-built replaysKT nationwide wired and wireless outage
At about 11:20 KST on a Monday, KT's wired and wireless internet, and services riding on it (card payments, stock trading, online classes, delivery apps), failed nationwide. Korea Herald reported disruption of up to 85 minutes (about 89 minutes is also widely reported); BleepingComputer reported roughly 40 minutes for the core of it and that all of KT's ~16.5 million customers were exposed. KT first blamed a DDoS attack and then corrected itself to a routing error. The Ministry of Science and ICT (MSIT) investigated and published its finding on 29 October 2021.
fail × 22021-10-13Fits no familyOVHcloud backbone outage from router misconfiguration during maintenance
At about 07:20 UTC OVH's global backbone failed for roughly an hour, taking customer services offline in all regions. The company was increasing DDoS-scrubbing capacity at its Vint Hill (US-East) data centre when a router was misconfigured; OVH isolated the device, corrected the settings and cancelled the change. The precise mechanism (my unverified recollection of OVH's later post-mortem is that Internet routes were injected into the IGP) was not confirmed in the sources opened for this entry. Slice note: OVHcloud is a hosting/cloud provider with its own global backbone rather than a telecom carrier; kept here because the failure was a backbone routing change, but it may overlap a cloud-provider slice.
fail × 12021-10-07Fits no familyTelia Carrier routing-policy update blackholes aggregates
At 16:00 UTC a routine routing-policy update in the Telia Carrier IP core caused traffic to prefixes inside its aggregates to be blackholed for about 65 minutes. Catchpoint reported BGP events on roughly 500k IPv4 and 32k IPv6 networks (more than half of the IPv4 table) and knock-on impact at Cloudflare, Fastly, Equinix Metal, Firebase and NS1. The config was rolled back at 17:05 UTC and services recovered gradually. Note: the outages-list thread dates the event 7 October; Catchpoint's write-up (published the next day) was summarised to me as 8 October.
fail × 22021-10-04Hand-built replaysFacebook: backbone taken down by a maintenance command
A maintenance command took the backbone down; DNS servers then withdrew their own anycast routes.
fail2021-04-16Mis-originationVodafone Idea (AS55410) mis-originates over 30,000 prefixes
At about 13:48 UTC AS55410 originated more than 30,000 prefixes that were not its own. Kentik measured a 13-fold surge of inbound traffic to the network. The bulk was withdrawn within minutes, with stragglers lingering.
fail × 2audit finding × 12021-03-24Layer 2Georgia Tech: building VLAN spanned into the Coda data centre loops
For about an hour (roughly 14:30-15:30) users could not reach research clusters in the Coda data centre, and connectivity problems were widespread across campus as core network equipment was affected.
audit finding × 12021-02-13Mis-originationCablevision Mexico (AS28548) leaks about 7,200 prefixes
For about 12 minutes from 04:36 UTC an AS that normally announces about 150 routes announced over 7,200 IPv4 prefixes, many of them more-specifics, through one of its upstreams.
fail × 2pass × 1audit finding × 12021-02-05Mis-originationMyanmar Twitter block leaks (Campana MYTHIC)
After the junta ordered Twitter blocked, AS136168 originated Twitter's /24 from 15:51 to 18:58 UTC. Only five networks accepted it, so impact stayed regional.
fail × 2pass × 1audit finding × 12020-12-17Layer 2Georgia Tech campus outage: spanning-tree loop propagates through shared firewall
From about 13:20 to 15:45 large parts of the campus network, including parts of the wireless network (eduroam), ResNet and the College of Computing, had limited or no connectivity to campus and Internet resources.
audit finding × 12020-09-29Mis-originationTelstra AS1221 mis-origination of ~500 prefixes (ProtonMail, Quad9 and others)
From 17:50 UTC on 29 September 2020 (early 30 September in Australia) AS1221 originated about 472 to 500 IPv4 prefixes belonging to 266 other networks in about 50 countries, mostly US and UK. The event peaked for about three hours. ProtonMail said about 30 percent of the internet's paths to it pointed at Telstra and that it incurred financial loss; Quad9, Cloudflare and Google Sydney ranges were also named.
fail × 3audit finding × 12020-08-30Hand-built replaysCenturyLink / Level 3 global outage from a Flowspec rule
From about 10:03 UTC for roughly 4.5 hours, Level 3's backbone dropped BGP sessions repeatedly; Cloudflare measured a 3.5% fall in global Internet traffic. Because the rule was distributed by BGP, sessions flapped as the rule was re-learned each time they recovered.
fail × 1unknown × 12020-08-24Accepting networkPremier DC (AS42910) repeated route leak
Qrator Radar, as summarised by MANRS, recorded a leak of about 1,403 prefixes that recurred several times over more than four hours.
audit finding × 12020-07-30Mis-originationTulix (AS10990) optimizer more-specifics hit Rogers, Charter, TWC
At 01:15 UTC AS10990 originated 179 more-specific prefixes of North American cable operators for about four minutes, then 145 again for about 52 minutes.
fail × 2pass × 1audit finding × 12020-07-21Route leakAS264462 leaks IX-learned routes to upstreams (Brazil)
From 09:14 to 12:07 UTC a small Brazilian ISP re-advertised 13,046 prefixes learned at the Sao Paulo exchange to its transit providers, placing itself in paths for 72 ASes.
fail × 2audit finding × 12020-07-17Hand-built replaysCloudflare: a policy term lost its only match and drew the backbone into Atlanta
A policy term's only match was deactivated, so it set local-preference 200 on every route and pulled the backbone's traffic into Atlanta.
fail × 1pass × 12020-06-15Fits no familyT-Mobile US nationwide voice/text outage (OSPF link-weight misconfiguration)
While integrating new routers in the Southeast, T-Mobile set Open Shortest Path First (OSPF) link weights wrongly on an already-active router that was not built to carry call-signalling traffic. When a leased fiber transport link failed at 12:33 PM EDT, traffic followed the lowest-weight path to that router, which dropped it (the dropped traffic was MPLS), isolating the Atlanta market; a latent IMS registration software flaw then turned that into a nationwide registration storm. The outage lasted over twelve hours; the FCC estimated at least 41% of calls on the network failed, including at least 23,621 failed 911 calls.
fail × 3pass × 22020-06-09Accepting networkIBM Cloud global outage: external provider flooded incorrect routes
IBM Cloud sites worldwide lost connectivity for about 3 hours. IBM said an external network provider sent it incorrect routing, which caused severe congestion across the IBM Cloud network.
audit finding × 32020-04-01Mis-originationRostelecom announces more-specifics of cloud and CDN space
From 19:28 UTC AS12389 originated thousands of more-specific prefixes (MANRS counts 4,569 unique; BGPmon over 8,800) of major cloud and CDN networks for about an hour.
fail × 2pass × 1audit finding × 12019-06-24Hand-built replaysVerizon / DQE / Allegheny: optimizer more-specifics leaked through a customer
A BGP optimizer's more-specifics leaked through a customer to a transit with no filter and no prefix limit.
fail2019-06-06Route leakSafe Host leaks 70,000 routes to China Telecom
From 09:44 UTC Safe Host leaked over 70,000 routes learned from peers and providers to China Telecom, which announced them onward. Traffic for large European mobile networks flowed through China Telecom for more than two hours.
fail × 2audit finding × 12019-05-08Mis-originationTWNIC Quad101 public DNS hijacked from Brazil
At about 15:08 UTC a small Brazilian AS originated the /24 of Taiwan's public resolver. Its provider passed it on, and some traffic went to Brazil for about three and a half minutes.
fail × 2pass × 1audit finding × 12018-11-12Hand-built replaysMainOne leak of Google prefixes to China Telecom
Between about 21:10 and 22:35 UTC, MainOne announced routes it had learned from Google (it peers with Google at the IXPN exchange in Lagos) to its provider China Telecom, which accepted and re-announced them; some large transit networks then preferred the leaked path. MANRS counted 212 leaked Google prefixes (ThousandEyes saw 180+, other reports about 150). Traffic to Google Search, G Suite and Google Cloud from many networks was pulled toward China Telecom and dropped, causing a roughly 74-minute disruption.
fail × 12018-10-11Fits no familyGoogle Cloud us-central1-c: BGP sessions deleted during router replacement rollback
While replacing the routers that link us-central1-c to Google's backbone, a rollback workflow pushed a configuration meant for the new routers onto the older routers. That deleted their BGP sessions, and the zone lost connectivity to the backbone for 41 minutes. 61% of VMs in the zone lost external connectivity.
fail × 12018-07-30Mis-originationIran TCI originates Telegram prefixes
For about two hours and fifteen minutes AS58224 announced more than 100 prefixes, ten of them Telegram's, and some propagated internationally.
fail × 2pass × 1audit finding × 12018-07-09 (culmination of years of activity)Accepting networkBitcanal 'hijack factory' disconnected
A Portuguese hosting firm announced unrouted or abandoned address space, at least 130 documented cases, and leased it to spammers. After a public NANOG thread and Oracle Dyn evidence, transit providers and exchanges disconnected it one by one; the last transit dropped on 2018-07-10.
audit finding × 12018-07-06 (also 07-10, 07-11 and later in July)Accepting networkBGP/DNS hijacks against US payment processors
On several dates the prefixes holding payment processors' authoritative DNS servers were hijacked, and forged answers with very long TTLs were served so resolvers kept the bad records after the routes vanished. The last Vantiv hijack lasted nearly three hours.
audit finding × 32018-05-31Required flowCloudflare 1.1.1.1 mitigated by its own DDoS system (Gatebot)
A deployment replaced Gatebot's hard-coded list of Cloudflare address ranges with a Provision API lookup but did not carry over the manual exception for the resolver ranges. Gatebot classified 1.1.1.1 traffic as an attack and applied DNS mitigations to it, making the resolver globally unreachable for 17 minutes.
fail × 1pass × 12018-04-24Accepting networkAWS Route 53 prefixes hijacked via eNet (MyEtherWallet)
For about 2 hours, more-specific routes covering Amazon Route 53 DNS address space were announced by AS10297 and accepted by some peers. DNS queries reaching the bogus servers were answered only for myetherwallet.com and directed users to a phishing site.
audit finding × 12018-01-24Required flowNHS Wales national data-centre outage (routine firewall change)
A routine firewall change cut access to both NWIS data centres on the morning of 24 Jan 2018. Twenty-six nationally hosted clinical systems (appointments, referrals, test results, GP test requesting) and NHS Wales email were unavailable; at one point at least half of GP practices were affected. Systems were restored by 16:15 and normal operations by 20:00.
fail × 1pass × 12017-12-12Mis-originationDV-LINK (AS39523) originates 80 high-profile prefixes
Starting 04:43 UTC AS39523 announced one prefix of its own and 80 belonging to large content networks. BGPmon saw two windows of about three minutes; Qrator reports activity from 04:44 to 07:19 UTC. Level3, Cogent, Hurricane Electric, Zayo and others accepted the routes.
fail × 2audit finding × 12017-11-06Fits no familyLevel 3 BGP route leak (Comcast and other US ISPs disrupted)
At 17:47 UTC Level 3 began globally announcing thousands of BGP routes learned from customers and peers that were meant to stay internal to its network, per Oracle/Dyn's analysis. Traffic for large eyeball networks such as Comcast and Bell Canada and for content providers such as Netflix was drawn through Level 3's misconfigured routers, causing a nationwide slowdown and outage for Comcast users and others. Many leaked routes stayed in circulation until about 19:24 UTC, roughly 90 minutes.
fail × 12017-10-21Route leakInfovale / Algar Telecom leak of CDN routes (Brazil)
From 11:09 to 11:27 UTC (18 minutes), Infovale announced routes learned from its peers (large content networks) to Algar Telecom, which passed them on to its own peers including Hurricane Electric. Networks as far away as the US briefly routed Google and Netflix traffic through Brazil, and BGPmon reported disruptions to Twitter, Google and DNS resolution (for example 8.8.8.0/24).
fail × 2audit finding × 12017-08-25Route leakGoogle leaks more-specifics to Verizon (Japan outage)
At 03:22 UTC Google leaked more than 160,000 prefixes to Verizon in Chicago, many of them more-specifics of other networks' space, including over 25,000 of NTT OCN's. KDDI and IIJ, as Verizon customers, accepted over 95,000 each, so Japanese domestic traffic headed for Chicago. Corrected within minutes, but services in Japan were disrupted for hours.
fail × 2audit finding × 12017-05-12Required flowWannaCry in the NHS (Internet-facing firewall exposure)
At least 81 of 236 English trusts and 603 primary-care and other organisations (595 GP practices) were infected or disrupted; NHS England identified 6,912 cancelled appointments and estimated more than 19,000 in total. Five A&E departments diverted patients.
fail × 12017-04-26Mis-originationRostelecom originates financial-services prefixes
From 22:36 to about 22:43 UTC AS12389 originated 50 prefixes belonging to other networks, a selection heavy with payment and security companies. BGPmon called it 'curious' because of the victim list.
fail × 2audit finding × 12017-03-08Required flowAT&T Mobility VoLTE 911 outage (trusted-IP whitelist push)
For about five hours AT&T VoLTE customers nationwide could not reach 911; roughly 12,600 unique callers failed. AT&T pushed an updated list of trusted IP addresses that omitted Comtech's addresses, cutting AT&T off from the systems that return 911 call-routing information. Rolling back took three minutes once the IP team was involved, but it took almost five hours to involve them.
fail × 1pass × 12017-01-05Mis-originationIran TIC censorship route for an adult-site /24 leaks via Omantel
Around 12:00 UTC on 5 January 2017, AS12880 began announcing 99.192.226.0/24, a more-specific of MojoHost's 99.192.128.0/17 that hosts adult sites censored in Iran. The route reached Omantel, which re-advertised it to its peers, so users outside Iran were also blackholed. It persisted about 28 hours, until roughly 16:00 UTC on 6 January, when the legitimate origin announced the same /24 and Omantel stopped propagating the bogus route.
fail × 2pass × 1audit finding × 12016-09-07Accepting networkBackConnect hijacks a Verdina /24
A DDoS-mitigation firm announced a Bulgarian host's /24 for about 50 minutes; more than half of monitored peers accepted it. Dyn's follow-up found a history of other suspicious announcements by the same AS.
audit finding × 12016-08-09RPKI and driftAustralian Census 2016 (#CensusFail): router lost its configuration on reboot; geoblock mis-scoped
On census night a series of small DDoS attacks led IBM to invoke its 'Island Australia' geoblocking plan, which one upstream had not fully implemented. One of two border routers was then rebooted and came back without its configuration, and the site was taken offline; the online census was unavailable for about 40 hours. The same geoblock also cut Australians off from the password-reset service, which was hosted offshore.
exit-code check × 22016-08-05Fits no familyGoogle Cloud: GCP prefixes announced from a single PoP during router drain
While removing a faulty router from service, engineers used a new traffic-diversion procedure. The configuration it applied caused some GCP IP ranges to be announced from a single point of presence in the southwestern US, pulling traffic into infrastructure with overly restrictive filters. Non-TCP/UDP traffic (ICMP, ESP, AH, SCTP) was dropped, Cloud VPN tunnels lost all data traffic and Cloud Router BGP sessions dropped for 99 minutes.
fail × 1pass × 12016-07-20RedundancySouthwest Airlines router partial failure (2,300 flights cancelled)
At 13:09 on 20 July a router in Southwest's Dallas Love Field data centre failed in a way its backup did not detect, choking hundreds of applications. About 2,300 flights were cancelled over four days and the airline put the cost in the tens of millions of dollars.
pass × 1audit finding × 12016-06-20Fits no familyTelia Carrier router misconfiguration (Europe traffic misrouted)
From about 12:10 UTC Cloudflare detected massive packet loss on Telia Carrier's backbone and shut its Telia ports at 12:30 UTC to route around it. Many European users lost access to services such as Slack and WhatsApp. The Register reported that a Telia engineer misconfigured a key router so that European traffic was sent toward Hong Kong, and that Telia sent an apology note to other network operators; Telia's own detailed report was not published.
fail × 12016-04-22Route leakinnofield (AS200759) leak at SwissIX after maintenance
From 17:09 to about 17:30 UTC, innofield announced 3,431 prefixes belonging to 576 autonomous systems, many as more-specifics, to the SwissIX exchange. Hurricane Electric and others accepted them, making destinations at Google, Amazon, Twitter, Apple and Akamai unreachable for affected networks for about 21 minutes.
fail × 2audit finding × 12016-04-11Hand-built replaysGoogle Compute Engine: every IP block withdrawn
Engineers retired one unused IP block; the management software pushed a configuration with every block removed, and every site stopped announcing them.
passfail2015-12-01Route leakChina Telecom carries Verizon APAC routes for about 2.5 years
According to Oracle's Doug Madory, routes from Verizon's Asia-Pacific network were partially routed through China Telecom from December 2015 until April 2018. The likely chain was that Verizon APAC peered with SK Broadband, which passed those routes only to China Telecom, which then announced them to the global internet. One observed effect was traffic from London to Australia's Department of Defence transiting China Telecom.
fail × 2audit finding × 12015-11-06Mis-originationBharti Airtel mis-originates about 16,000 prefixes
Starting 05:52 UTC AS9498 claimed origin for about 16,123 prefixes. Because two large networks propagated the routes, the event spread widely and, by one academic account, lasted until about 14:40 UTC.
fail × 2audit finding × 12015-06-30Route leakAxcelX leaks Amazon routes via Hibernia
A Boston data-centre provider re-advertised Amazon routes to its transit provider Hibernia, inserting both into paths to AS14618. Traffic terminated in Hibernia's network with heavy loss until the leak was withdrawn.
fail × 2audit finding × 12015-06-12Route leakTelekom Malaysia leaks 179,000 prefixes to Level 3
From 08:43 UTC TM announced about 179,000 prefixes learned from its providers and peers to Level 3, which accepted them and preferred them as customer routes. Traffic from much of the world squeezed through TM's links, causing loss and slowdowns for about two hours.
fail × 2audit finding × 12015-05-13Layer 2AMS-IX peering LAN loop during 100GE module testing
At 12:20 CEST, while engineers tested newly installed 100GE modules ahead of planned maintenance, four 100GE ports were looped while still members of the production peering VPLS. Broadcast traffic over the loop made member routers' MAC addresses appear behind the looped ports, line-card CPUs spiked, and about 500 of 600 route-server BGP sessions (74.5% of sessions platform-wide) dropped. The NOC shut the ports at 12:29 and most sessions were back by 12:40.
audit finding × 12015-04-10Fits no familyGoogle Compute Engine us-central1: traffic sent to a router running a test configuration
During routine planned maintenance, a miscommunication caused production traffic to be sent to a datacenter router that was running a test configuration. A link saturated and us-central1 saw 18 to 47% packet loss for 14 minutes until the router was disabled.
fail × 2pass × 22015-03-26Mis-originationEnzu BGP optimizer leaks 7,000 more-specifics at Any2 Los Angeles
From 10:38 to 14:26 UTC more than 7,000 synthetic more-specific prefixes, with fabricated origin ASes matching the real owners, escaped Enzu's network. Carriers including AT&T, Comcast, Telstra, Sprint and Deutsche Telekom had prefixes affected.
fail × 2pass × 1audit finding × 12015-03-12Route leakHathway leaks Google prefixes to Bharti Airtel
Hathway, which peers with Google, leaked 336 Google IPv4 prefixes to its transit provider Airtel, which announced them onward at European exchanges and to large carriers. BGPmon dates the Google service interruption to 08:58 to 09:14 UTC on 12 March, affecting users in Europe and India; ThousandEyes (per search-result summary) describes the leak as persisting from about 10:30 UTC on 11 March to 09:15 UTC on 12 March. Networks preferred the leaked path (9498 17488 15169) because Airtel was their customer. Note: this is separate from Airtel's own large origination event of 6 November 2015.
fail × 2audit finding × 12015-03-12 (some prefixes from 2015-03-07)Accepting networkVega (Ukraine) attracts UK traffic including the Atomic Weapons Establishment
A bad route from Vega, BT's reseller in Ukraine, drew traffic for a set of BT-hosted customers through Ukraine for about 90 minutes on 12 March, with a handful diverted for five days.
audit finding × 32014-12-09Mis-originationSyrian Telecom mis-originates about 1,480 prefixes
From 08:33 to about 08:37 UTC AS29386 originated 1,481 prefixes belonging to 306 other ASes, including YouTube, Akamai and Incapsula space. Traffic from networks that selected the routes headed toward Syria for a few minutes.
fail × 2audit finding × 12014-10-02Route leakCloudflare route leak by Internexa (Medellin)
Internexa leaked routes so that large amounts of traffic bound for Cloudflare data centers worldwide were sent to Cloudflare's single Medellin site. North American traffic to Cloudflare dropped about 50% for 49 minutes until Cloudflare cut the Internexa sessions.
fail × 2audit finding × 12014-09 (exact day not confirmed)Route leakVolumeDrive leaks routes to Atrato
RFC 7908 cites this as a Type 1 'hairpin' leak: a multi-homed customer re-advertised routes learned from one provider to another provider, which accepted them, disrupting traffic to distant networks.
fail × 2audit finding × 12014-08-27Fits no familyTime Warner Cable nationwide Internet outage
At about 4:30 AM ET, during overnight network maintenance, an erroneous configuration was propagated across TWC's national backbone. RIPE Labs observed severe BGP prefix withdrawals across most of TWC's seven ASNs from roughly 09:30 to 11:00 UTC, with 0-2% reachability for several of them and at least 64 single-homed downstream networks taken offline. Internet and on-demand video were knocked out in all TWC markets, affecting on the order of 11 million residential broadband customers, with service largely restored by about 6:00 AM ET (TWC later said 7:30 AM).
fail × 22014-08-25Fits no familyStack Exchange iptables change blocks load-balancer return traffic
All Stack Exchange sites were down for about six minutes (19:26-19:32 UTC). A change to the iptables rules on the HAProxy load balancers, pushed from Git by Puppet, caused SYN/ACK replies from the IIS web tier to be dropped, so the load balancers could no longer complete connections to any backend.
fail × 1unknown × 12014-08-12Fits no family512k day: Verizon de-aggregation overflows router TCAM
Around 07:48 UTC Verizon briefly announced about 15,000 new more-specific /24s (RFC 7908 says about 22,000), pushing the global table past 512,000 routes. Routers whose hardware route memory defaulted to 512K entries fell back to software forwarding or dropped routes, causing scattered slowdowns and outages.
fail × 22014-04-02Mis-originationIndosat mass mis-origination of ~417,000 prefixes
From 18:26 UTC to about 21:15 UTC Indosat, which normally originates about 300 prefixes, originated 417,038 prefixes belonging to other networks, close to the full global table. Most of the bogus routes did not spread far, but 8,182 prefixes were seen by more than ten BGPmon probes, and some networks lost reachability. BGPmon notes Indosat had a similar incident in 2011.
fail × 2audit finding × 12014-03-29Accepting networkTurk Telekom hijack of public DNS resolver addresses
After Turkish users switched to foreign DNS resolvers to evade a Twitter block, Turk Telekom installed /32 routes for the resolver addresses and answered the queries from its own servers, returning falsified results. The interception ran from 29 March until about 7 April 2014 and affected users inside Turk Telekom's network.
audit finding × 12014-02 (through 2014-05)Accepting networkCanadian ISP Bitcoin mining-pool hijacks
Short, repeated more-specific hijacks redirected miners' connections to an attacker-run pool that kept the rewards, an estimated $83,000 over about four months. All announcements traced to a single router at one Canadian ISP and stopped three days after Secureworks notified the upstream.
audit finding × 12013-12-17Fits no familyAWS sa-east-1: bad config on restarted network device advertised invalid routes
Utility power loss and generator failures took down part of one AZ in Sao Paulo. During recovery a technician manually restarted a network device and introduced a bad configuration, so it advertised invalid routes and degraded internet connectivity for both AZs in the region for about 20 minutes.
fail × 12013-08-16 (to 2013-08-22)Accepting networkHacking Team / Aruba announce a Santrex /24 for the Italian ROS
After Santrex's hosting failed, the operators of a surveillance tool lost contact with their implants. Aruba announced Santrex's /24 for six days so a replacement server at 46.166.163.175 could regain control. Revealed by the 2015 Hacking Team leak and confirmed in historical BGP data.
audit finding × 12013-07-31 (through August 2013)Mis-originationIceland Opin Kerfi / Siminn traffic redirection
Opin Kerfi began originating routes for 597 networks of a US provider, and Renesys saw redirections from nine Icelandic ASes behind Siminn. Traffic was pulled to Iceland and then delivered onward to its destination.
fail × 2pass × 1audit finding × 12013-03-21Accepting networkCB3ROB hijacks a Spamhaus name-server /32
During the large DDoS against Spamhaus, AS34109 announced a /32 for one Spamhaus DNS server and answered every blocklist query with a positive result (127.0.0.2). Networks that accepted the /32 over peering got wrong spam verdicts.
audit finding × 12013-03-03FlowspecCloudflare 2013: Flowspec rule crashes every edge router
While mitigating a DDoS attack, Cloudflare pushed a Flowspec rule matching an impossible packet length range; Juniper edge routers consumed all memory and crashed network-wide, taking Cloudflare offline for about an hour.
fail × 1unknown × 22013-02 (almost daily through February; again briefly in May 2013)Mis-originationBelarus GlobalOneBel traffic redirection
Renesys documented repeated events, minutes to hours long, in which victim prefixes were re-originated so traffic flowed through Belarus and was then forwarded on to the real destination, a working man-in-the-middle. The victim list changed daily.
fail × 2pass × 1audit finding × 12012-12 (late December; BGPmon write-up dated 2013-01-08)Mis-originationAvenir Telematique route optimizer leak
For about 30 minutes a route-optimizer appliance's synthetic more-specific routes escaped AS35625 to its peers, redirecting traffic for Microsoft, Facebook, Cogent, Bell Canada, Verizon, Level3 and others toward France.
fail × 2pass × 1audit finding × 12012-11-06Route leakMoratel leak of Google routes via PCCW
At about 02:24 UTC on 6 November 2012 (evening of 5 November US Pacific time), Moratel announced paths to Google prefixes to its upstream PCCW, which accepted and propagated them. Google services were unreachable for about 27 minutes for an estimated 3 to 5 percent of internet users, most heavily in Hong Kong. A Cloudflare engineer contacted Moratel and the routes were withdrawn minutes later.
fail × 2audit finding × 12012-08-08Route leakDery Telecom leaks a full table to Bell Canada
Starting 17:27 UTC a small Quebec provider re-advertised a full or near-full table learned from one upstream to its other upstream, Bell. Bell selected 107,409 of those routes as best. Operators reported session flaps, pegged router CPUs and reachability problems for Bell and Tata customers.
fail × 2audit finding × 12012-05-05Fits no familyFog Creek network loop during planned switch reconfiguration
During a planned weekend network maintenance (5-6 May 2012) a loop formed between several switches. The resulting BPDU flood caused spanning tree to flap and hosted services were unavailable for about two hours in total.
fail × 1pass × 12012-02-23Hand-built replaysDodo / Telstra: a customer re-announces the full table
A customer's export filter stopped filtering; it re-announced the full table to Telstra, which prefers customer routes and passed it on until its upstreams' prefix limits tore the sessions down.
fail2012-01 (blog post date; incident date not confirmed)Layer 2Etsy global outage from multicast on unprepared switches
Etsy tried multicast-based replication of Solr search indexes; the multicast traffic flooded the production network and caused a site-wide outage, after which Etsy moved to BitTorrent for index distribution.
audit finding × 12011-10-01IPv6F-root IPv6 route leaks out of Beijing
For about 25 hours the local-scope anycast announcement of F-root's IPv6 prefix from the Beijing node was visible globally, so IPv6 queries to F-root from around the world could be answered from inside China.
fail × 2pass × 12011-04-21Fits no familyAWS EC2/EBS us-east-1: traffic shifted onto low-capacity redundant network
During a change to upgrade capacity on the primary EBS network in one Availability Zone, traffic was shifted off a primary router incorrectly. Instead of going to the other primary router it went onto the lower-capacity redundant EBS network, isolating many EBS nodes; when connectivity returned a re-mirroring storm left 13% of volumes in the AZ stuck. Full recovery took about 3 days.
fail × 1pass × 12011-03-22Route leakFacebook traffic detours through China Telecom and SK
For about nine hours, AT&T and a list of other carriers reached several Facebook prefixes through China and Korea. Facebook's announcements via Level3 and Global Crossing carried five prepends, so the unfiltered path re-announced by China Telecom looked shorter and won.
fail × 2audit finding × 12011-01-14Mis-originationIndosat mis-originates about 2,800 prefixes
From 12:19 to 12:57 UTC AS4761, which normally originated about 100 prefixes, originated about 2,800 belonging to 824 other ASes, including 8.8.8.0/24, Akamai, Amazon and US government space. Some routes reached up to 50 dispersed vantage points.
fail × 2audit finding × 12010-08-22 (BGPmon post dated 2010-08-23 says 'this Sunday')Accepting networkEvolva Telecom originates Google Public DNS 8.8.8.0/24
AS30890 originated 8.8.8.0/24 for about seven minutes. BGPmon saw it from 14 RIS peers in four countries.
audit finding × 12010-07-09Mis-originationAnexia originates a Google /24 (copy-paste error)
AS42473 originated Google's 74.125.127.0/24. It was noticed and withdrawn quickly by Anexia's engineers.
fail × 2pass × 1audit finding × 12010-04-08Mis-originationChina Telecom / CHINANET-IDC-BJ mis-originates about 37,000 prefixes
From 17:54 to 18:10 UTC AS23724, which normally originates about 40 prefixes, originated about 37,000. Roughly 10% propagated outside China through AS4134, touching prefixes of Dell, CNN, Amazon.de and US government and military networks. It later became the basis of a US-China commission report.
fail × 2audit finding × 12009-11-19Fits no familyFAA FTI flight-plan network outage (Salt Lake City router)
A router replaced during maintenance at the FTI hub in Salt Lake City carried a routing error in its configuration. From about 05:00 EST flight-plan (NADIN) and traffic-flow data stopped moving nationally for roughly four hours; controllers entered flight plans by hand and hundreds of flights were delayed. The fault also took down a second node in Hampton, Georgia, and did not fail over.
fail × 12009-10-10Accepting networkWind Telecomunicazioni 85,000-prefix mis-origination (Italy)
At about 07:22 UTC AS9035 originated roughly 85,000 prefixes that were not its own. Only two BGPmon peers, both in Italy, saw the routes, and the event cleared in about two minutes.
audit finding × 12009-02-16Hand-built replaysSuproNet 252-times AS-path prepend
SuproNet announced a single prefix with its AS number prepended 252 times, producing an AS path of about 256 entries. Cisco IOS routers that received the path and applied their own prepending mishandled the over-long attribute, sent malformed updates, and had BGP sessions reset repeatedly. Global update rates spiked (Renesys measured a peak above 100,000 updates per second, per search-result summaries) and instability lasted until the route was withdrawn.
fail × 22008-11-11Mis-originationCTBC (Brazil) full-table origination
Between 01:55 and 02:15 UTC, AS16735 originated 267,947 distinct prefixes, effectively the whole internet table, as if they were its own; related announcements continued until about 03:07 UTC. BGPmon says propagation was limited: only the RIPE RIS collector at the Sao Paulo exchange saw it, so global impact was small.
fail × 2audit finding × 12008-08-10Accepting networkPilosov-Kapela DEFCON man-in-the-middle demonstration
A live, authorised demonstration that an attacker can announce a more-specific of a victim prefix, use AS-path poisoning so one upstream keeps a clean return path, and forward intercepted traffic on to the victim, hiding the detour by adjusting TTL. The slides show the ALTDB route object being added with no verification.
audit finding × 22008-02-24Hand-built replaysPakistan Telecom YouTube hijack via PCCW
Ordered by the Pakistani government to block YouTube, Pakistan Telecom announced 208.65.153.0/24, a more-specific of YouTube's 208.65.152.0/22, at 18:47 UTC. Its upstream PCCW propagated the route globally and YouTube was unreachable for much of the internet. YouTube countered with its own /24 at 20:07 and /25s at 20:18, and PCCW withdrew AS17557's routes at 21:01 UTC, about 2 hours 14 minutes after the start.
fail2007-08-11Layer 2LAX Customs and Border Protection LAN outage
CBP's local network at LAX slowed from ~12:50 PDT and stopped passing database queries at 16:16; full processing resumed at 23:40. More than 17,000 arriving international passengers were held, about 60 aircraft sat on the tarmac for hours, and some flights diverted to Ontario, CA. Technicians eventually restored service by disconnecting a circuit carrying 12 devices in the Bradley terminal.
audit finding × 12006-01-22Mis-originationCon Edison hijack of Panix and other prefixes
Con Edison's network began originating prefixes it did not own, including Panix's 166.84.0.0/16, and its upstreams accepted and propagated them. New York ISP Panix lost connectivity for several hours until it announced two more-specific /17s and the upstream was asked to reject the bogus route. Renesys's same-day analysis on NANOG (as quoted in search results) counted about 70 newly announced prefixes, around 25 of them with no previous relationship to AS27506.
fail × 2pass × 1audit finding × 12005-05-07Accepting networkCogent originates a Google /24 (Google May 2005 outage)
At 14:37 UTC AS174 began originating Google's 64.233.161.0/24. A Carleton University study found 49.1% of the ASes re-advertising that prefix switched to the bogus path. Google's own outage later that day (about 22:10 UTC, 15 to 60 minutes) was publicly attributed by Google to internal DNS misconfiguration, so the link between the two is unproven.
audit finding × 12004-12-24Mis-originationTTNet (AS9121) full-table mis-origination
On Christmas Eve 2004 AS9121 announced over 100,000 routes to its peers as if it were the path to most of the Internet. Many large networks carried more than 25,000 bad paths, some as many as 100,000, and large parts of the Internet were unreachable or misdirected toward Turkey for hours.
fail × 2audit finding × 12003-01-25Required flowSQL Slammer enterprise disruption (Bank of America ATMs, Continental, Seattle 911)
A 376-byte UDP worm against SQL Server saturated networks worldwide within minutes; reported effects included most of Bank of America's ATMs going offline, airline check-in delays and a 911 dispatch centre reverting to paper.
fail × 12002-11-13Layer 2Beth Israel Deaconess Medical Center spanning-tree collapse
The hospital's flat, all-Layer-2 campus network collapsed repeatedly over roughly four days (13-18 Nov 2002). Clinicians reverted to paper, lab turnaround went from ~45 minutes to as long as five hours, and the emergency department went on diversion for hours. Cisco flew in Catalyst 6509s and rebuilt the core with Layer-3 routing.
audit finding × 12001-01-23Fits no familyMicrosoft.com / MSN / Hotmail DNS unreachable after edge-router change
At 18:30 PST on 23 Jan 2001 a technician changed the configuration of the routers in front of Microsoft's DNS servers. Microsoft.com, MSN, Hotmail, Expedia and other properties became unresolvable for a large share of the Internet for about 22.5 hours, although the web servers themselves were up. All four authoritative name servers sat behind the same router pair.
fail × 11997-10 (exact day not confirmed)Accepting networkUUNet AS701 de-aggregation of 128.0.0.0/9
Months after AS 7007, AS701 originated the IPv4 space inside 128.0.0.0/9 as /24s, disrupting traffic to a large share of the routing table of the day. Known mainly through a NANOG thread and later retrospectives.
audit finding × 21997-04-25Hand-built replaysAS7007 incident (MAI Network Services route leak)
At about 11:30 AM Eastern, AS7007's border router received a full view of roughly 23,000 routes from a downstream ISP, de-aggregated them into /24s, stripped the AS path and re-announced them as originated by AS7007. Because the /24s were more specific than the legitimate routes, a large part of Internet traffic was drawn to AS7007 and black-holed. The bad routes lingered in other networks for hours even after the router was disconnected.
failunknownFits no familyValve / Steam disconnected from upstreams by a BGP configuration error
Steam became unreachable when Valve's network lost its connections to three transit providers at once.
fail × 1Not replayed, and why
27 further outages in the catalog were in scope but not rebuilt. Each carries the reason Hammerhead's catalog gives. The catalog lists 314 incidents in all; the other 153 are outside what a configuration model can speak to (software, hardware, power, fibre, DNS content) and are not shown here.
Flow Grenada island-wide outage with total BGP withdrawal
Traffic began dropping around 11:30 local time on 9 February, disappeared entirely around 20:00, and recovered by about 23:30 (03:30 UTC on 10 February), roughly 12 hours in total. Cloudflare observed complete loss of the operator's announced IPv4 space coinciding with zero traffic, bracketed by large spikes in BGP announcements.
Why not replayed: Total withdrawal is what the advertisement gate fails on; nothing published shows a change rather than a failure caused it.
Other · catalog confidence low · blog.cloudflare.com
Azure Front Door: control-plane metadata defect after protection bypass
A defect in a recently rolled-out AFD control plane produced erroneous tenant metadata. When protection systems were bypassed for a configuration clean-up, the metadata reached the data plane and triggered a bug; AFD and CDN had latency and timeouts for about 8 hours, worst in Africa and Europe.
Why not replayed: A cloud-service (Front Door) resource, not a device config.
Software bug · catalog confidence high · azure.status.microsoft, azure.status.microsoft
Cloudflare / AWS us-east-1 peering congestion
One customer's traffic surge from AWS us-east-1 saturated Cloudflare's peering links with AWS. AWS withdrew some BGP prefixes to shift load, which pushed traffic onto other already strained paths; a peering link running at half capacity and an overdue interconnect upgrade made it worse. Traffic between Cloudflare and us-east-1 was degraded for nearly 4 hours.
Why not replayed: No configuration change started it, and the aggravating step is load moving onto a link already at half capacity: load is in no config.
Capacity / traffic shift after change · catalog confidence high · blog.cloudflare.com
Telefonica Spain network-upgrade outage hitting 112 lines
Overnight network upgrade work by Telefonica disrupted fixed voice and internet services for businesses and public bodies from the early hours, and 112 emergency call centres in several regions (reports variously name Aragon, Galicia, Extremadura, Andalusia, Valencia, the Basque Country, Navarra and La Rioja) had to publish alternative numbers. Services were restored by around midday. One secondary write-up attributes the fault to maintenance on Telefonica's MPLS network, but I found no Telefonica statement saying so.
Why not replayed: A planned upgrade severed emergency-service circuits, the required-flow family's shape, but nothing published says what the upgrade changed.
Other · catalog confidence medium · spainenglish.com, techblog.comsoc.org, beyondmachines.net, datacenterdynamics.com, fierce-network.com
BART systemwide shutdown: redundant network devices 'not having proper connectivity'
All train service was suspended from the start of service until about 09:30 on a Friday because the control centre could not see the system. Crews restored visibility by isolating and disconnecting the devices involved. BART said the legacy train-control system was not a factor.
Why not replayed: BART has not said what 'not having proper communication' between the redundant devices meant in configuration terms.
Redundancy lost before failure · catalog confidence medium · kqed.org, abc7news.com, richmondside.org
Innove Communications sub-prefix of SingNet
An academic study (NDSS) reported a long-lived /24 more-specific of SingNet space originated by Innove, diverting traffic without triggering alarms, as an example of 'stealthy' hijacks that only some vantage points see.
Why not replayed: Nothing public confirms the sub-prefix came from a configuration line; the mis-origination family covers the shape from every seat.
Hijack origination · catalog confidence low · anapaya.net, en.wikipedia.org
GitHub: internal load balancer config change dropped Git traffic
A configuration change related to traffic routing and testing made GitHub's internal load balancer drop requests between services Git depends on. All Git operations were unavailable for 49 minutes.
Why not replayed: GitHub did not say which platform the load balancer is; a flow rule with its port (the required-flow family) is the check if it were F5 or A10, which are parsed.
Load balancer / vip config · catalog confidence medium · github.blog
UK Border Force e-gates nationwide outage
Passport e-gates failed at Heathrow, Gatwick, Manchester, Edinburgh, Birmingham, Bristol, Newcastle and other airports from about 19:44 until shortly after midnight; officers processed arrivals manually and long queues formed.
Why not replayed: The operator names its own network and nothing public says what failed.
Other · catalog confidence low · hansard.parliament.uk, theyworkforyou.com, computerweekly.com, itv.com
Vodafone Egypt 4G outage after morning updates
Between 05:15 and 11:30 local time mobile internet on Vodafone Egypt was disrupted, with Cloudflare measuring traffic as much as 70 percent below normal. Cloudflare reports that Vodafone was afterwards required to compensate customers and was fined by Egypt's National Telecommunications Regulatory Authority.
Why not replayed: The update appears to have been in the mobile core or radio layer; nothing public says it touched IP routing or firewalls.
Software bug / upgrade · catalog confidence low · blog.cloudflare.com
Virgin Media UK nationwide outage with BGP withdrawals
From about 00:30 UTC Virgin Media's prefixes were withdrawn from the global routing table and traffic fell to near zero, with brief restorations, until about 07:00-07:30 UTC; a second shorter disruption followed at about 15:00-16:30 UTC. ThousandEyes observed that where routes remained, traffic was dropped at Virgin's edge, suggesting control-plane distress, and that AS6830 was removed from paths during mitigation. More than 27,000 Downdetector reports were logged; total customers affected were not published.
Why not replayed: Every announcement withdrawn in a maintenance window is the shape the advertisement gate fails on (Cloudflare BYOIP 2026, Google Cloud 2016); Virgin Media never said a change caused it.
Other · catalog confidence medium · blog.cloudflare.com, kentik.com, thousandeyes.com, theregister.com, ispreview.co.uk
CircleCI: Kubernetes upgrade corrupted iptables rules
A staged Kubernetes upgrade left kubelet and kube-proxy on incompatible versions, iptables-restore failed and service routing inside the main cluster broke. Jobs could not start for about 7.5 hours.
Why not replayed: Kubernetes CNI resources, not a device config; the corrupted iptables rules were a side effect of the upgrade, not a pushed candidate.
Software bug · catalog confidence high · github.com, k8s.af, status.circleci.com, reddit.com
Lufthansa global IT outage (construction severs fibre in Frankfurt)
Check-in and boarding systems failed group-wide; flights into Frankfurt were halted or diverted and thousands of passengers were affected.
Why not replayed: A fibre cut, and no source says the backup shared it. Shared-risk groups can be declared now (preflight --srlg), but the only account read (The Register, 15 February 2023) has "no insight into whether Lufthansa's presumed backup for that connection also failed", and the catalog row has no source for one duct carrying every circuit, so the group would be invented.
Hardware / power / fiber · catalog confidence low
Swisscom planned-maintenance outage incl. emergency numbers
From about 22:30 on 11 February 2020 until shortly after midnight, internet-based services, fixed telephony, mobile and TV failed across large parts of Switzerland, and emergency numbers were unreachable by landline in several cantons (English-language reports describe about 1.5 hours across 11 cantons on the night of 11-12 February). It was one of six nationwide Swisscom incidents between 17 January and 19 February 2020 that prompted a report by the regulator BAKOM.
Why not replayed: A planned change cut emergency-call reachability, which the required-flow family covers in shape, but the published cause is too vague to say what was changed.
Other · catalog confidence medium · srf.ch, parlament.ch, blick.ch, swissinfo.ch
Google Cloud June 2019 network congestion (us-east)
A maintenance event intended for a few servers in one region was, through two misconfigurations and an automation bug, applied to network control plane jobs in several physical locations at once. With the control plane descheduled, BGP routing between sites was withdrawn after the fail-static window and available capacity collapsed, causing heavy congestion and packet loss in eastern US regions for roughly 3.5 to 4.5 hours.
Why not replayed: The misconfiguration lived in the job scheduler, not in router configs.
Network automation pushed bad config · catalog confidence high · status.cloud.google.com
Google Cloud us-central1: configuration change during network expansion
During a network expansion in us-central1, a configuration change caused packet loss and reduced bandwidth for inter-region transfer and replication traffic in two windows totalling about 3.5 hours. Internet and VM-to-VM traffic were not affected.
Why not replayed: The public report does not say what the backbone change was; a capacity reduction is load, which is in no config.
IGP / backbone config · catalog confidence medium · status.cloud.google.com
Manchester University NHS Foundation Trust network outage
A roughly four-hour network outage across several hospital sites forced staff onto pen and paper; some operations were postponed and outpatient appointments rearranged. It happened the same day as the unrelated NHS Wales outage.
Why not replayed: Described as a network outage with no published cause.
Other · catalog confidence low · digitalhealth.net, ibtimes.co.uk
Amadeus Altea global check-in outage ('network issue')
Altea, used by about 125 airlines, was disrupted from about 10:30 GMT. The core interruption was around 15 minutes but services were not fully restored until about 14:30 GMT; check-in delays were reported at Heathrow, Gatwick, Paris CDG, Zurich, Singapore, Melbourne, Johannesburg and Washington Reagan.
Why not replayed: Attributed to the network without saying whether it was a change, hardware or software.
Other · catalog confidence low · datacenterdynamics.com, money.cnn.com, npr.org, ciodive.com
Safaricom nationwide outage (voice, data, M-Pesa)
Voice, SMS, data, M-Pesa and enterprise services failed nationwide starting between about 09:30 and 09:40 local time. Early reports put the outage at around two hours; the Daily Nation reported disruption from 09:40 until 16:30. Banks integrated with M-Pesa were affected, and Business Daily estimated up to KSh 2.6 billion in M-Pesa transactions were lost. The Communications Authority said it does not tolerate downtime beyond one hour and indicated penalties unless the cause was an act of God.
Why not replayed: The root cause was never published.
Other · catalog confidence low · nation.africa, techweez.com, standardmedia.co.ke
Google Compute Engine europe-west1-c: invalid firewall rules after control-plane migration
Hours after engineers migrated firewall-rule management in europe-west1-c to a new integration layer, a routine remastering event loaded incorrect network configuration. Up to 17% of instances in the zone lost all network connectivity for 62 minutes.
Why not replayed: SDN virtual-firewall rules, not a device config any parser reads; the outage also needed a software race.
ACL / firewall rule · catalog confidence high · status.cloud.google.com
Wikimedia partial network outage: IXP flood plus latent BGP confederation misconfiguration
Another Equinix Ashburn customer flooded the exchange fabric with IPv6 neighbour-discovery multicast while a GTT MPLS link between ulsfo and eqiad amplified packets, overloading a core router's control plane. OSPF became unstable, and an inconsistent BGP confederation configuration in ulsfo produced a routing loop between ulsfo, eqord and eqiad. US West Coast users were affected for about 3.5 hours.
Why not replayed: A latent BGP confederation inconsistency that looped only under an IXP flood; confederations are modelled, the flood is load. Queued.
Routing policy error · catalog confidence high · wikitech.wikimedia.org
American Airlines hub ground stop ('connectivity issues')
American requested ground stops at Dallas/Fort Worth, Chicago O'Hare and Miami from about 12:41 ET; systems were restored roughly two hours later.
Why not replayed: Network-layer wording with no published cause.
Other · catalog confidence low · money.cnn.com, time.com, npr.org
United Airlines nationwide ground stop ('router issue')
The FAA issued a ground stop for all United flights at 08:26 ET at the airline's request; it lasted well over an hour and delayed or cancelled several hundred flights. It fell on the same day as the NYSE halt, and officials ruled out an attack.
Why not replayed: United never said whether the router fault was hardware, software or configuration.
Other · catalog confidence medium · npr.org, aviationtoday.com, abcnews.com, engadget.com
GitHub aggregation-switch rollout: redundant links wrongly disabled
While installing new aggregation switches, GitHub hit bridge loops and then a link-protection feature that shut every uplink when one was deliberately taken down, causing 18 minutes of hard downtime. A separate switch firmware problem kept MAC address tables from updating, leaving some servers unreachable afterwards.
Why not replayed: GitHub's account (5 December 2012) names no MLAG; that is its 22 December outage. The loop came from a partial-link-failure watchdog the vendor said is not supported between the aggregation and access switch models, which is a platform support matrix no config states, and the lasting damage was an aggregation-switch firmware bug that stopped MAC learning. Neither the vendor, the models nor the configuration is public, so a corpus would be invented; MLAG pairing (hammerhead_topo::l2::mlag_pairing) does not change that.
Vlan / trunk change · catalog confidence high · aphyr.com, github.blog, github.blog, github.com, queue.acm.org
GoDaddy DNS outage ('corrupted router data tables')
From about 10:00 EDT for roughly six hours, GoDaddy's DNS servers were unreachable, taking down an estimated millions of hosted and DNS-only customer sites and email. A claimed DDoS attack was ruled out.
Why not replayed: GoDaddy never said whether a change, a bug or hardware corrupted the tables.
BGP / WAN routing · catalog confidence medium · availabilitydigest.com
Windows Azure West Europe: misconfigured network device
A misconfigured network device disrupted traffic to one compute cluster in West Europe. Once a device limit on external connections was reached, previously unknown issues surfaced in another network device in the cluster and complicated recovery; the compute service was unavailable for about 2.5 hours.
Why not replayed: The sources do not name the device type or the setting that limited external connections.
Load balancer / vip config · catalog confidence medium · datacenterknowledge.com, azure.microsoft.com, datacenterknowledge.com
London Stock Exchange / Turquoise 'networking issue' outages
Three separate disruptions: on 26 Nov 2009 the exchange could not complete trades from about 09:30 and blamed connectivity problems; on 5 Oct 2010 trading was disrupted for about an hour; on 2 Nov 2010 the Turquoise platform was down for two hours.
Why not replayed: The operator named the network and published nothing further.
Other · catalog confidence low · computerworld.com, theregister.com, spectrum.ieee.org
Google: web traffic mistakenly routed through Asia
An error in a Google system directed part of Google's web traffic through Asia, creating a bottleneck. About 14% of users saw slow or interrupted Search, Gmail, YouTube and other services for roughly an hour.
Why not replayed: Google never said which system or what change; the path-change gate covers the shape when it is a router change.
Routing policy error · catalog confidence medium · techcrunch.com, en.wikipedia.org, googleblog.blogspot.com
Exported from Hammerhead commit 0ee9732 by scripts/import-incidents.ts.