CenturyLink / Level 3 global outage from a Flowspec rule

What CenturyLink said on the day

• 1. The final customer notice (OUT, pasted by Jörg Kost under "latest statement (Flowspec)"): "On August 30, 2020 10:04 GMT, CenturyLink identified an issue to be affecting users across multiple markets. The IP Network Operations Center (NOC) was engaged, and initial research identified that an offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing across multiple elements throughout the CenturyLink Network. The IP NOC deployed a global configuration change to block the offending flowspec announcement, which allowed BGP to begin to correctly establish. As the change propagated through the network, the IP NOC observed all associated service affecting alarms clearing and services returning to a stable state."
• 2. The notes history before it (OUT, pasted by Stephen Flynn). 13:19:02 GMT: "The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets." 14:46:29 GMT: "confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability." 15:31:25 GMT: "The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices."
• 3. Customers were told their own sessions might not come back unaided (OUT): "Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session."
• 4. The notice did not say what the rule was, and the list did not know. Randy Bush, the same afternoon (OUT-2): "opsec paranoid here wonders about the content of the flowspec announcement and whence it originated."

Timeline and scale

• 5. Start. CF: "At 10:03 UTC our monitoring systems started to observe an increased number of errors reaching our customers' origin servers." and "Starting at 10:04 UTC, there were a significant number of BGP updates." OUT-2 (Tribus): "IPv4 flapping and no traffic at all on 3356 transit links since approximately 10:00 UTC at multiple EMEA locations".
• 6. End. CF: customers single-homed to the provider "continued to see 522 errors until CenturyLink/Level(3) resolved their issue around 14:30 UTC". TE: the outage "did not fully resolve until approximately 3:30PM UTC". CP: "more than 5 hours (1000 UTC till 1530 UTC on a Sunday morning)". CBR: "The event wasn't fully cleared until 15:10 GMT." The sources do not agree on one end time; the notes history in fact 2 brackets it.
• 7. Traffic. CF: "Globally, we saw a 3.5% drop in global traffic during the outage, nearly all of which was due to a nearly complete outage of CenturyLink's ISP service across the United States."
• 8. BGP churn. CF: "Under normal conditions, the Internet sees about 1.5MBs – 2MBs of BGP updates every 15 minutes. At the start of the incident, the number of BGP updates spiked to more than 26MBs of BGP updates per 15 minute period and stayed elevated throughout the incident." CP attributes the announcements to "AS 3356 (Level3)", which before the outage "was appearing in the AS path of about 14% of the IPv4 routes and 10% of the IPv6 routes collected by the RouteViews collector".
• 9. It looked IPv4-only from outside. OUT-2 (Bonfiglio): "It's apparently affecting IPv4 only, v6 works just fine."

How the rule came to be wrong (second-hand: CBR quoting the RFO)

• 10. What Flowspec was being used for. CBR, quoting the RFO: "As a large influx of traffic is identified from a set IP address, the Operations Engineering Team utilizes Flowspec announcements as one of many tools available to block the corrupt source from sending traffic to the CenturyLink network."
• 11. The request and the fault. CBR, quoting the RFO: "The Operations Engineering Team was using this process during routine operations to block a single IP address on a customer's behalf as part of our normal product offering. When the user attempted to block the address, a fault between the user interface and the network equipment caused the command to be received with wildcards instead of specific numbers. This caused the network to recognize the block as several IP addresses, instead of a single IP as intended. The user interface for command entry is designed to prohibit wildcard entries, blank entries, and only accept IP address entries."
• 12. The safeguard that did not fire. CBR, quoting the RFO: "A secondary filter that is designed to prevent multiple IP addresses from being blocked in this fashion failed to recognize the command as several IP addresses. The filter specifically looks for destination prefixes, but the presence of the wildcards caused the filter to interpret the command as a single IP address instead of many, thus allowing it to pass… the problematic protocol [then] propagated through many of the edge devices on the CenturyLink Network."
• 13. It was not a DDoS response. CBR: the cause "was not, as we had initially speculated, the result of a botched attempt to mitigate a Distributed Denial of Service attack. Rather, it was down to a failed attempt by the company to 'block a single IP address' on behalf of a customer". TE reads the same document the same way: "the customer requested CenturyLink block traffic from a specific IP address. The flowspec for this request was accidentally implemented with wildcards, rather than isolated to a specific IP address."
• 14. Afterwards. CBR: "The Flowspec announcement platform has been disabled from service on the CenturyLink Network in its entirety", to "remain offline until extensive testing is conducted", while the secondary filter is "modified to prohibit wildcard entries."

What Flowspec is, and what outsiders inferred

• 15. CF: "Flowspec is an extension to BGP, which allows firewall rules to be easily distributed across a network, or even between networks, using BGP … It allows you to efficiently push rules across an entire network almost instantly." TE: "It functions similarly to access control lists (ACLs), but unlike ACLs, which are static, flowspec, as part of BGP, is dynamic." CP cites it as RFC 5575.
• 16. Cloudflare's example rule is labelled speculation by its author. CF: "We can only speculate what happened at CenturyLink/Level(3) … The status report indicates that the Flowspec rule prevented BGP itself from being announced. We have no way of knowing what that Flowspec rule was, but here's one in Juniper's format that would have blocked all BGP communications across their network." The example is:

route DISCARD-BGP { match { protocol tcp; destination-port 179; } then discard; }

• 17. The re-learn loop is also an inference, not a CenturyLink statement. CF introduces it with "One possible explanation is": "every router in CenturyLink/Level(3)'s network would receive the Flowspec rule. They would then block BGP. That would cause them to stop receiving the rule. They would start back up again, working their way through all the BGP rules until they got to the offending Flowspec rule again. BGP would be dropped again. The Flowspec rule would no longer be received. And the loop would continue, over and over." TE gives the same account and adds the reason the rule does not stick: "the flowspec rule would not persist past termination of BGP".
• 18. Why it took hours. CF, again under "we can only speculate": the rule and the update load "made it difficult for them to login to their own interfaces", and "it also may have been that the Flowspec rule was not issued by CenturyLink/Level(3) themselves but rather by one of their customers. Many network providers will allow Flowspec peering." (Fact 11 later settled the second point: CenturyLink's own staff issued it.)
• 19. Other networks had to cut CenturyLink off to help it recover. CF: "Several of the other tier-1 providers took action, it appears at CenturyLink/Level(3)'s request, to de-peer their networks." TE: "Telia, indicated that at approximately 14:00 UTC CenturyLink requested the provider no longer peer with its network in order to reduce the number of route announcements it received".
• 20. Withdrawals were not honoured while it lasted. CF: the network "was not honoring route withdrawals and continued to advertise routes to networks like Cloudflare's even after they'd been withdrawn." TE: "Level 3 was not able to honor any BGP changes from peers during the incident, most likely due to an overwhelmed controlplane."

What is NOT on the record

• 21. The rule's match fields, action and origin router. The router vendors and models. The backbone's IGP, its route-reflection design, and how the Flowspec address family was deployed (which routers installed rules, whether local-install was restricted by interface). Whether any validation of the rule ran besides the "secondary filter" of fact 12. None of the sources above says, and we found no source that does.

None of the following is on the public record. Each is a choice we made so that the corpus parses, converges and runs in seconds. If one of them is wrong, the replay is wrong in that respect.

• A1. Topology. Nine routers: two cores that are also iBGP route reflectors, three provider edges that are their clients, two customers and two settlement-free peers. Every edge has one link to each core and the cores have one link between them. The real backbone had thousands of routers in hundreds of sites; fact 12's "many of the edge devices" is the only structural statement on the record and is why the filter lands on edges here.
• A2. Protocols. OSPF area 0 as the IGP, loopback-to-loopback iBGP with route reflection, eBGP to customers and peers. Fact 21: none of this is public. OSPF was chosen over IS-IS only because it is the better exercised path in this tool.
• A3. Numbers. The provider is AS 64500, from the documentation range, not AS 3356. Customers originate the RFC 5737 documentation blocks 198.51.100.0/24 and 203.0.113.0/24; PE-CE links are /30s cut from 192.0.2.0/24; the peers originate sixteen /24s from the shared address space 100.64.0.0/10. They are placeholders. An earlier draft used 198.18.0.0/15 for the peers; this build's Junos front end installs that block as a system discard route on every Junos device, so it was moved to keep the question out of the results.
• A4. Vendors. Fact 21: not public. The corpus is deliberately mixed so that each candidate exercises more than one front end: CORE-1 is IOS-XR, EDGE-2 is Junos (brace form), the other seven are IOS. CF's use of "Juniper's format" for its example says nothing about what CenturyLink ran, and we infer nothing from it.
• A5. Syntax chosen to suit this build's parsers. Four places where the natural way to write the healthy network does not survive this tool, each recorded with a reproducible fixture in RESULT.md ("Odd things"): the Junos iBGP group carries a redundant peer-as 64500, because without it the parser leaves remote-as at 0 and the sessions never establish; the Junos edge carries its customer link as a passive OSPF interface instead of exporting next-hop self, because that policy action is a hard parse error; the Junos import policy matches a prefix-list rather than a route-filter, because a route-filter is stored as a reference to an ACL that does not exist and rejects every route; and the IOS-XR file has no comment header and the IOS core no bgp cluster-id, because both count as unparsed lines and the base snapshot has to be 100% classified for the unparsed-lines delta to mean anything. Every one of these makes the tool's job easier. None touches the change under test.
• A6. The rule. Fact 16 is speculation and we adopt it as such. The story the corpus tells is ours: customer B asks for one source, 100.64.5.66, to be blocked from reaching its BGP speaker 203.0.113.1 on tcp/179; both hosts arrive as wildcards (fact 11); what is left is "tcp, port 179, discard". That is consistent with facts 1, 11 and 16 and proven by none of them. The real rule may have matched something else entirely that also caught BGP.
• A7. `after_flowspec` is the rule as static configuration. Junos: routing-options flow route MITIGATION-TICKET with match { protocol tcp; port 179; } and then discard, plus family inet { unicast; flow; } on the iBGP group. IOS-XR: class-map type traffic / policy-map type pbr / flowspec … service-policy type pbr, plus address-family ipv4 flowspec under BGP and each neighbor. Two devices and two syntaxes so that both front ends are exercised; a real deployment has one injection point. It also bundles enabling the address family with the rule, which the real network had done long before (fact 10 calls it "routine operations"). And a real rule of this kind is usually never configuration at all: see RESULT.md section 5d.
• A8. `after_acl_equivalent` is a translation, and a narrowing one. The same match as an inbound IOS ACL (deny tcp any any eq bgp, deny tcp any eq bgp any, permit ip any any) and an inbound Junos firewall filter (separate destination-port 179 and source-port 179 discard terms), bound to the six backbone-facing interfaces of the three edges. A Flowspec rule is normally installed on every interface of every router that accepts it; binding only the edges' backbone interfaces keeps the expected outcome easy to state and check by hand: all six edge-to-reflector iBGP sessions die, the five eBGP sessions and the core-to-core session survive.
• A9. Timing, load and the loop are not modeled at all. The simulator computes converged state. It has no notion of a session flapping, of a rule that disappears when the session carrying it drops, of update queues or of CPU (facts 8, 17, 18, 20).