Cloudflare: a policy term lost its only match and drew the backbone into Atlanta

The network

• 1. CF, The Cloudflare Backbone: "The backbone is a series of private lines between our data centers that we use for faster and more reliable paths between them. These links allow us to carry traffic between different data centers, without going over the public Internet."
• 2. The backbone routers exchange routes over iBGP. CF: "Local-preference is a transitive property on iBGP sessions (it will be transferred to the next BGP peer)", and the change caused the router "to send all its BGP routes to all other backbone routers".
• 3. Edge routers learn local routes from compute nodes, at local-preference 100. CF: "at the time, local routes that the edge routers received from our compute nodes had a local-preference of 100."
• 4. The router that was changed is in Atlanta and its prompt reads atl01# (fact 8). CF names no vendor. The session it prints is Junos configuration mode: show | compare, the [edit policy-options policy-statement ...] path, and the inactive: tag that JN-DEACT documents.
• 5. The network does not depend on the backbone. CF: "our network is built to function properly without a backbone."

The prelude

• 6. CF, Timeline (all UTC): "First, an issue occurred on the backbone link between Newark and Chicago which led to backbone congestion in between Atlanta and Washington, DC." The itemised timeline gives the same two events by site code: "20:25: Loss of backbone link between EWR and ORD" and "20:25: Backbone between ATL and IAD is congesting".

The change

• 7. Intent. CF: "As there was backbone congestion in Atlanta, the team had decided to remove some of Atlanta’s backbone traffic. But instead of removing the Atlanta routes from the backbone, a one line change started leaking all BGP routes into the backbone."
• 8. The change, as CF prints it:

{master}[edit]
atl01# show | compare
[edit policy-options policy-statement 6-BBONE-OUT term 6-SITE-LOCAL from]
!       inactive: prefix-list 6-SITE-LOCAL { ... }

• 9. The term, as CF prints it under "The complete term looks like this:":

from {
    prefix-list 6-SITE-LOCAL;
}
then {
    local-preference 200;
    community add SITE-LOCAL-ROUTE;
    community add ATL01;
    community add NORTH-AMERICA;
    accept;
}

• 10. CF: "This term sets the local-preference, adds some communities, and accepts the routes that match the prefix-list."
• 11. CF: "The correct change would have been to deactivate the term instead of the prefix-list."
• 12. Mechanism. CF: "By removing the prefix-list condition, the router was instructed to send all its BGP routes to all other backbone routers, with an increased local-preference of 200. Unfortunately at the time, local routes that the edge routers received from our compute nodes had a local-preference of 100. As the higher local-preference wins, all the traffic meant for local compute nodes went to Atlanta compute nodes instead."

The failure

• 13. CF: "With the routes sent out, Atlanta started attracting traffic from across the backbone." From the summary: the error "caused all traffic across our backbone to be sent to Atlanta. This quickly overwhelmed the Atlanta router and caused Cloudflare network locations connected to the backbone to fail."
• 14. Nothing became unreachable by routing. The harm CF describes is one location attracting everyone's traffic and being overwhelmed by it. Of its monitoring view CF says: "The red and orange region at the top shows CPU utilization in Atlanta reaching overload, and the white regions show affected data centers seeing CPU drop to near zero as they were no longer handling traffic."
• 15. Scale. CF: "an outage for Internet properties and Cloudflare services that lasted 27 minutes. We saw traffic drop by about 50% across our network." It was "localized to certain geographies": "The affected locations were San Jose, Dallas, Seattle, Los Angeles, Chicago, Washington, DC, Richmond, Newark, Atlanta, London, Amsterdam, Frankfurt, Paris, Stockholm, Moscow, St. Petersburg, São Paulo, Curitiba, and Porto Alegre. Other locations continued to operate normally."
• 16. Timeline, CF, verbatim: "21:12 to 21:39: ATL attracted traffic from across the backbone"; "21:39 to 21:47: ATL dropped from the backbone, service restored"; "21:47 to 22:10: Core congestion caused some logs to drop, edge continues operating"; "22:10: Full recovery, including logs and metrics". Recovery was by removal: "the Atlanta router was disabled and traffic began flowing normally again at 21:39."
• 17. CF: "this was not caused by an attack or breach of any kind."

What Cloudflare changed afterwards

• 18. CF: "Introduce a maximum-prefix limit on our backbone BGP sessions - this would have shut down the backbone in Atlanta, but our network is built to function properly without a backbone. This change will be deployed on Monday, July 20."
• 19. CF: "Change the BGP local-preference for local server routes. This change will prevent a single location from attracting other locations’ traffic similarly. This change has been deployed following the incident."

What Junos does with the statements involved (vendor documentation)

• 20. JN-DEACT, Description: "Add the inactive: tag to a statement, effectively commenting out the statement or identifier from the configuration. Statements or identifiers marked as inactive do not take effect when you issue the commit command." JN-CLI, Table 1, words the last clause as "are ignored when you issue the commit command."
• 21. JN-MATCH: "If you omit the from, all routes are considered to match." (from is in code font in the original.) Facts 20 and 21 together are the mechanism CF describes in fact 12: the only match condition is ignored, so the term matches every route.
• 22. show | display set prints an inactive statement as its set lines plus a deactivate line. JN-SET shows a configuration containing

inactive: unit 1 {
    family inet {
        address 10.0.0.1/8;
    }
}

displayed by show | display set as

set interfaces fe-0/0/0 unit 1 family inet address 10.0.0.1/8
deactivate interfaces fe-0/0/0 unit 1

(We reflowed the braces, which the archived 12.1X46 page runs together on one line; the two display set lines are as printed.)

What the record does not say

• 23. CF prints one term and one compare. It does not print the rest of 6-BBONE-OUT, the contents of prefix-list 6-SITE-LOCAL, what the 6- prefix in those names means, where the policy is applied, the IGP, any AS number, how many routes moved, how the backbone is wired beyond the two links in fact 6, whether route reflectors are used, what the compute nodes run, or any capacity figure for the Atlanta router or its links.

None of the following is on the public record. Each is a choice we made so that the corpus parses, converges and runs in seconds, or something we believe and did not re-read. If one of them is wrong, the replay is wrong in that respect.

• A1. Address family. The names 6-BBONE-OUT and 6-SITE-LOCAL may well mean IPv6. CF does not say (fact 23). The corpus is IPv4 throughout because this build of Hammerhead models no IPv6 forwarding at all (CLAUDE.md, "IPv6 forwarding is unmodeled end to end"). If the real policy was the IPv6 one, the replay differs from it in address family and not in mechanism. We kept the names exactly as printed.
• A2. What follows the term. "Leaking" (fact 7) implies that before the change most BGP routes were not exported to the backbone, so something after term 6-SITE-LOCAL rejected them. We add one term, 6-REJECT-REST { then reject; }. From memory, not re-read: without an explicit reject, Junos falls through to the default BGP export policy, which advertises active BGP routes, and there would have been nothing to leak.
• A3. Where the policy is applied. export 6-BBONE-OUT on one internal BGP group holding every other backbone router, full mesh, no route reflectors. The name and "to all other backbone routers" (fact 12) suggest it; CF does not show it.
• A4. Sites and wiring. Four sites, the four in fact 6: atl01, iad01, ewr01, ord01. Reading IAD as Washington, DC, EWR as Newark and ORD as Chicago comes from setting CF's prose beside its timeline; CF never states the mapping. The backbone is a line, EWR - IAD - ATL - ORD. The EWR - ORD link is left out of both snapshots because it had been lost 47 minutes before the change (fact 6) and Hammerhead reads configuration, not link state: had we included it, the model would have treated it as up. That a broken EWR - ORD ring is why ATL - IAD congested is our inference. The real backbone had at least the nineteen locations of fact 15 and far more links.
• A5. IGP. OSPF area 0 on /31 backbone links, metric 10, loopbacks passive. The compute-facing and transit-facing interfaces are passive in OSPF so that iBGP next hops resolve without next-hop self, which the printed term does not set. CF says nothing about the IGP.
• A6. Compute nodes. One per site, written in FRR syntax, speaking eBGP to the edge router from a private AS, announcing the site's own /16 and two service /24s. The same two /24s, and the same service addresses on loopback, exist at every site: that is anycast, which CF's post implies ("traffic meant for local compute nodes went to Atlanta compute nodes instead") and never names; that Cloudflare serves from anycast addresses is our memory, not re-read. What the real nodes ran, how many there were and how they peered is not public. The edge import policy sets local-preference 100 explicitly; fact 3 gives the value, not how it was set.
• A7. Transit. Two stand-in transit providers in Cisco IOS syntax, each serving two sites, each originating the same sixteen placeholder /24s from Null0 statics. "All its BGP routes" (fact 12) would have included routes like these; CF does not enumerate them. The /24s are 100.64.0.0/24 to 100.64.15.0/24 from RFC 6598 shared address space, which nobody holds. One router per provider attached to two cities is an abstraction of "a provider", not a claim about anyone's network.
• A8. Prefix-list contents. 6-SITE-LOCAL holds the site's own /16 (10.1.0.0/16 at Atlanta). A second list, 6-ANYCAST, and the policies 6-COMPUTE-IN, 6-COMPUTE-OUT, 6-TRANSIT-IN, 6-TRANSIT-OUT are ours entirely.
• A9. Communities. The names SITE-LOCAL-ROUTE, ATL01 and NORTH-AMERICA are CF's (fact 9). Their values (65000:100, 65000:1001, 65000:200) are invented, as are IAD01, EWR01, ORD01 by analogy.
• A10. AS numbers. All from the private range: backbone 65000, compute 65101 to 65104, transit 64601 and 64602. Cloudflare's real AS number appears nowhere.
• A11. Scale. 54 (device, prefix) best paths move in the corpus. The real number is not public.
• A12. The two other renderings. CF shows brace form only. The display set rendering in junos_set/ follows the shape of JN-SET's example (fact 22); that Cloudflare ever held its configs in that form is not claimed. The Arista EOS rendering in eos/ exists only so the rest of the pipeline can be evaluated on a change it parses cleanly. EOS has no inactive:; the equivalent edit is deleting the match ip address prefix-list line from route-map sequence 10 and leaving the sequence. That a permit sequence with no match clause matches every route on EOS is from memory, not re-read.
• A13. The control. after_intended/ is our rendering of fact 11, "deactivate the term": inactive: term 6-SITE-LOCAL { ... } in Junos, sequence 10 deleted in EOS. CF does not print that change.
• A14. Which routes an export term sees. JN-MATCH says a term with no from matches "all routes" (fact 21), which on Junos means every active route of any protocol, direct and OSPF included. CF says "all its BGP routes" (fact 12). Hammerhead evaluates a BGP export policy against BGP routes only, which agrees with CF's description. Whether the real router also exported non-BGP routes is not on the record.
• A15. Load is not modeled at all. The harm was congestion (facts 13 and 14). The simulator computes converged routing state. It has no traffic volumes, no link or router capacity, no CPU, and no notion of time. With preflight --link-capacity an operator can DECLARE link capacities and offered traffic, and the declared demand is then routed over both snapshots. That is still no traffic model: the numbers are whoever declared them, and router CPU (what fact 14 describes) is in neither.
• A16. The capacity figures are ours, and illustrative. Fact 23 says the record gives no capacity figure, and it gives no traffic volume either. tests/gauntlet/cloudflare_2020/capacity/link_capacity.yaml declares every link at 10 Gb/s and 6 Gb/s offered at each site's edge towards the anycast service address 198.51.100.1; link_capacity_roomy.yaml is the same demand at 40 Gb/s a link, as a control. With the policy widening and the path change waived, the change fails on load alone: atl01's link to its compute node would carry 24 Gb/s against 10, and iad01's backbone link to atl01 12 Gb/s against 10. At 40 Gb/s a link it passes. Rows in tests/gauntlet/replays-capacity.yaml, written by tools/replays/gen_family_capacity.py.