Cloudflare: policy terms reordered in 19 data centres

The network

• 1. Nineteen data centres ran the architecture that failed. CF22, Background: "Over the last 18 months, Cloudflare has been working to convert all of our busiest locations to a more flexible and resilient architecture. In this time, we've converted 19 of our data centers to this architecture, internally called Multi-Colo PoP (MCP): Amsterdam, Atlanta, Ashburn, Chicago, Frankfurt, London, Los Angeles, Madrid, Manchester, Miami, Milan, Mumbai, Newark, Osaka, São Paulo, San Jose, Singapore, Sydney, Tokyo."
• 2. It is a Clos fabric with a spine layer. CF22: "A critical part of this new architecture, which is designed as a Clos network, is an added layer of routing that creates a mesh of connections. ... This layer is represented by the spines in the following diagram."
• 3. CF22-FIG1 draws, top to bottom: Internet; two routers labelled edge01 and edge02; three labelled spine01, spine02, spine03, each connected to both edges and not to each other; under each spine two switches labelled sw01 and sw02; under each switch a stack labelled Metals.
• 4. Policies are ordered. CF22, Incident timeline and impact: "operators define policies which decide which prefixes ... are advertised to peers ..., or accepted from peers. These policies have individual components, which are evaluated sequentially. The end result is that any given prefixes will either be advertised or not advertised." CF22-FIG2 draws a "BGP Policy Evaluator" between "Network Prefix" and "Peers", containing "Term 1 / Match X → Accept", "Term 2 / Match Y → Accept", "Term n / Match Z → Reject".
• 5. What the site-local prefixes are for. CF22: "These prefixes allow our metals to communicate with each other, as well as connect to customer origins."

The change

• 6. Its purpose. CF22, Technical description: "we were rolling out a change to standardize the BGP communities we attach to a subset of the prefixes we advertise. Specifically, we were adding informational communities to our site-local prefixes."
• 7. Its process. CF22: "a Change Request ticket was created, which includes a dry-run of the change, as well as a stepped rollout procedure. Before it was allowed to go out, it was also peer reviewed by multiple engineers. Unfortunately, in this case, the steps weren't small enough to catch the error before it hit all of our spines."
• 8. What it looked like on a router that was NOT a spine ("The change looked like this on one of the routers"), four stanzas of this shape:

[edit policy-options policy-statement 4-COGENT-TRANSIT-OUT term ADV-SITELOCAL then]
+      community add STATIC-ROUTE;
+      community add SITE-LOCAL-ROUTE;
+      community add TLL01;
+      community add EUROPE;

and the same four lines under 4-PUBLIC-PEER-ANYCAST-OUT, 6-COGENT-TRANSIT-OUT and 6-PUBLIC-PEER-ANYCAST-OUT, each in term ADV-SITELOCAL then. CF22: "This was harmless, and just added some additional information to these prefix advertisements."

• 9. What it looked like on the spines ("The change on the spines was the following"):

[edit policy-options policy-statement AGGREGATES-OUT]
term 6-DISABLED_PREFIXES { ... }
!    term 6-ADV-TRAFFIC-PREDICTOR { ... }
!    term 4-ADV-TRAFFIC-PREDICTOR { ... }
!    term ADV-FREE { ... }
!    term ADV-PRO { ... }
!    term ADV-BIZ { ... }
!    term ADV-ENT { ... }
!    term ADV-DNS { ... }
!    term REJECT-THE-REST { ... }
!    term 4-ADV-SITE-LOCALS { ... }
!    term 6-ADV-SITE-LOCALS { ... }
[edit policy-options policy-statement AGGREGATES-OUT term 4-ADV-SITE-LOCALS then]
community delete NO-EXPORT { ... }
+      community add STATIC-ROUTE;
+      community add SITE-LOCAL-ROUTE;
+      community add AMS07;
+      community add EUROPE;
[edit policy-options policy-statement AGGREGATES-OUT term 6-ADV-SITE-LOCALS then]
community delete NO-EXPORT { ... }
+      community add STATIC-ROUTE;
+      community add SITE-LOCAL-ROUTE;
+      community add AMS07;
+      community add EUROPE;

• 10. How to read it. CF22: "In this diff format, the exclamation marks in front of the terms indicate a re-ordering of the terms. In this case, multiple terms moved up, and two terms were added to the bottom. Specifically, the 4-ADV-SITE-LOCALS and 6-ADV-SITE-LOCALS terms moved from the top to the bottom. These terms were now behind the REJECT-THE-REST term, and as might be clear from the name, this term is an explicit reject:"

term REJECT-THE-REST {
    then reject;
}

• 11. The summary sentence. CF22: "While deploying a change to our prefix advertisement policies, a re-ordering of terms caused us to withdraw a critical subset of prefixes."

The failure

• 12. The direct effect. CF22: "As this term is now before the site-local terms, we immediately stopped advertising our site-local prefixes, removing our direct access to all the impacted locations, as well as removing the ability of our servers to reach origin servers."
• 13. The secondary effect. CF22: "the removal of these site-local prefixes also caused our internal load balancing system Multimog (a variation of our Unimog load-balancer) to stop working, as it could no longer forward requests between the servers in our MCPs. This meant that our smaller compute clusters in an MCP received the same amount of traffic as our largest clusters, causing the smaller ones to overload."
• 14. Scope. CF22: "Even though these locations are only 4% of our total network, the outage impacted 50% of total requests."
• 15. Timeline, all UTC, from CF22: - "03:56 UTC: We deploy the change to our first location. None of our locations are impacted by the change, as these are using our older architecture." - "06:17: The change is deployed to our busiest locations, but not the locations with the MCP architecture." - "06:27: The rollout reached the MCP-enabled locations, and the change is deployed to our spines. This is when the incident started, as this swiftly took these 19 locations offline." - "06:32: Internal Cloudflare incident declared." - "06:51: First change made on a router to verify the root cause." - "06:58: Root cause found and understood. Work begins to revert the problematic change." - "07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically." - "08:00: Incident closed."
• 16. Getting back in was hard. CF22: "Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations."
• 17. It was not an attack. CF22: "This was our error and not the result of an attack or malicious activity."

What Cloudflare said it would change

• 18. Process. CF22: "While we did use a stagger procedure for this change, the stagger policy did not include an MCP data center until the final step. Change procedures and automation need to include MCP-specific test and deploy procedures".
• 19. Architecture. CF22: "Ultimately the policy statement that caused the incorrect routing advertisement will be redesigned to prevent an unintentional incorrect ordering."
• 20. Automation. CF22: "we will be concentrating on automation improvements that enforce an improved stagger policy for rollouts of network configuration and provide an automated "commit-confirm" rollback."

None of the following is on the public record. Each is a choice we made so that the corpus parses, converges and runs in seconds. If one of them is wrong, the replay is wrong in that respect.

• A1. Vendor and syntax of the spines and edges. CF22 never names a vendor. Its excerpts use [edit policy-options policy-statement ...], term, then reject; and community add, which is Junos configuration syntax, so the spines, the edges and the stand-in transit router are written in Junos brace form. That is an inference from syntax, not a statement by Cloudflare.
• A2. Every term body. CF22 prints the term NAMES and elides every body except REJECT-THE-REST (fact 10) and the community delete NO-EXPORT line inside the two site-local terms (fact 9). What each term matches is ours: every IPv4 term matches one prefix-list, every 6- term matches one IPv6 route-filter. The names, the two orders and then reject; are the only parts of AGGREGATES-OUT that come from the record.
• A3. The "before" order. CF22 prints the order AFTER the change and says the two site-local terms "moved from the top to the bottom" (fact 10). 6-DISABLED_PREFIXES carries no ! marker, so we leave it first and put 4-ADV-SITE-LOCALS, 6-ADV-SITE-LOCALS directly after it, ahead of the product terms. Their exact original position among "the top" is not stated.
• A4. `after/` is the reorder and nothing else. The real commit also added four community add lines to each site-local term (fact 9). We leave them out of after/ so that the only difference between the snapshots is term order, and we run the additions alone as a control (generate.py --variant intended, see RESULT.md). Community names STATIC-ROUTE, SITE-LOCAL-ROUTE and EUROPE in that control are CF22's; their numeric values, and POD0n in place of the site code, are ours.
• A5. Topology. Two edges, three spines, that naming, and the edge-to-every-spine wiring are CF22-FIG1. We draw one switch per spine where the figure draws two, name them s1-sw01 to s3-sw01 because hostnames must be unique in a snapshot, and model the metals as a connected subnet on the switch, not as devices. transit01 stands in for the cloud labelled "Internet"; one router with one "customer origin" subnet is ours.
• A6. Switch vendor. CF22 says nothing about the switches. They are written in Arista EOS syntax for a modelling reason, not a historical one: the anycast service prefixes have to be originated in every pod, the topology builder joins devices that have interfaces in the same subnet into one segment (RESULT.md, "Odd things", item 5), and this build does not originate a Junos static route into BGP (RESULT.md, gap G3). EOS network statements over Null0 statics avoid both. ip routing is omitted from those files because the EOS parser leaves it unclassified (RESULT.md, "Odd things").
• A7. BGP layout. CF22 does not say which sessions exist or whether they are internal or external. We run eBGP on every link with one private AS per tier or pod (edges 65000, spines 65101 to 65103, switches 65201 to 65203, transit 64496), no IGP, and no route reflection. AGGREGATES-OUT is the spines' export policy toward the edges; that it faces the edges is our reading of "stopped advertising" plus the figure, not a quotation. The edges' 4-TRANSIT-OUT borrows the term name ADV-SITELOCAL from fact 8, which CF22 shows on a non-MCP router.
• A8. What "site-local" contains. Fact 5 says the prefixes serve metal-to-metal and metal-to-origin traffic; fact 12 says losing them removed "our direct access to all the impacted locations". We therefore put three things in each pod's site-local list: the metals subnet, the spine loopback and the switch loopback. This choice decides whether the reachability sweep sees the failure, because the sweep probes one loopback per device and never a metals subnet. RESULT.md runs the corpus both ways (--variant no-infra-loopbacks) and reports both.
• A9. How the site-locals reach the spine. The community name STATIC-ROUTE (fact 9) hints that the real ones were static routes. In the committed corpus the switch announces them to the spine over eBGP and the spine's term matches them by prefix-list, because that is a shape this build evaluates. The static shape is run as a variant (--variant static-origin) and its result is in RESULT.md.
• A10. Addresses. Everything is from documentation or private space: anycast service prefixes are /27s of 192.0.2.0/24, site-locals are /26s and /32s of 198.51.100.0/24, the customer origin is 203.0.113.0/24, links and the remaining loopbacks are 10.0.0.0/8, IPv6 is 2001:db8::/32. Real anycast prefixes are not /27s. No address here is Cloudflare's.
• A11. IPv6. The 6- terms are present, in CF22's order, and inert: this build models no IPv6 forwarding, and an IPv6 entry inside a Junos prefix-list is a fatal parse error (RESULT.md), so the terms match with route-filter and no interface carries an IPv6 address.
• A12. Not modelled at all. Timing, the staggered rollout (facts 7, 15, 18), the load balancer and the overload of small clusters (fact 13), the loss of management access as an operational event (fact 16), and engineers reverting each other (fact 15). The simulator computes converged routing state for two snapshots and nothing else.