Facebook: backbone taken down by a maintenance command

The network

• 1. There is a private backbone that joins every facility. M1: "The backbone is the network Facebook has built to connect all our computing facilities together", made of "tens of thousands of miles of fiber-optic cables".
• 2. Facilities come in two kinds. M1: some are "massive buildings that house millions of machines", and "others are smaller facilities that connect our backbone network to the broader internet and the people using our platforms."
• 3. A user request lands at a small facility and crosses the backbone. M1: the request "travels from your device to the nearest facility, which then communicates directly over our backbone network to a larger data center."
• 4. Routers carry that traffic, and taking backbone capacity out of service is routine. M1 says engineers "often need to take part of the backbone offline for maintenance", and gives fibre repair, added capacity and router software updates as examples.
• 5. The small facilities answer DNS. M1: "One of the jobs performed by our smaller facilities is to respond to DNS queries."
• 6. The authoritative name servers sit on well-known addresses that are announced with BGP. M1: the name servers "occupy well known IP addresses themselves, which in turn are advertised to the rest of the internet via another protocol called the border gateway protocol (BGP)."

The change

• 7. It was a maintenance command, not a planned design change. M1 says that during a routine maintenance job a command was issued "with the intention to assess the availability of global backbone capacity", and that it "unintentionally took down all the connections in our backbone network", disconnecting the data centres globally.
• 8. The operator's first statement located the change on the backbone routers. M0: "configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication." M0 also calls the root cause "a faulty configuration change on our end" and says there was "no malicious activity".
• 9. There was a pre-execution check, and it failed. M1: "Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping the command."
• 10. The content of the command is not public. Neither M1 nor M0 gives the command, the router vendor, the operating system, the routing protocols on the backbone, or the number of routers or links affected.

The failure

• 11. First effect: total loss of connectivity between data centres and the Internet. M1: "This change caused a complete disconnection of our server connections between our data centers and the internet."
• 12. Second effect, which M1 says "made things worse": the DNS servers withdrew their own BGP advertisements. This is designed behaviour. M1: the DNS servers "disable those BGP advertisements if they themselves can not speak to our data centers", because that indicates an unhealthy connection.
• 13. That rule fired everywhere at once. M1: "the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP advertisements."
• 14. The DNS servers were healthy machines that had made themselves unreachable. M1: "our DNS servers became unreachable even though they were still operational. This made it impossible for the rest of the internet to find our servers."
• 15. What an outside network saw was a withdrawal of the DNS prefixes specifically. CF: "At 15:58 UTC we noticed that Facebook had stopped announcing the routes to their DNS prefixes." CF shows 185.89.218.0/23 and 129.134.30.0/23 as examples of the affected prefixes and gives Facebook's AS number as 32934.
• 16. Timeline as seen from outside (CF, all UTC). "at around 15:40 UTC we saw a peak of routing changes from Facebook"; Cloudflare opened an internal incident at 15:51 titled "Facebook DNS lookup returning SERVFAIL"; Facebook's DNS "stopped being available at around 15:50 UTC and returned at 21:20 UTC"; "At around 21:00 UTC we saw renewed BGP activity from Facebook's network which peaked at 21:17 UTC."
• 17. Public resolvers answered SERVFAIL for Facebook names, and retry traffic loaded resolvers everywhere. CF reports resolvers handling "30x more queries than usual".
• 18. The outside observer did not know the cause. CF attributes the root cause to Facebook's own statement: externally they "saw the BGP and DNS problems", "but the problem actually began with a configuration change".

Recovery

• 19. Remote access was gone. M1 says the data centres could not be reached by normal means "because their networks were down", that the loss of DNS "broke many of the internal tools" used to investigate outages, and that "primary and out-of-band network access was down", so engineers were sent on site.
• 20. Physical and system security slowed the repair. M1 describes the facilities as "hard to get into" and the hardware as difficult to modify even with physical access.
• 21. Restoring the backbone restored everything, and had to be done gradually. M1 says that once backbone connectivity returned "everything came back up with it", and reports power dips "in the range of tens of megawatts" that made a sudden reversal risky for electrical systems and caches.
• 22. This exact scenario had not been rehearsed. M1: "we've never previously run a storm that simulated our global backbone being taken offline".

None of the following is on the public record. Each is a choice we made so that the corpus parses, converges and runs in under a second. If one of them is wrong, the replay is wrong in that respect.

• A1. Vendor and syntax. No vendor, operating system or protocol is public (fact 10). Every device is written in Cisco IOS syntax because that is where this parser models interface shutdown, OSPF and BGP sessions most completely (100% of lines classified, see RESULT.md). That is a statement about our parser and says nothing about what Meta runs. Meta's DNS speakers are servers (fact 12), not routers; writing them as IOS devices is a stand-in so that they can originate BGP routes in this model.
• A2. What the command did, as configuration. The record says a command "took down all the connections in our backbone network" (fact 7) and that the change was on "the backbone routers" (fact 8). We express that as shutdown on every backbone-facing interface of both backbone routers: ten lines on two devices, and nothing else differs between before/ and after/. Removing the IGP adjacencies or the iBGP sessions instead would give the model the same converged state. We do not know the real form.
• A3. Topology. Two backbone routers, two edge facilities and two data centres, with every facility dual-homed to both backbone routers, are ours. It is the smallest shape that has "small facilities that front the Internet", "large facilities reached only over the backbone" and "a backbone whose loss separates them" (facts 1 to 3). The real backbone is global and far larger.
• A4. Routing design. OSPF area 0 on the backbone, iBGP between loopbacks with the backbone routers as route reflectors, next-hop-self at the edge, eBGP from each DNS speaker to its local peering router, and eBGP from each peering router to one peer. All of it is ours. The record says only that DNS addresses are "advertised to the rest of the internet via" BGP (fact 6). AS 65000, 65101 and 65102 are private; AS 64496 and 64497 are from the documentation range.
• A5. The DNS withdrawal is NOT in the committed `after/` snapshot, on purpose. This is the central modelling decision. The real withdrawal was made by software on the DNS servers when a health check failed (facts 12 and 13). A configuration snapshot cannot express "stop advertising when you cannot reach the data centres", and no configuration line changed on any DNS server. So after/ contains only what a change reviewer would actually have been handed: the backbone change. The withdrawal is downstream runtime behaviour. We did not add it to after/ by hand, because a tool that is given the consequence as an input has not predicted it. We did separately test the nearest thing IOS configuration can say, conditional advertisement (neighbor ... advertise-map ... exist-map ...), in a scratch copy of the corpus. That variant, and what the tool made of it, is reported in RESULT.md; it is not committed because it is a second guess layered on A1.
• A6. The health-check target. Fact 12 says the DNS servers test whether they can "speak to our data centers". What they probe, how often and with what threshold is not public. intent.yaml encodes it as "each DNS speaker can reach one service address in one data centre" (10.128.0.1, 10.129.0.1). The addresses and the rule are ours.
• A7. Prefixes. The anycast DNS prefixes are 192.0.2.0/24 and 198.51.100.0/24, the peering links are in 203.0.113.0/24 (all RFC 5737 documentation space), the peers' subscriber pools are in 100.64.0.0/10 (shared address space), and everything internal is RFC 1918. We deliberately did not use the two real prefixes CF shows (fact 15). Both DNS sites originate the same two prefixes and hold the same two service addresses, which is what makes it anycast; that the real deployment is anycast is our reading of "well known IP addresses" served from many "smaller facilities" (facts 5 and 6), not a quoted statement.
• A8. The peers are inside the snapshot. ISP-A and ISP-B exist so the model has somewhere to show whether the Internet still holds a route to the DNS prefixes. No operator holds its peers' configurations. RESULT.md therefore also reports a run with both files left out, with and without the boundary declaration in boundary.yaml. Every value in that file is ours.
• A9. Outbound policy. That each peering router advertises only the DNS prefixes to its peer (route-map TO-PEER) is ours. CF says other Facebook routes stayed announced while the DNS prefixes went (fact 15); we did not model those other routes because a documentation-range corpus has no address space left for them, and they play no part in the failure.
• A10. Timing, scale and recovery are not modelled at all. The simulator computes converged routing state. Nothing here represents the speed of the failure, the 30x resolver load (fact 17), the loss of out-of-band access (fact 19), physical access (fact 20) or the power and cache risks of restoration (fact 21).

Remembered, not re-read

We recall the following from general coverage of the incident. None of it was in the text we fetched on 2026-09-20, so none of it is used as a fact, in the corpus or in RESULT.md: the names of Facebook's authoritative name servers; reports that building access systems and internal communication tools failed for employees; the total outage duration quoted in press coverage (CF's times in fact 16 give roughly 15:40 to 21:20 UTC for DNS, which is the only duration we rely on); and any figure for the number of prefixes or BGP updates involved.