HomeIncident library
Incident library

INCIDENT REPLAY · 2020-06-09

IBM Cloud global outage: external provider flooded incorrect routes

IBM Cloud sites worldwide lost connectivity for about 3 hours. IBM said an external network provider sent it incorrect routing, which caused severe congestion across the IBM Cloud network.

Accepting networkRouting policy errorCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: IBM Cloud; unnamed external network provider

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • IBM Cloud sites worldwide lost connectivity for about 3 hours. IBM said an external network provider sent it incorrect routing, which caused severe congestion across the IBM Cloud network.
  • Who: IBM Cloud; unnamed external network provider

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The family template (tools/replays/gen_family_audit.py): IBM-EDGE-1 (AS 36351, an IBM Cloud edge) has a session to PROVIDER-1 (AS 64500, the external provider that flooded incorrect routes) with no inbound prefix filter, no prefix limit, no AS-path filter, and a second session with every safeguard, so that each finding is one row.
  • The bad routes were a third party's. Whether IBM's edge would accept and prefer them is a property of its own configuration: modelled as a provider session with no inbound filter and no prefix limit.
  • No change is replayed: the announcing network's change is not ours to gate. The audit runs on the accepting router's healthy configuration.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

IBM-EDGE-1's own router, no change: no inbound filter and no prefix limit

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.ebgp_peer_unfiltered_and_unlimited. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • IBM-EDGE-1
  • 64500

First run, before the fixes this batch prompted

first: fail (finding)

1 row(s) returned (expected 0); first: s.device=IBM-EDGE-1 s.peer=172.16.0.2 s.remote_as=64500

Pinned in tests/gauntlet/replays-audit.yaml.

IBM-EDGE-1's own router, no change: no inbound filter

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.route_map_in_attached. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • IBM-EDGE-1
  • 64500

First run, before the fixes this batch prompted

first: fail (finding)

1 row(s) returned (expected 0); first: s.device=IBM-EDGE-1 s.peer=172.16.0.2 s.remote_as=64500

Pinned in tests/gauntlet/replays-audit.yaml.

IBM-EDGE-1's own router, no change: no prefix limit

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.max_prefix_configured. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • IBM-EDGE-1
  • 64500

First run, before the fixes this batch prompted

first: fail (finding)

1 row(s) returned (expected 0); first: s.device=IBM-EDGE-1 s.peer=172.16.0.2 s.remote_as=64500

Pinned in tests/gauntlet/replays-audit.yaml.

What changed in Hammerhead because of it

From the “What was fixed” table in Hammerhead's replay notes, where each fix landed after a failing test. “Found by” is copied as written; a family name means the batch this incident belongs to.

Inbound max prefix length and AS-path filter on BgpSession

prefix_list_max_length and a new ebgp_inbound_as_path_filter audit stop being stubs: a /32 for somebody else's address wins by longest match wherever it is accepted; a prefix filter generated from a forged registry object says nothing about whose path.

Found by: Accepting-network family

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: The bad routes were a third party's, but whether IBM's edge would accept and prefer them (import filters, max-prefix) is a property of IBM's own configs that can be audited before an incident.

Imported from docs/replays/ibm-cloud-2020/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.