HomeIncident library
Incident library

INCIDENT REPLAY · 2016-08-09

Australian Census 2016 (#CensusFail): router lost its configuration on reboot; geoblock mis-scoped

On census night a series of small DDoS attacks led IBM to invoke its 'Island Australia' geoblocking plan, which one upstream had not fully implemented. One of two border routers was then rebooted and came back without its configuration, and the site was taken offline; the online census was unavailable for about 40 hours. The same geoblock also cut Australians off from the password-reset service, which was hosted offshore.

RPKI and driftRedundancy lost before failureCatalog confidence: high

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Australian Bureau of Statistics; IBM Australia (prime contractor); Nextgen and Vocus (upstream ISPs)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account high.

Sources

What the public record says

  • On census night a series of small DDoS attacks led IBM to invoke its 'Island Australia' geoblocking plan, which one upstream had not fully implemented. One of two border routers was then rebooted and came back without its configuration, and the site was taken offline; the online census was unavailable for about 40 hours. The same geoblock also cut Australians off from the password-reset service, which was hosted offshore.
  • Who: Australian Bureau of Statistics; IBM Australia (prime contractor); Nextgen and Vocus (upstream ISPs)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • Two border routers in one AS (stand-in AS 64500), CENSUS-BORDER-1 and CENSUS-BORDER-2, each with its own upstream (stand-ins for the two ISPs), iBGP and OSPF between them, and an inbound ACL on each upstream link standing in for the "Island Australia" geoblock (192.0.2.0/24 and 203.0.113.0/24 stand in for offshore space).
  • The record says the router came back without its configuration and that a hard power-off test would have shown the "reboot and configuration loading problem". It does not say which lines were missing. Modelled as CENSUS-BORDER-2's saved configuration holding its loopback, its two links and OSPF, and none of the BGP, the originating static, the host-facing Vlan100 or the geoblock: what it runs with was configured on the running router and never saved.
  • saved is what the router would boot from, passed to drift as --golden; running is what it runs, passed as --observed. The control, saved-after-write-memory, is the same pair after the running configuration was saved.
  • The geoblock's other failure, blocking Australians from a password-reset service hosted offshore, is a required-flow question and is not replayed here.
  • Cisco IOS syntax and stand-in address space throughout. A reconstruction, not any operator's configuration.
  • Nothing here claims the outage would have been prevented. The replay says what the tool reports on the reconstruction.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

the redundant border router's saved configuration against what it runs

now: exit 2

hammerhead drift, a saved configuration against the one running. Pinned exit code: 2 (findings).

Findings pinned (exactly this list)

  • CENSUS-BORDER-2 acl_rule_added
  • CENSUS-BORDER-2 acl_rule_added
  • CENSUS-BORDER-2 acl_rule_added
  • CENSUS-BORDER-2 bgp_neighbor_added
  • CENSUS-BORDER-2 bgp_neighbor_added
  • CENSUS-BORDER-2 interface_acl_changed
  • CENSUS-BORDER-2 modeled_field_changed
  • CENSUS-BORDER-2 modeled_field_changed
  • CENSUS-BORDER-2 static_route_added

Output pinned (the output contains each line)

  • "peer": "172.16.2.2"
  • "prefix": "198.18.0.0/20"
  • "rule": "deny ip 192.0.2.0 0.0.0.255 any"
  • "observed_acl": "ISLAND-AUSTRALIA"
  • "section": "prefix_lists"
  • "section": "interfaces"

Pinned in tests/gauntlet/replays-rpki-drift.yaml.

the same pair after the running configuration was saved: the control

now: exit 0

hammerhead drift, a saved configuration against the one running. Pinned exit code: 0 (none).

Pinned in tests/gauntlet/replays-rpki-drift.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: Two pieces are config-visible: running-versus-saved configuration drift on a redundant router (a drift finding) and a geoblock ACL that denies a flow the service itself needs (offshore password reset); the DDoS trigger and the upstream's missing filter are not.

Imported from docs/replays/census-2016/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.