HomeIncident library
Incident library

INCIDENT REPLAY · 2023-02-05

TIM Italy international interconnection failure

On Sunday 5 February 2023 TIM fixed-line customers from Milan to Palermo lost or had degraded internet access for several hours; Cloudflare Radar showed nationwide traffic loss on AS3269 and AS16232. An Internet Society Pulse analysis (returned by search; the page itself refused my fetch) noted that TIM depends on Sparkle alone for international connectivity, so a fault on that interconnection had no alternative path.

RedundancyOtherCatalog confidence: medium

A reconstruction, not the operator's network. Hammerhead's authors rebuilt the change from the public record as a small network of their own. No configuration here is the operator's. What follows is what the tool reported on that reconstruction; it is not a claim that the tool would have prevented the outage.

What happened

Who: Telecom Italia / TIM (AS3269, AS16232); international transit via Sparkle (AS6762)

The summary at the top is paraphrased from the sources below. Hammerhead's catalog rates its confidence in this account medium.

Sources

What the public record says

  • On Sunday 5 February 2023 TIM fixed-line customers from Milan to Palermo lost or had degraded internet access for several hours; Cloudflare Radar showed nationwide traffic loss on AS3269 and AS16232. An Internet Society Pulse analysis (returned by search; the page itself refused my fetch) noted that TIM depends on Sparkle alone for international connectivity, so a fault on that interconnection had no alternative path.
  • Who: Telecom Italia / TIM (AS3269, AS16232); international transit via Sparkle (AS6762)

What the reconstruction assumed

Decided by Hammerhead's authors to build a network that runs. None of it describes the operator's network.

Everything below is ours.

  • The record says TIM depended on Sparkle (AS 6762) alone for international connectivity. We model AS 3269 as three routers with one eBGP session to AS 6762, on TIM-BORDER-1; the carrier's router is not in the snapshot, as an upstream's never is.
  • The healthy variant adds a second carrier (a stand-in AS, 64520) on TIM-BORDER-2. TIM's real topology has far more routers and sessions, customer sessions among them; the check counts every eBGP session to a network outside the snapshot, customer or upstream, because a configuration does not say which is which. On a snapshot that included TIM's customer sessions it would not fire, and it would take a list of which peers are upstreams to ask the question properly.
  • What failed on the interconnection is not public, and nothing here models it.
  • No change is replayed: nothing in the record says a configuration change started it. The audit runs on the configuration as it stood.
  • Stand-in address space throughout. A reconstruction, not any operator's configuration, and not a claim that the outage would have been prevented.

What Hammerhead said

Each run below is pinned by Hammerhead's replay tests at commit 0ee9732: the verdict and the quoted text are checked against the tool's output, and a run that disagrees fails the test. Reason lines are copied verbatim; a pinned line can be part of a longer reason. Every preflight run uses --max-blast-radius 100, so a change is never counted as caught merely because it touched many devices of a small network.

AS 3269 as built: one session to the outside, on TIM-BORDER-1

now: fail (finding)

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.as_single_external_session. Pinned status: fail, a finding whose evidence contains the text below.

Evidence pinned (the evidence contains each line)

  • 1 row(s)
  • s.local_as=3269
  • TIM-BORDER-1
  • 172.16.3.2
  • s.remote_as=6762

First run, before the fixes this batch prompted

first: no-such-check (the check did not exist yet)

Pinned in tests/gauntlet/replays-redundancy.yaml.

the healthy variant: a second carrier on TIM-BORDER-2

now: pass

An audit of one snapshot, no change: hammerhead nqe-library --mode run, check nqe.bgp.as_single_external_session. Pinned status: pass, the check finds nothing.

First run, before the fixes this batch prompted

first: no-such-check (the check did not exist yet)

Pinned in tests/gauntlet/replays-redundancy.yaml.

Limits

  • The configurations are a reconstruction. Where the record is silent, the assumptions above fill the gap; if one is wrong, the replay is wrong in that respect.
  • A verdict describes the reconstructed network's converged state. Timing, traffic load, and anything a device does that no configuration states, such as software reacting to a health check or a firmware bug, are outside it.
  • A pass is reported as a pass. A fail says a gate fired on the reconstruction, nothing more.
  • Why the catalog counts this outage as partly in scope: The single-upstream dependency is a redundancy finding a topology and failure-simulation tool reports, but whether a config change triggered the interconnection fault is unknown.

Imported from docs/replays/tim-italy-2023/FACTS.md and the replay manifests in the Hammerhead repository at commit 0ee9732.