All posts

Fibersight: designing a network observatory

network go infrastructure

This is a design write-up, not a build log. Fibersight is a plan for a mesh of probe workers, one per cloud provider, pinging and traceroute-ing the internet from ten places on the planet. Each will report to a hub that tries to work out which submarine cable a path probably rode. This post covers the design and the two rounds of review that reshaped it. Build status is at the end.

The framing I use is a CT scan of the internet’s circulatory system, and it landed with a physicist friend I briefed.

The short version

  • Ten Go probe workers across five-plus cloud providers, reporting to a hub that will store results and serve a visualization.
  • Multi-cloud is the premise: one provider alone measures that provider’s private backbone, not the public internet.
  • Two rounds of adversarial design review, six reviewers total, found nine P0-severity problems before any code existed. All nine resolved in the design.
  • Submarine cables sell as physical capacity; traceroute sees IP routing, and most cables will never show up as themselves.
  • Budget: about $57 to $59/mo total with storage-growth margin (workers ~$32/mo, hub $13 to $15/mo). The first review’s naive $17 to $32/mo estimate was too low.

What it’s built from

Workers will run in Go, doing ICMP pings and Paris traceroutes, and sign results with Ed25519 rather than mTLS. Planned locations: Ashburn, Oregon, São Paulo, Paris, Tokyo, Singapore, Mumbai, Sydney, Cape Town, and Toronto, chosen for cable corridors, not spread. The hub is a single GCE box planned for Postgres 16, TimescaleDB, and Grafana, self-managed since Cloud SQL doesn’t support the extension. The frontend plan is Astro 5: a flat Leaflet map first, a CesiumJS globe later.

What two rounds of review found

Each round was three Claude reviewers, each briefed as a different specialist. I chose the personas and judged the findings.

The first round, a network engineer, a cloud architect, and a GIS specialist, found three P0s. Probing only inside GCP would measure Google’s private WAN, not the internet; multi-cloud became mandatory. TimescaleDB doesn’t run on Cloud SQL, so the design as written couldn’t deploy. IP geolocation as the primary signal was unreliable: 40 to 70% of backbone IPs geolocate to administrative addresses, not physical ones. The design now leads with RTT bounding, ASN affinity, and PeeringDB data, with geolocation demoted to a tiebreaker.

The second round, the cable-systems reviewer, the SRE reviewer, and a security architect, found six more P0s; the cable-systems reviewer’s findings reshaped the project. Consortium ownership doesn’t show up in traceroutes; the carrier using the cable does. MAREA maps to a Telxius ASN, but Microsoft’s traffic on it shows AS8075 and Meta’s shows AS32934. EllaLink’s own ASN barely appears, since Google buys capacity on it and shows its own ASN end to end. The signal holds for roughly 30% of cables, where the operator is also a carrier. Fiber pairs are invisible too: FASTER has eight to sixteen pairs, owned separately by Google, KDDI, SingTel, and China Telecom, indistinguishable in a traceroute.

The SRE and security findings were more ordinary: a replay attack on signed headers, an exposed enrollment token, and a hub that could OOM under a buffer-drain storm. The design now calls for a nonce and replay cache, out-of-band token delivery, and rate-limited draining.

The cable-systems reviewer’s summary stuck with me: “this design observes IP routing topology, not submarine cable topology. These are not the same thing and the gap is not bridgeable with the proposed signals alone.” Carrier-operated cables get a 70 to 80% identification rate. Hyperscaler-lit cables score 0.4 confidence even when the scoring model computes 0.9. That calls for a scoring system that says so, not a smarter model.

What was hard

A fused score from weak signals never exceeds the confidence of its strongest one, capping false certainty from stacked weak signals. Path elimination is the strongest signal, though it needs the corridor’s full cable inventory. On a thin corridor like South America to Africa, only SACS and SAIL exist. Ruling one out gets 0.85 to 0.95 confidence, assuming traffic doesn’t route the long way around. Probe placement at chokepoints matters more than probe count.

Where it stands

The design survived two rounds of adversarial review before any implementation existed, the order I’d recommend for anything with this many moving parts. As of September 2026, nothing is deployed. No probe is live and no cable identification has been measured on real traffic. The design is done. The code is not started.

The review’s own estimate: about fifty cents in API calls, catching a design that would otherwise have shipped confidently wrong on 40% of its core feature.

Next

The design is waiting on real traffic: until probes are live, the confidence scores and the budget stay estimates. What’s left, mostly where to place a few more probes, is logistics, not architecture.

Back to all posts