The status dot that broke sync

The sync dot on the desktop is the smallest thing in the app. Seven pixels. Green when a peer is reachable over iroh, orange when none are, dim while the first check is still out. Click it, pairing opens. That’s the entire feature.

It shipped alongside the work that finally closed peer-to-peer sync across every client, which felt right at the time. Sync worked. The dot said so. Ship both.

Then the dot went orange while the peer sat there, paired, same wifi, obviously fine.

The dot wasn’t lying, and that’s the part that took me a minute to accept. Sync really was broken. The peer really was unreachable. The peer was unreachable because something asked whether it was reachable. The instrument was the outage.

the obvious design

The thing I’d built isn’t clever and it isn’t careless. It’s what you’d write.

You want to know whether a peer answers. iroh hands you an endpoint, you dial a peer by its public key, you time the round trip. So: bind an endpoint, dial each known peer, record who answered and how fast, tear it down. Transient, cheap, runs on a timer, gets out of the way when it’s done.

The endpoint needs an identity to dial with, and the device already has one, an ed25519 keypair at ~/.outl/identity.key, one node id per machine. You hand the probe that key. Of course you do. It’s this device asking. A different key would mean the peer sees a stranger knocking.

Thirty-ish lines. Read-only. Touches none of outl’s data. The tests never caught it, because the failure needs a second device and a live transport to exist at all.

the symptom

Two things showed up together.

On the dialing side, quinn came back with CONNECTION_REFUSED. Not a timeout, not a NAT failure, not silence. Refused. Something answered the door and turned the connection away, which is a far stranger thing to see than nothing at all. A timeout means the packet went nowhere. A refusal means it arrived, at something, and that something said no.

On the relay, a line I’d never had reason to read:

Another endpoint connected with the same endpoint id

That log line is the whole bug stated plainly, and I still had to go read iroh’s source to understand what it was telling me.

the root cause

A relay exists because two devices behind NAT can’t find each other. Each device registers a home relay on startup, and the relay helps them hole-punch a direct path, or forwards their already-encrypted bytes when hole punching fails. It never holds a key, so it never reads your notes. That’s the deal, and it’s the one non-P2P hop in outl.

For the relay to forward anything it has to know where a given node lives. So it keeps a table: node id to the endpoint currently holding that route. In the relay server it’s a DashMap<EndpointId, ClientState>.

One entry per node id. Not per connection. Per identity.

So when the probe bound the device’s SecretKey and registered, the relay did exactly what its data structure says it does. It looked up that node id, found the live sync endpoint sitting there, and replaced it. The probe held the route now. Every inbound datagram for this device went to the probe.

And the probe didn’t serve SYNC_ALPN. It had one job, dial out and measure, so it never advertised the sync protocol. A peer dialing this device got routed, correctly, to an endpoint with no idea what it was being asked for. Refused.

The sync endpoint sat there healthy and deaf. Still dialing out fine. Just no longer receiving anything, silently, with nothing in its own logs to say why. iroh’s endpoint.rs::same_endpoint_id_relay asserts this behavior directly. It isn’t a bug in iroh. It’s the documented consequence of a route keyed by identity.

Read it back in order and it’s almost funny. The status check borrows the device’s identity to ask a question. To the relay, borrowing the identity is being the device. The relay hands over the mail. The probe can’t read mail. Sync stops. The probe reports orange.

I built a thing that asked “is sync working?” and the asking is what made the answer no.

the nuance that changed the rule

The easy lesson is “never bind the device identity twice.” I wrote that rule down. Then I went to verify it against iroh 1.0.0’s source, and the real rule turned out to be more useful.

Two endpoints sharing a node id, both serving SYNC_ALPN, coexist fine. The relay still keeps one route and the last to register still holds it. The loser loses inbound. But it keeps dialing out, so catch-up still works, it pushes and pulls on its own dials. And whichever one holds the route serves inbound from the same ops-*.jsonl on disk, so it answers correctly on behalf of both.

Better: it’s a stable hijack, not a flap. The loser never steals the route back, because the holder’s socket keeps answering pings. When the holder leaves, the route returns. No thrash, no split.

This is load-bearing, not trivia. It’s why the GUI and the MCP server can both bind the device identity at once, so you can have an agent writing to your graph through MCP while the desktop app is open, and neither breaks the other. That’s a capability I’d have thrown away if I’d shipped the easy rule.

The crime was never sharing the identity. The crime is binding the identity without serving the ALPN. An endpoint that can take the route has to be able to do the job the route implies. Take the mail, read the mail.

the fix: stop asking

Once the rule is stated that way, the fix stops being “make the probe safe” and becomes “why is there a probe at all?”

The transport was already doing this work. It dials every known peer at boot. It dials on every catch-up tick. It dials whenever gossip says a peer has new ops. Every one of those dials already knew whether it reached the peer and how long it took. That information existed, got used for one branch, and got thrown away.

So health.rs isn’t a monitoring system. It’s a map the existing dials write to on their way past:

pub(crate) fn record_success(&self, peer: EndpointId, started: Instant) {
    let rtt = started.elapsed().as_millis() as u64;
    // reachable: true, last_rtt_ms: Some(rtt)
}

pub(crate) fn record_failure(&self, peer: EndpointId) {
    // reachable: false, but keep the prior rtt so the UI can still
    // say "last seen 40ms, now offline"
}

The GUI reads a snapshot of that map. Zero new endpoints, zero new dials. The dot is a byproduct of work that was happening anyway, and it’s more truthful than the probe was, because it reports real sync traffic instead of a synthetic ping that proved nothing about whether sync itself would land.

A peer the transport hasn’t dialed yet this session is simply absent from the map, and the UI renders that honestly: dim, unknown. Not green, not orange. I don’t know yet.

The old probe_peers still exists, but it’s CLI-only now, for outl peer status, where there’s no running transport and therefore no route to steal. The GUI is forbidden from calling it, and that’s written into the crate as an invariant, because a rule that lives only in my head is a rule I violate in six months.

what I’d take from this

Observability isn’t free, and the bill isn’t always CPU.

Every instinct I have says a status check is the safe kind of code. Read-only. Mutates nothing. Can’t hurt. But “read-only” is a claim about your data structures, and the probe wasn’t touching outl’s data at all. It was touching a routing table in someone else’s process, in a system where identity is the route. There is no passive observer at that layer. Showing up is an action.

And it generalizes past p2p. When you want to know whether something is working, check whether your system already knows. Mine did. It was dialing peers constantly and discarding the answer, and I went and built a second, more fragile way to ask the same question. The reliable instrument was the work itself. I wasn’t listening to it.

The dot is green now. It’s green because a real peer answered a real sync dial, which is the only evidence that ever mattered.

The reachability logic is in the outl repository, at crates/outl-sync-iroh/src/health.rs.