iroh transport internals

Contributor-facing internals of outl-sync-iroh, the crate behind the default iroh transport. Two things live here, both extracted from crates/outl-sync-iroh/CLAUDE.md so the crate file can stay focused on invariants: the pinned iroh 1.0.0 API surface (the calls that differ from every tutorial you will find) and the named test catalog (regression + chaos).

The architecture, the load-bearing invariants (“one endpoint per identity”, the append-serialization rule, workspace identity), and the module layout stay in crates/outl-sync-iroh/CLAUDE.md. The user-facing sync story is sync.md; the relay threat model is relay.md.

iroh 1.0.0 API notes (load-bearing)

The 1.0.0 surface differs from older tutorials — pin these:

• iroh::SecretKey, not iroh::key::SecretKey. SecretKey::from_bytes(&[u8;32]), to_bytes() -> [u8;32], public() -> PublicKey, generate(&mut rand::rng()).
• EndpointId = PublicKey is the node identifier type. iroh::PublicKey is the concrete struct; parse from string with .parse().
• Endpoint::builder(presets::N0) — the builder takes a discovery preset arg. presets lives at iroh::endpoint::presets.
• ProtocolHandler::accept(&self, conn: Connection) -> Result<(), AcceptError> is a native async fn and receives an already-accepted Connection (not Connecting, no manually-boxed future).
• endpoint.connect(id, b"alpn") takes the ALPN bytes directly.
• SendStream::finish() is sync and returns Result; write_all/read_to_end are async.
• Gossip: gossip.subscribe(topic, peers).await? returns a topic handle; .split() yields (GossipSender, GossipReceiver). Events are iroh_gossip::api::Event::Received(message) with message.content and message.delivered_from. StreamExt comes from n0_future.
• GossipSender::broadcast(&self, msg: bytes::Bytes) -> Result<(), ApiError> takes bytes::Bytes and &self (not &mut), so the sender can live in a drain task that announce_local_ops feeds via an mpsc channel.
• No NodeTicket type ships in iroh 1.0.0 / iroh-base 1.0.0. The pairing “ticket” is a base64 of serde_json(EndpointAddr). EndpointAddr is Serialize/Deserialize with public id: EndpointId and addrs: BTreeSet<TransportAddr>; endpoint.addr() returns the current one, and endpoint.connect(addr, alpn) takes impl Into<EndpointAddr>, so the decoded value feeds straight back into connect.
• Accept loop (host side): endpoint.accept().await -> Option<Incoming>, then incoming.accept()? -> Accepting, then .await -> Connection. Connection::accept_bi() / open_bi() / close(VarInt, &[u8]) as usual.

Module layout

Every split below was forced by the file-size guard, but each one landed on a seam that was already there.

engine.rs re-exports delta_sync, SyncProtocolHandler and the coordination types, so crate::engine::delta_sync and crate::engine::AppendLock keep resolving for engine_catchup, engine_gossip, engine_pairing and test_support.

Regression suite (Pilar 2)

Every bug hand-found during the sync saga has a NAMED, permanent test — the name IS the bug, so a failure is self-explanatory. Pure guards live in #[cfg(test)] next to the code. Over-the-wire (real QUIC, loopback) guards live in four suites: tests/regression.rs (one row per saga bug), plus three organised by question instead — tests/revocation.rs (may this dialer be served?), tests/membership_trust.rs (how does a node id get into peers.json?) and tests/oplog_guards.rs (what may an authorized peer’s bytes do to ops/?). The three question-suites keep deny cases outnumbering allow cases on purpose, and their allows are load-bearing: a check that refuses everything passes every deny test ever written. Shared seed/read/wait helpers stay in tests/common/mod.rs (read-only); saga-specific helpers live inside regression.rs.

Names map 1:1 to the saga checklist; do NOT delete one without deleting the bug it guards.

Chaos/concurrency tests (Pilar 3)

The regression suite pins one named bug per row; the chaos battery (tests/chaos.rs) instead hammers the same wire code (real delta_sync + SyncProtocolHandler via test_support) with STRESS loads (N writers × M ops). Every test runs over real QUIC on loopback under #[tokio::test(flavor = "multi_thread")].

Determinism (a flaky chaos test is worse than none): randomness is a seeded xorshift (chaos_helpers::Rng); the only true nondeterminism is network timing, so every wait uses common::STEP_TIMEOUT + wait_until, never a fixed sleep. Sizes are bounded (≤ 64 ops, ≤ 8 tasks).

Why raw bytes, not all_ops: JsonlStorage::reload recovers glued …}}}{… lines on read, so all_ops would MASK an append-lock failure. chaos_helpers::assert_every_line_is_one_json_value reads the ops-<actor>.jsonl bytes directly — the only thing that reveals whether the lock held.

Helpers live in tests/chaos_helpers/mod.rs, not tests/common/mod.rs (clippy duplicate_mod allows one common loader per test binary).

STOPGAP: IPv4-only bind (iroh 1.0.0 multipath workaround)

All four endpoints bind IPv4-only through bind::n0_builder_ipv4_only — run_iroh, bind_pairing_endpoint, probe_peers, bind_sync_endpoint. The bind module owns the bug, the fix and the revert condition; dial and accept must both go through it, because dropping IPv6 on one side only lets the other advertise a dead path.

It narrows the bug, it does not close it. Multipath opens paths to all of a peer’s candidate addrs at once and one unreachable addr stalls the whole connect/accept (MultipathNotNegotiated, ~30s) rather than converging on a working path. Binding IPv4-only removes the usual offender (a global IPv6 addr that is “No route to host”) but an unreachable IPv4 addr stalls it identically — a VM bridge or VPN utun addr in our own ticket, or a peer’s stale DHCP lease in theirs. Signature: sendmsg error: … HostUnreachable / Host is down toward one addr, then a connect timeout with the relay up the whole time.

Configurable relay (default: outl’s own)

n0_builder_ipv4_only(relay_url: Option<&str>) picks the relay on top of the IPv4-only STOPGAP. Default is outl’s own dedicated relay, DEFAULT_RELAY_URL (use1-1.relay.avelino.outl.iroh.link, via RelayMode::custom) — the n0 public relay proved slow/unreachable on some networks. A non-empty [sync] relay_url overrides it; a parse error falls back to presets::N0 with a warning. Pairing / status / test pass None, which is not “the n0 preset” — None resolves to DEFAULT_RELAY_URL too, so every endpoint rides outl’s relay by default. Only the long-lived sync endpoint threads the configured one (run_iroh ← IrohSyncTransport::new ← outl_config::load().sync.relay_url()), so only a deployment that overrides [sync] relay_url sees a split. See docs/relay.md / docs/config.md.

Revert condition: delete the bind module once iroh > 1.0.0 ships the multipath fallback fix, and let every call site go back to the plain dual-stack Endpoint::builder(presets::N0) builder (details in the module docs).

One endpoint per identity, elected not assigned

The detail behind crates/outl-sync-iroh/CLAUDE.md → “One endpoint per identity, elected not assigned”, which keeps the rule; this is the mechanism, the failure modes, and what the lease refuses.

A device binds at most ONE iroh endpoint at a time, and which process gets it is decided by a lease, not by what kind of client it is.

Why the route is single: a second endpoint registering the same secret key replaces the active client in the relay’s DashMap<EndpointId, ClientState>. All inbound datagrams then route to the newcomer and the original silently stops receiving (endpoint.rs::same_endpoint_id_relay asserts this). The demoted endpoint’s outbound catch-up stalls too for any relay-only peer, because that peer’s QUIC return traffic is addressed to the node id and lands on whoever is ACTIVE. So a second endpoint breaks the first’s sync in both directions. This is not the “stable, benign hijack” an earlier version of this document claimed; believing it is what let outl mcp serve silently kill the desktop’s sync to an off-LAN iPhone. If the newcomer doesn’t accept SYNC_ALPN at all, the dialer additionally gets quinn CONNECTION_REFUSED — the older “connection refused, nothing syncs” bug (a transient status-probe or the GUI’s old pairing endpoint stealing the route).

The lease (lease::EndpointLease). An advisory flock on endpoint.lock, a sibling of the identity key, so the arbitration scope follows the node id automatically (desktop / TUI / CLI / MCP share ~/.outl/; mobile’s sandbox identity never contends). build_transport (device.rs) is the one place that takes it, so no client has to remember to. Released by the kernel when the holder exits: no TTL, no stale lease, no daemon.

The endpoint thread owns the lease, and the two ways of getting that wrong are opposite. start() moves it into the outl-iroh-sync thread, where it is bound first and therefore dropped last, so the claim ends exactly when run_iroh returns. Leave it on the struct instead and a failed .bind() kills the thread while the transport keeps the claim, locking every other process on the device out of an endpoint forever — issue #220 again, this time with a padlock. Drop it any earlier and you reopen the reverse. shutdown() only sends a oneshot. A client that drops the transport right after (the desktop, on a workspace swap) would free the lease while run_iroh is still closing the endpoint, and a second endpoint could bind onto the same node id in that window. A transport that was built but never started still holds it, which is what outl sync needs when it exits early.

Two failure modes at acquire time, deliberately opposite (lease.rs). The lease file failing to open (permission, read-only mount) is fail-closed: there is no arbiter and no way to know whether someone is already bound, so granting would grant to everyone. The file opening but refusing to lock (ENOLCK, a mount with no locking) is fail-open with a warning. The file is ours, only the locking is missing, and refusing everyone leaves the device with no endpoint at all, which is the failure the lease exists to remove.

A refusal says which one it is. try_acquire returns Result<EndpointLease, LeaseDenied>, and LeaseDenied is HeldByAnotherProcess or LeaseFileUnusable { path, error }; TransportOutcome::EndpointBusy and PeerProbe::EndpointBusy carry it through to the client. The degradation is identical either way (file transport, stay off the wire), so a caller that only degrades ignores the payload. Every caller that words this for a human must read it. “Another outl process holds the endpoint” sends a user whose ~/.outl is read-only hunting for an outl mcp serve that is not running, and no process exiting will ever free a lease nobody took.

Why a lease and not a policy. The rule used to be “only the GUI binds; the MCP server and the CLI are passive writers”. That kept two endpoints apart, but it assumed a GUI exists. On a headless machine (an agent driving outl mcp serve) nobody bound an endpoint, so the device’s ops never left and no peer’s ops ever arrived — silently, with outl peer status on the other device just showing “offline” (issue #220). The constraint was never “only the GUI”; it is “one live endpoint per identity”, and that is a question about who got here first, which only a lock can answer. Losing the election is a working state, not a failure: the loser runs outl_actions::FileSyncTransport and converges through the shared ops/ dir, which the holder pushes out on its MAINTENANCE_RESYNC pass.

Known limitation: the lease is per device, so it is also per workspace holder. The lock is a sibling of identity.key, not of the workspace, because the thing being arbitrated is the node id and there is exactly one of those per device. A process holding the endpoint for workspace A therefore keeps a process on workspace B off the wire. B’s ops only leave the machine when a process that does hold the endpoint opens B — the shared ops/ fallback converges B across local processes, not across devices. Scoping the lease per workspace would not fix this; it would let two endpoints bind the same node id, which is the exact failure this section exists to prevent. The real fix is one endpoint multiplexing every open workspace (the sync protocol already carries WorkspaceId per request), and that is a redesign of engine::run_iroh, not a change to the lease. Until then, a user running two workspaces at once P2P-syncs the one whose process got there first.

Pinned by tests/endpoint_lease.rs (one_process_binds_the_device_endpoint_and_the_next_one_is_told_to_stay_off_the_wire) plus the unit tests in lease.rs.

Non-sync endpoints are the sharper case, and they take the lease too. An endpoint that does NOT serve SYNC_ALPN is worse than a competing one: a dialer routed to it gets CONNECTION_REFUSED instead of a working peer. The status probe (status::probe_peers) is the only one left, and it now asks for the lease like everything else, returning PeerProbe::EndpointBusy instead of binding when it loses. It used to be exempt on the grounds that “the CLI has no running transport to conflict with”. That stopped being true the moment outl mcp serve could hold the endpoint, and outl peer status is precisely the command a user runs to diagnose sync, so it must not be the thing that breaks it.

Three call sites, three rules:

1. Sync endpoint (engine::run_iroh) — the one allowed long-lived endpoint. Router accepts SYNC_ALPN + gossip ALPN + PAIRING_ALPN (rule 3) + SNAPSHOT_ALPN (see “Phase-2 blob transfer”), all advertised in its .alpns(). All catch-up / boot / gossip / pairing dials go out through this endpoint (the one bound in run_iroh); no helper spins up a second.
2. Status (status::probe_peers) — binds a transient endpoint, and only if it wins the lease. It returns PeerProbe::EndpointBusy rather than binding when it loses, so it can never demote the transport it was run to inspect. A client that has its own running transport reads peer_health() instead and never calls this at all.
3. Pairing — the split is about holding an endpoint, not about being a GUI:
   • Transport running → IrohSyncTransport::pair_host / pair_join reuse the live sync endpoint. The host (accept) side is the PAIRING_ALPN router handler (engine_pairing::PairingProtocolHandler), armed by pair_host via a shared PairingHub; the join side dials out on the same endpoint. After a successful pair the new peer is persisted to peers.json and an immediate delta_sync is fired against it (engine_pairing::drain_pair_completions) — no app restart, no 8s catch-up wait.
   • No transport of our own (the ephemeral CLI, or a GUI that lost the lease) → pairing::host_pairing / join_pairing bind a one-shot endpoint and close it (endpoint.close().await) before returning. This is the one sanctioned exception to the lease. It does take the route from the holder for the length of the handshake, and it is worth it because pairing is rare, explicit and short, while the alternative is a user who cannot add a device. Nothing else may bind around the lease.

Phase-2 blob transfer (snapshot + asset)

Two ALPNs ship binary blobs that are NOT ops. Both mount on the one sync endpoint’s router, hold no workspace lock, never write the op log, and are best-effort (failure = logged no-op).

Both authorize the dialer through authz before reading a byte off disk, the same fail-closed peers.json verdict SYNC_ALPN uses. They did not, for two releases, and that is the whole of what outl peer remove failed to revoke: a removed device kept a complete read of the graph and of every uploaded file. These two carry no request body, so they have no workspace_id line to validate — and need none: peers.json is per workspace, so a device paired into a different one is refused by node id alone.

Snapshot — SNAPSHOT_ALPN outl-snapshot/1, engine_snapshot.rs. A freshly-paired device pulls a peer’s snap-<actor>.bin and boots from settled state, not the full op log (pull_snapshot_from_peer). Responder SnapshotProtocolHandler authorizes, then sends one length-prefixed frame (empty when absent); the puller writes snap-<peer-actor>.bin under .outl/snapshots/ and fires peer_ready_tx.

Asset — ASSET_ALPN outl-asset/1, engine_assets.rs. Uploaded files live at <root>/assets/<hash>.<ext> (content-addressed by SHA-256); their bytes NEVER enter the op log (outl_actions::asset). Since a device holds N assets, the protocol negotiates a manifest: responder AssetProtocolHandler authorizes, then sends its assets/ basenames (protocol::encode_asset_manifest). The refusal has to land before the manifest — the names are content hashes and their count is the size of the user’s upload history, so shipping it and refusing the blobs still leaks. It also has to be re-asked per request, not once per connection: this is the only protocol here that serves an unbounded number of payloads over one connection, at the initiator’s pace, so a connect-time-only check left a peer approved a moment before outl peer remove draining assets/ for as long as it kept asking. SYNC_ALPN is per exchange on a pooled connection and has never had that shape; the next frame is this protocol’s next exchange. The initiator pulls each file it lacks as a blob frame (atomic tmp+rename). is_safe_asset_name (both sides, anti-traversal) blocks any non-basename; the initiator re-hashes each file (outl_md::asset::hash_bytes) and drops a name mismatch. Fires after the post-pair snapshot pull and every catch-up delta_sync (run_catch_up’s pull_assets).