blog · · engineering

Two 'today' pages: debugging a bug I couldn't inspect

The journal on my phone flickered between two versions of the same day. I fixed the relay. Still flickering. I fixed a reload race in the frontend. Still flickering. I went to attach a debugger and the webview wouldn't let me. So I rendered the debug output onto the phone screen itself, and the numbers said the bug was in a layer I hadn't looked at once. This is the debugging story, the two wrong turns, the improvised instrument, and the split-brain in the op log it finally exposed.

A
11 min read

The journal on my phone was flickering.

Not slow, not blank. Flickering. It would show today’s page, and half a second later it would flip to a different version of today with different blocks, then flip back. Same date at the top, two different bodies underneath, trading places every couple of seconds. The desktop, on the same wifi, showed one stable page and never blinked.

I want to walk through how I chased this, because I spent a good while fixing things that weren’t the bug, and the moment it broke open was an instrument I built on the phone screen when the normal debugger refused to attach. The wrong turns are the useful part. Anyone can tell you the answer after they know it.

the setup, in one paragraph

outl is a local-first outliner and it syncs peer to peer, no server holding your notes. Every change is an operation appended to a log, one log file per device. The outline you see is a projection of that log, replayed through a CRDT so every device converges to the same tree. Two devices reconcile by exchanging the operations the other is missing, over iroh, which gives you a QUIC connection to another device addressed by its public key and does the hard part of punching through NAT. The phone and the desktop both had the workspace, they were talking, and the desktop was fine. So whatever was wrong was specific to how the phone turned the log into pixels.

That framing is what sent me down the first wrong path.

wrong turn one, it must be the network

The phone is the flaky one. It backgrounds, it moves between wifi and cellular, its sockets get suspended by iOS. So my first instinct was that sync was delivering operations in a bad order or a bad state, and the flicker was the phone catching up to a moving target.

The logs supported it, or seemed to. The desktop was holding a stale address for the phone, connects were stalling, the relay we leaned on was slow. All real problems, and I fixed them: bounded the connect with a timeout and a fallback, made the stored peer address self-heal from the live socket, and moved us off the shared public relay onto one we run.

That last one is worth a beat, because the relay is the only piece of this that isn’t peer to peer. Its whole job is to help two devices find each other and, when they can’t punch a direct hole through NAT, forward their already-encrypted bytes. It never has a key, so it can’t read your notes, but it can see metadata: that two specific devices are talking, and when. We’d been riding iroh’s shared public relays, which is exactly right for a demo and wrong for the layer that decides whether your notes reach your phone. No SLA I control, and no separation from everyone else’s traffic. iroh runs a hosting service for precisely this, so we’re on a dedicated relay of our own now, the default, with a one-line config for anyone who’d rather point at theirs. I’ll be honest about what that is and isn’t: it’s still iroh’s infrastructure carrying the bytes, a dedicated endpoint instead of a shared pool, and moving the relay onto a box that’s entirely ours is the next step, not this one. But in a product whose pitch is “no provider in the loop,” the single non-P2P hop should get as close to ours as it can, and a dedicated relay is a lot closer than a public pool. Good fixes, all of them. Shipped them.

Still flickering.

That’s the tell I should have read faster. When you fix a plausible cause and the symptom doesn’t move at all, not better, not worse, you didn’t fix the bug. You fixed a different bug that happened to be nearby. Connectivity was genuinely broken and worth repairing, but it was never why the page oscillated.

wrong turn two, it must be a reload race

Second theory, and a better one. The mobile frontend reloads the page from several triggers: a periodic poll, the app returning to the foreground, a signal that peer operations landed, the initial mount. Each reload does an async pull and then renders what it read. Nothing ordered them. So a slow reload that started earlier could finish later and slam an older render on top of a newer one. Classic out-of-order write.

That’s a real race and I’m glad I found it. I added a generation counter: every reload captures a number when it starts and only applies its result if it’s still the latest when it finishes. A stale reload drops its read instead of rendering it.

The frontend hot-reloaded the change onto the phone. Still flickering.

I almost talked myself into “the fix didn’t fully take”. The reload guard lives in code that registers on mount, and hot reload doesn’t re-run mount, so maybe the old handler was still live. Plausible, and it sent me in circles for a bit. But underneath the plausible excuse was the same tell from wrong turn one: I’d fixed a real thing and the symptom hadn’t flinched.

I needed to stop theorizing and see what the render loop was actually doing.

the tooling wall

The clean way to see a webview’s internals is Safari’s Web Inspector. Connect the phone, open the Develop menu, pick the app’s webview, read the console. outl’s mobile UI is a webview, so this should have just worked.

It didn’t. The device showed up in Safari. The webview didn’t. “No inspectable contents.” Web Inspector was on in the phone’s settings, the app was in the foreground, and still nothing to inspect.

The reason is a change in iOS 16.4: a WKWebView is not inspectable by default anymore, the app has to explicitly opt in by setting the webview inspectable, and this build didn’t. Fixing that means touching native code and rebuilding and re-signing, which is exactly the kind of yak I didn’t want to shave in the middle of a hunt. And even a console.log needs somewhere to land, and I’d just lost the place it lands.

So I couldn’t attach a debugger, couldn’t open a console, couldn’t stream logs off the device. The one thing I could still do was draw pixels, because drawing pixels is the app’s whole job.

put the debugger on the screen

If the only surface I have is the screen, then the screen is where the debug output goes.

I added a small fixed panel at the bottom of the app, a few lines of monospace over a dark strip, showing the last several render events. Each line was one call to apply a view: a sequence number, the generation counter from my reload guard, the number of blocks in the view, and the first eighteen characters of the first block. Thirteen lines of code, no build system, hot-reloaded onto the phone in seconds.

Then I just watched the strip while the page flickered, and read it off the screen.

#10 gen13 b6 ```call:calc-salar
#9  gen12 b4 sdfsdfdsf
#8  gen11 b6 ```call:calc-salar
#7  gen9  b4 sdfsdfdsf
#6  gen6  b6 ```call:calc-salar
#5  gen4  b4 sdfsdfdsf

There it was, and it said the opposite of what I’d been assuming.

reading the tea leaves

Two states, alternating. One had four blocks and started with sdfsdfdsf. The other had six blocks and started with the call:calc-salary code fence. The screen was flipping between those two, over and over.

Now the important column: gen. It goes up every single time. 4, 6, 9, 11, 12, 13. Never repeats, never goes backwards.

That killed the reload-race theory dead. My generation guard was doing its job perfectly. Every applied render was the latest one, no stale reload was sneaking through. If the race were the cause, I’d see an old generation land on top of a new one. I never did. Each render was fresh, current, the newest read available.

Which means the reads themselves were disagreeing. Reload at generation 12 asked the backend for today’s page and got the four-block sdfsdfdsf version. Reload at generation 13, a moment later, asked the same backend for the same day and got the six-block version. The frontend was faithfully rendering the latest answer every time. The backend was just giving a different answer every time.

The bug was not in the layer I’d spent two sessions in. It was underneath, in how the op log projected “today” into a page at all. I’d been debugging the messenger.

the split-brain

Once I knew to look at the projection, it took one query. I listed the pages in the workspace and filtered for that day’s slug, expecting one. There were two.

7PJ8A59K0JZ2BWHTAK6FEBAJSF   2026-07-10   journal
01KX5JE12070P508T0NA08ZGEE   2026-07-10   journal

Two separate page roots, same date, different node ids, each holding different blocks. The day had split in two, and every time the phone opened “today” it resolved the slug to whichever root came up first, which wasn’t deterministic across reloads. State A, state B, state A. The flicker was the projection flipping a coin between two real pages that should have been one.

And the ids told the origin story. 7PJ8A59K... is what you get when you derive a page id deterministically from its slug, a hash of 2026-07-10 folded into a ULID. Both devices, and both code paths, compute the same id for the same day, on purpose, so a page created independently on two devices lands on the same node and merges. 01KX5JE1... is a plain time-based ULID. Random. Minted fresh at creation.

So something created today’s journal the deterministic way, and something else created it the random way, and now there were two.

why it happened

The random-id path was reconcile. When a markdown file lands on a device without its id sidecar, an external editor touched it, a peer shipped only the .md and not the sidecar yet, a crash interrupted the write, outl reconciles the file back into the op log. And in that one path, when there was no sidecar to read the root’s id from, it minted a fresh id with NodeId::new() instead of deriving it from the filename.

Everywhere else in the codebase, a page root’s id comes from its slug. This one path, on this one condition, didn’t. So a sidecar-less journals/2026-07-10.md reconciled into a brand-new root that had nothing to do with the deterministic one the app had already made. Two roots, same slug, and a projection that couldn’t decide.

It only fired under a specific alignment, a file present without its sidecar on a device that also had the real journal, which is exactly why it survived every test on a clean single-device workspace and only showed up once two devices were shipping files at each other.

the fix

Two parts. Stop making new duplicates, and heal the ones already there.

Stopping new ones is small. Reconcile now derives the root id from the file’s slug, the same deterministic derivation everything else uses, so a sidecar-less file lands on the canonical node instead of a fresh one. I pulled the derivation into a single function that every path calls, so there’s one place that decides what a slug’s id is and it can’t drift again.

Healing the existing ones is a boot-time repair. On startup, group the page roots by slug, and for any slug with more than one root, pick the canonical one, move every child of the duplicates under it, and send the emptied duplicates to the trash. Every step is a normal operation through the op log, so it converges to every device the same way any edit does. It’s idempotent, a clean workspace is a no-op, and it runs on the boot path of every client. I ran it against the live workspace and watched two roots become one with nothing lost.

what the debug strip actually taught me

The fix is satisfying but it’s not the part I’ll carry. The part I’ll carry is that I spent two rounds fixing real bugs that weren’t the bug, because each of my theories was plausible and I was reasoning instead of measuring. The symptom sat perfectly still through both fixes and I kept explaining away the stillness.

What broke it open wasn’t a cleverer theory. It was thirteen lines that made the invisible visible, on the one surface I had left. The gen column going up was the whole answer: it ruled out my entire current theory in a single glance and pointed one layer down. I didn’t need a debugger. I needed to see the numbers the code was actually producing, and I’d been unable to see them because I was attached to seeing them the usual way.

When the normal instrument won’t attach, don’t stop measuring. Move the instrument to whatever surface you’ve still got. A phone can’t open a console but it can render a rectangle, and a rectangle with the right six numbers in it is a debugger.

The fixes shipped in PR #150.