blog · · engineering

OUTL-4 was a bucket, not a crash

OUTL-4 held two unrelated crashes: a Windows WebView2 failure at startup, and a TUI panic reading `index out of bounds: the len is 0 but the index is 13`. They shared one issue id because every frame in an outl panic arrives as `<unknown>`, so the default grouping had nothing to tell them apart, and the frames were unnamed because release CI uploaded the CLI and TUI debug symbols to a Sentry project called `outl-cli` while the binary reports as `tui@<version>` into `outl`. The fix is a 130-line module that fingerprints a panic by its already-redacted exception value, plus one word in a workflow. The cost is the mirror image: a panic message that embeds a changing index is now a fresh issue every time it fires.

A
10 min read

OUTL-4 is a Windows crash at startup, from desktop@0.13.0-beta.265.

OUTL-4 is also a panic inside the terminal client, from tui@0.13.0-beta.273, reading index out of bounds: the len is 0 but the index is 13.

Those are not two readings of one crash. They are two crashes, in two binaries, on two operating systems, filed under one issue id. On 31 October I published a post titled both ways outl dies on open happen before any of my code runs, which opens by saying what OUTL-4 is. It was describing one of the two tenants and calling it the building.

an issue id is a group, and this group had no key

A Sentry issue is not a crash. It is a set of events that share a fingerprint, and when the client sends no fingerprint the server derives one from the event. The stack trace is the obvious input, and for an outl panic the stack trace is a column of bare addresses: the very first production event the reporter ever delivered, OUTL-1, arrived with every frame as <unknown>, and the outl binary kept arriving that way after that post fixed the desktop’s half.

I cannot tell you from the outside which input the server hashed, and it does not matter, because every candidate reads the same bytes. The frames are identical across unrelated panics, since they are not frames, they are addresses from different builds with nothing to resolve them against. The exception type is identical too: every event the panic integration produces is the same type on every one of them. So a stack hash, a type hash, or any combination of the two puts a Windows setup failure and an out-of-bounds index in the terminal in the same bucket.

A bucket with no key still presents itself as a bug. It has one title, one first-seen, one last-seen, and an event count. The title and the first event described the WebView2 failure, so that is what I read, and the count I attributed to it was the sum of two unrelated populations. Nothing about the dashboard says “these events have nothing in common”. There is no such view, because from the server’s side they had everything in common: an identical fingerprint.

the symbols existed, and went to a project the events never reach

The frames are unnamed on purpose, up to a point. [profile.release] in the root Cargo.toml deliberately does not strip:

# `strip = "symbols"` used to live here and is gone on purpose (RFC 0377 §5).
# A crash report from a stripped binary arrives as a list of bare addresses
# instead of function names, which is the same as no crash report at all.
debug = "line-tables-only"
split-debuginfo = "packed"

Release CI uploads the debug files with sentry-cli debug-files upload, and only then strips the binary it publishes. The archive a user downloads, and the Homebrew bottle, which is the same binary staged a second time in Homebrew’s Cellar layout, have had their debug information taken out by then. Every name in a crash report from those builds comes from the upload. Sentry matches an upload to an event by debug id, and crates/outl-telemetry/CLAUDE.md already said as much when it explained why the scrubber redacts the image path and keeps the id.

The upload step in .github/workflows/release.yml for the job that builds outl named a project:

env:
  SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
  SENTRY_ORG: outl
  SENTRY_PROJECT: outl

The last of those lines used to read outl-cli. The binary’s own DSN is OUTL_SENTRY_DSN_CLI, and its events land in the project named outl, which is where OUTL-prefixed issue ids come from. Symbols in one project, events in another, both operations succeeding, the job green.

The reason it is not even a CLI project’s worth of events is in crates/outl-cli/src/main.rs:

let _telemetry = outl_telemetry::init(
    if is_tui { "tui" } else { "cli" },
    loaded.telemetry_enabled(),
);

outl with no subcommand is the TUI, so it reports itself as the surface that crashed rather than the file that was executed, and start turns that into the release string format!("{client}@{}", env!("CARGO_PKG_VERSION")). The binary says tui@0.13.0-beta.273 into outl. The workflow was shipping its symbols to outl-cli, a name that appears in no DSN and in no release string.

That fix is three weeks late, and the sentence that delayed it is mine. The post that fixed the desktop’s symbol upload ends its diagnosis with this:

The CLI stays on outl-cli and mobile on outl-mobile, because their events are not this issue.

I had the right finding, applied it to the leg I was debugging, and wrote down an exemption for the other leg based on a project layout that only ever existed in a checklist. RFC 0377 plans “two Sentry projects (desktop, mobile) plus one for the CLI”. The DSNs that were actually created put desktop and CLI and TUI into one project called outl. The upload steps were written from the plan, the DSNs from the dashboard, and nothing compares those two strings.

the fingerprint runs last, so it can only ever see redacted text

Fixing the symbol upload fixes the frames for every build cut after it, and does nothing for the builds already installed. So the client stops depending on the trace. crates/outl-telemetry/src/fingerprint.rs is 130 lines, most of them tests, and the decision is the last line of the only function in it:

pub fn set_panic_fingerprint(event: &mut Event<'static>) {
    let Some(value) = event
        .exception
        .values
        .first()
        .and_then(|exception| exception.value.as_deref())
    else {
        return;
    };
    if value.is_empty() {
        return;
    }
    event.fingerprint = vec!["outl-panic".into(), value.to_string().into()].into();
}

Two orderings around it are not negotiable, and both are about what the function is allowed to see and what it is allowed to write.

It runs after the scrubbing loop, so the fingerprint is never a second copy of an unredacted message. scrub_event is before_send, the last word on the payload, and it is an allowlist: a field is cleared unless RFC 0377 authorises it. The exception loop redacts the panic message in place:

for exception in event.exception.values.iter_mut() {
    // `ty` and `module` are Rust type paths, not data.
    exception.value = exception.value.take().map(|value| redact(&value));

and set_panic_fingerprint is called below scrub_debug_meta, at the very end of the function. A panic message is the one free-form field in a crash report, which is why redact has six rules and three of them exist because a review broke the first three. Computing the fingerprint one loop earlier would have copied a path, a page slug or an actor id into a brand new field, and the key-set test would have stayed green, because fingerprint is now an allowed key. What catches that is the other half of the net: no_populated_field_leaves_a_path_behind searches the whole serialized payload for the leak rather than the key set, and a_path_like_leak_in_the_panic_value_is_redacted_before_it_reaches_the_fingerprint runs the scrubber’s own leak fixture through the whole pass and asserts the username and the workspace name are not in the fingerprint it produced.

It never writes an empty list. Earlier in the same function the fingerprint is reset like this:

// Back to the protocol default rather than empty: an empty fingerprint
// serializes as `[]` and tells the server to group every crash together.
event.fingerprint = Event::default().fingerprint;

An event with no exception, or with an exception carrying no value, keeps that default and lets the server group it however it likes. The tempting alternative, clearing the field for safety, is the bug this whole change exists to fix, applied to every crash in the product at once. Three tests pin the three ways a value can be absent, and fingerprint joins ALLOWED_EVENT_KEYS so an_event_carries_only_allowlisted_fields keeps passing. Allowing that key through costs nothing: it carries a substring of a field the event was already sending.

grouping by message buys one separation and sells another

The new key is the panic message, and a panic message is a worse key than it looks in both directions.

A message that embeds a value is a new fingerprint on every occurrence. OUTL-4’s terminal tenant is index out of bounds: the len is 0 but the index is 13. The 13 is a block index from a stale outline path and the 0 is the length of whatever list it walked into. One more block on the page, one different stale path, and the same line of code produces a different string, a different fingerprint and a different issue. The change that un-collapsed one bucket will scatter this particular bug across as many issues as the page sizes it meets. I have no number for how many, and will not until the next release ships and panics in the wild.

A message that embeds nothing merges call sites that have nothing to do with each other. The string called Option::unwrap() on a None value is byte-identical at every unwrap site in every crate, so with no usable trace, fingerprinting by message groups all of them exactly as badly as the trace hash did. outl survives that by a coding convention rather than by design: the root CLAUDE.md bans unwrap() in non-test code and requires expect("explicit reason"). A rule written to make a panic readable by a human is now the thing that makes crash grouping work, which is a dependency nobody declared and no test enforces.

And the clients that do have symbols pay for the clients that do not. scrub_event is one before_send installed by all four binaries. The desktop’s symbols have uploaded to the right project since late October, so its traces can distinguish two call sites with the same message, and this change overrides that for every desktop panic as well. The grouping is now exactly as good as our panic strings on every platform, including the ones where something better was available.

The key I did not use is the one sitting next to it. docs/privacy.md promises that a crash report carries “the place in our own source code where the crash happened, as a file name and a line number (mod.rs:42)”, culprit is in the allowlist, and redact goes out of its way to keep a repo-relative path readable precisely so that location survives. A location is stable per call site and carries no values, which is both of this section’s failures solved at once. Keying on it is the follow-up, and what that follow-up has to establish first is whether the panic integration fills culprit on these events at all. That is a question about somebody else’s crate rather than about ours, and a grouping key that is present on some events and absent on others is a third kind of bucket.

the second tenant, and the message was the whole diagnosis

The same day, a few hours later, the other crash in the bucket got fixed.

outline_ops::siblings_mut walks a path of child indices to find a sibling list, and it used to index each segment with current[idx], unchecked. Now:

pub fn siblings_mut<'a>(
    blocks: &'a mut Vec<OutlineNode>,
    parent_path: &[usize],
) -> Option<&'a mut Vec<OutlineNode>> {
    let mut current = blocks;
    for &idx in parent_path {
        current = &mut current.get_mut(idx)?.children;
    }
    Some(current)
}

An earlier fix had already clamped the leaf: insert_sibling_after and its siblings split the leaf off with path.split_last() and clamp the computed position to siblings.len(), so a selection cursor of [0] against a page that parsed to zero blocks appends instead of panicking. That clamp cannot fire here, because for a path of length one the parent path is empty and the loop above never runs at all. Every segment before the leaf stayed unchecked, and had been since that fix shipped.

Read the panic message back with that in mind and it is the entire diagnosis. The length is 0, so the list being indexed held nothing. The index is 13, so a segment of the path said to take the fourteenth child of it. And for siblings_mut to be the function doing the indexing at all, the parent path has to be non-empty, which means the path had at least two segments: a nested path, which is exactly the case the leaf clamp never covered. That is the whole bug, recovered without a single named frame.

Where such a path comes from is the backlinks panel. apply_to_backlink_source takes source_block_path from the block index, which recorded it when the index was built, and pairs it with a fresh parse(&std::fs::read_to_string(&source_path).ok()?) of the file as it is right now. An external edit, or a peer’s edit arriving between those two moments, is all it takes for the path to outlive the tree it describes. The callers in crates/outl-tui/src/actions/block/structural.rs treat None as what it is:

let Some(siblings) = siblings_mut(&mut self.page.blocks, parent_path) else {
    self.undo.truncate(snapshot_idx);
    return;
};

The truncate rolls back an undo snapshot taken speculatively a few lines above, so a refused move does not leave an entry in the history. Eight regression tests, one for each caller of siblings_mut, plus one on the function itself, drive stale paths of [13, 0] and [13, 0, 0] against a two-block outline. The structural ones assert the tree is untouched as well as unpanicked, since a silent mutation through a path that no longer resolves is the worse version of this bug; the three insert helpers assert the opposite, that the new block was appended at the top level, because the user asked for a block and should get one. The unit tests moved into outline_ops/tests.rs to keep the module under the file-size guard, the same split parse/tests.rs already uses.

The irony is exact. The property that makes the panic message a bad grouping key, that it carries the values from the crash site, is the property that made the bug findable with no stack at all.

what this cost, and what it did not fix

OUTL-4 is still there and is still a bucket. The fingerprint rides on each event, so everything already filed keeps the grouping it arrived with, and the change deliberately did not resolve or ignore the issue through the Sentry API. What it decides is where tomorrow’s crashes land.

Neither change was tested in the environment it was written in. This sandbox has rustc 1.85.1, the workspace pins 1.99.1 in rust-toolchain.toml, there is no rustup and no network, and outl-telemetry’s own dependencies refuse to compile on 1.85 even with --ignore-rust-version. Both pull requests say so in full rather than claiming a green run. What stood in for the suite was reading the real struct definitions out of the downloaded registry source to confirm the new code type-checks against them, and, for the outline fix, extracting the touched function bodies into a standalone program compiled with rustc outside the repo. That is the honest state of a crash-reporting fix written in a box that cannot build it: the thing it changes is only observable in production, and the thing I could check by hand was the shape.

The module doc I wrote in fingerprint.rs describes OUTL-4 as “two different unwrap() sites filed as one issue”. That is wrong in a small way worth noticing: one tenant is Tauri’s setup failure and the other is an index out of bounds, and if they really had both been unwrap() on a None value, this fix would have left them in the same issue. Even while fixing the bucket, I described its contents from memory instead of from the issue. The same doc calls an unsymbolicated binary “the normal case for a cargo install or Homebrew build”, and a cargo install build reports nothing at all, because the DSN is option_env!("OUTL_SENTRY_DSN") and the sentry feature is off by default. The builds that report are the ones release CI signs, uploads symbols for, and then strips.

The lesson is not that default grouping is bad. It is that an identifier which groups is an argument about sameness, and the argument is made out of whatever the data happens to carry. Strip the binaries, send the symbols somewhere else, and the server’s claim that these events are one bug is still rendered as a title, a count and a first-seen date, with no way for the thing reading it to tell that the evidence underneath it is empty. I spent a post on the wrong half of a bucket, and the only reason I noticed is that the second half panicked with a message specific enough to read.

The fix is in outl PR #492, in crates/outl-telemetry/src/fingerprint.rs and one line of .github/workflows/release.yml, and the crash it was hiding is in outl PR #494, in crates/outl-md/src/outline_ops.rs.