blog · · engineering

reading the workspace minted a writer

One machine's workspace had twenty actors in it, sixteen of them holding fewer than 500 ops. An actor in outl is a writer: it owns an ops-<actor>.jsonl nothing else appends to. Most of those identities were created by commands that write nothing. Every workspace open went through resolve_write_actor, which answers a held device lock by minting a fresh ActorId and locking that, which is correct for a process about to append and wrong for outl workspace info on a thirty-second poll beside outl serve. Compaction drops inert ops and keeps identities, so the cost outlives the bytes. The three obvious repairs each fail for a different reason, and the most instructive is that merging a small actor back into the device actor rewrites the HLC tiebreak. The fix had to be at the open, which is why the workspaces already holding twenty actors keep all twenty.

A
11 min read

A workspace on one machine had twenty actors in it. Sixteen of them held fewer than 500 ops.

An actor in outl is a writer. Each one owns a file, ops-<actor>.jsonl, that nothing else appends to, and that is the whole reason a last-write-wins file transport like iCloud or Syncthing cannot lose an op: two writers never touch one file. Twenty actors should mean twenty devices, or at least twenty processes that each had something to write. This was one machine with about nine automated writers on it, and most of those identities had been created by commands that write nothing at all.

outl workspace info prints a few counts and exits. Run it on a thirty-second poll beside a running outl serve, which is exactly what a container health check looks like, and it left a fresh ops-<ulid>.jsonl behind on every tick. The reason the published Docker image ships no HEALTHCHECK is written down in docs/self-hosting.md, and it was this: the probe would have been the outage.

a held lock is a writer’s question, and the read was asking it

Every opener went through one function. outl_ws::open resolves which actor this process may write under, and the resolution is four lines of crates/outl-core/src/lock.rs:

pub fn resolve_write_actor(
    ops_dir: &Path,
    config_actor: ActorId,
) -> Result<(ActorWriteLock, ActorId), LockError> {
    match ActorWriteLock::try_acquire(ops_dir, config_actor) {
        Ok(lock) => Ok((lock, config_actor)),
        Err(LockError::AlreadyHeld(_)) => {
            let ephemeral = ActorId::new();
            let lock = ActorWriteLock::try_acquire(ops_dir, ephemeral)?;
            Ok((lock, ephemeral))
        }
        Err(other) => Err(other),
    }
}

That is the right answer to the question it was asked. The lock is an advisory flock on ops/.lock-<actor>, so it arbitrates between processes on one machine, and the thing it must prevent is two of them appending to one actor’s log. When the device actor is taken, a process that is about to write cannot wait and cannot share, so it takes a new identity and writes there. The cost is one more actor, paid by a process that genuinely needed one.

The read inherited all of it. A read has nothing to append, holds no claim worth arbitrating, and still asked whether it could have the device actor, still lost, still minted, still locked. And it lost in the worst possible way: not occasionally, the way a race loses, but every single time. The holder in the measured pile is a long-lived process, outl serve or a GUI, which takes the device actor at startup and keeps it for the session. A poll against a machine like that does not sometimes mint. It mints on every tick, forever, at whatever interval the poll runs.

compaction drops ops and keeps identities

A new file in ops/ sounds like a disk problem, and disk is the one cost here that goes away.

outl compact exists and works: it rewrites each log in place, dropping provably inert ops. What it does not do is drop an actor. An identity whose ops have all been collected is still a name in the directory, and that name is load-bearing in four places at once. It is an entry in the HashMap<ActorId, ActorClock> both sides exchange on every sync handshake. It is a key in the per-actor vector clock a snapshot cutoff has to be, because a single global high-water mark silently drops a lagging peer’s op. It owns two index sidecars: a cold boot on a 217,811-op workspace rebuilds 435,622 index entries across 40 sidecar files, two per actor. And it owns a zero-byte .lock-<actor> that nothing ever removes.

The per-actor work is also repeated, not paid once. oplog_census::actor_census reads one offset index per ops-*.jsonl to build the vector clock, and the catch-up loop re-dials each peer on an 8 second tick, so every actor in the directory is a file probe on every exchange with every peer. An actor holding one op costs very nearly what an actor holding fifty thousand costs, because what is being counted is identities.

So the bug is not that a read did extra work. It is that reading my own workspace created something permanent, and the permanence is structural rather than an oversight. This is the same gap I walked into with the trash, where an append-only log preserved every byte and still could not answer the question I eventually wanted to ask. The log guarantees that nothing is lost. It guarantees nothing at all about what can be reclaimed.

merging a small actor back rewrites what the log already decided

Sixteen files under 500 ops looks like a cleanup job. Fold them into the device actor, rewrite the actor field on their ops, delete the files. The issue said that is not safe, and it is right, for a reason that lives in nine lines of crates/outl-core/src/hlc.rs:

impl Ord for Hlc {
    fn cmp(&self, other: &Self) -> Ordering {
        self.physical_ms
            .cmp(&other.physical_ms)
            .then(self.logical.cmp(&other.logical))
            .then(self.actor.0.cmp(&other.actor.0))
    }
}

The actor is not a label on an op, it is the last third of its position in the total order. Two ops stamped in the same millisecond with the same logical counter are separated by comparing the two actors’ ULIDs, and nothing else. The move-op CRDT converges by replaying ops in exactly that order, undoing and redoing to slot a late one into place, so rewriting an actor field is rewriting the order a tree was built in. Ops that used to tie and resolve one way resolve the other way, on this device, while every peer that already ingested them keeps the original answer.

Where the rewrite collapsed a tie entirely it would be worse than a reorder. Op identity is the whole Hlc { physical_ms, logical, actor }, and Workspace::apply deduplicates on it with self.log.contains_ts(&op.ts). Two ops from two actors that happen to share a millisecond and a counter become byte-identical timestamps after a merge, and the second one is then discarded as a re-delivery of the first. Silent, and in the direction that deletes.

That is also why the GUI clients refuse a contended open rather than forking an actor, which docs/clients.md has said for a while: a GUI’s HlcGenerator is built in setup(), before a workspace is even picked, so swapping only the storage actor would leave two live generators stamping the same actor id, and two generators ticking in the same millisecond emit the identical triple. A fallback that loses ops is worse than the collision it dodges.

the other two repairs do not survive their own measurement

A reusable pool of secondary actor ids, persisted per device, is the design that most people reach for second. It is also a second store of actor identity, sitting outside the workspace and synced by nothing, which is precisely the kind of state outl’s ninth invariant demands four answers about: who writes it, who reads it, how a test gets its own copy, and what cleans it up. Reuse would additionally have to resume each id’s HLC from its existing log on every run. That is real machinery, and the issue’s definition of done did not ask for it.

Waiting briefly for the device lock instead of minting is the one that sounds cheapest and fails on the measurement rather than on principle. The holder in the pile that was actually counted is a long-lived process that does not release the lock for the session. A one second wait, or a ten second wait, loses the same way the immediate attempt loses, and then the command either mints anyway or fails. It bounds nothing.

Every repair aimed at cleanup is aimed at the wrong end. So the change is at the open.

so a read stopped asking

outl_ws::open_read takes the shared workspace lock, which admits any number of concurrent openers, resolves this device’s actor from the device store, and skips the question entirely:

let (actor_lock, actor, ephemeral_actor) = if opts.writes {
    let (lock, actor) =
        resolve_write_actor(&paths.ops, device_actor).map_err(WsError::internal)?;
    (Some(lock), actor, actor != device_actor)
} else {
    (None, device_actor, false)
};

The actor it reports is still the device actor, because the id names what this device writes under and the read is not claiming anything. Readers merge every ops-*.jsonl regardless, so the per-actor lock was never a read gate in the first place.

The Default impl for OpenOptions is hand-written, and the comment next to it is the reason. A derived Default zeroes a bool, and a zeroed writes turns open(), the function every mutating subcommand uses, into a read:

impl Default for OpenOptions {
    fn default() -> Self {
        Self {
            write_snapshots: false,
            writes: true,
        }
    }
}

Everything after that is a backstop placed where it fails loudly. After per-page shards are registered, and only after, the open calls Workspace::disallow_local_writes, which reaches every shard rather than just the global log: a shard is a fresh JsonlStorage opened with writes allowed, so disallowing the global one alone would leave a shard able to create a file. JsonlStorage::append_ops_inner then refuses before the create(true) that would mint the file:

if !self.writes_allowed {
    return Err(StorageError::Backend(
        "refusing to append: this storage was opened read-only and holds no write lock"
            .to_string(),
    ));
}

And WsCtx::commit refuses earlier still, before commit_page runs the mutation, because the storage refusal fires after the in-memory tree has already moved. Slug repair is skipped on a read for the same reason: it can emit ops, and the next writing open still runs it.

The CLI’s read subcommands open through it now: workspace info, search, query, page get / list / render, block get / tree, backlinks, history, export, tag, trash list, template list / resolve, prop get / list, daily range. open() is untouched, and a writer that loses the device lock still mints, because a writer still has to.

doctor counts them and will not delete one

The report that was supposed to surface this had two holes. It printed a line about lock files only when at least one lock existed, so a workspace full of peers’ logs, which carry no .lock-<actor> on this device, showed nothing. And comparing locks against logs never fires on the real pile, because the lock files are never deleted either, so a device that minted one actor per process sits at locks roughly equal to logs.

It reports the ops-*.jsonl count on its own now, and warns on the locks past a threshold that owns its own number:

/// The device actor, a GUI, `outl serve` and the MCP server are four,
/// and two overlapping CLI writes on top of that is still an ordinary
/// machine. Eight is the pile issue #298 measured: about nine automated
/// writers, and the locks are never deleted, so the count tracks every
/// process that lost the race rather than how many are alive now.
pub(in crate::cmd::doctor) const EPHEMERAL_ACTOR_WARN_AT: usize = 8;

Seven stays an informational line, pinned by fewer_write_locks_than_the_threshold_stay_a_note, because a warning on an ordinary machine teaches a user to skim the one that matters. And nothing is ever offered for deletion. A .lock-<actor> is arbitration state, not cache: remove one a live process is holding and the next process creates a fresh inode at the same path, flocks that, and appends to the same ops-<actor>.jsonl. It is also zero bytes, so there is no reclaim to weigh against it.

what a read still leaves behind

Opening storage still writes an index sidecar. .ops-<actor>.idx is a dot-prefixed local boot cache that never rides the sync surface, and it is explicitly out of scope here, which means a read-only open is not a read-only directory. The test helper that proves no actor was minted had to be written around that, and says so:

/// The names that are actor identities or the locks that gate them.
/// Index sidecars are a boot cache and are not part of this set.
fn actor_names(ops: &Path) -> BTreeSet<String> {

A read also still runs Workspace::seed_clock, and a caller that reaches past WsCtx::commit into Workspace::apply directly gets the pre-existing shape: the in-memory tree mutates and then the storage append fails, leaving a dirty tree and an unchanged log. The CLI does not do that, and the test asserts both halves of the behaviour rather than only the one the CLI exercises.

Skipping slug repair has a visible cost. A split-brain page root, two roots sharing one slug, stays split until the next writing open, and until then a read shows both.

Writers that lose the lock still mint, so a machine whose overlapping processes are all long-lived writers keeps growing actors at the same rate it always did. This change bounds the unbounded automated case and nothing else.

And the workspaces already at twenty actors keep all twenty. Every ops-*.jsonl, every .lock-<actor>. Nothing rewrites an id, which is what keeps the peers that already ingested those ops converging on the same tiebreaks, and it is the honest consequence of the argument against merging: if rewriting an actor is unsafe as a repair, it is unsafe as a migration too.

The question I did not ask early enough is not whether an operation is expensive. outl workspace info is cheap, and it was cheap the whole time. The question is what an operation creates that nothing will ever remove, and the answer to the fourth of invariant 9’s questions, what cleans this up, had been “nothing” for long enough that the rate it was being fed at stopped looking like part of the design. A garbage collector you have decided not to write is a defensible choice exactly until something starts allocating in a loop.

open_read is in crates/outl-ws/src/lib.rs, the refusals are in crates/outl-core/src/storage/jsonl/append.rs and crates/outl-core/src/workspace/router.rs, the counting is in crates/outl-cli/src/cmd/doctor/repair/index_sidecars.rs, and the alternatives and what they cost are written up as RFC 0298 in the outl repository.