The slugifier only knew Latin, and it was choosing node ids

[[你好]] and [[世界]] opened the same page.

So did [[Rust 笔记]], except that one opened [[Rust]]. Every #tag written in a script the slugifier did not know collected on that same page as well. This was not two files colliding on one name on disk. It was one node, carrying one id, the same id on every device, holding the blocks and the backlinks of every name that reached it.

a slug is one path component, and also the page’s identity

The module doc at the top of crates/outl-md/src/slug.rs opens with a sentence that is the whole defect:

//! Slugify page names for stable filesystem paths.

That describes a filename function, and a filename function is allowed to be lossy. [[São Paulo]] becomes pages/sao-paulo.md, [[Q4 plan]] becomes pages/q4-plan.md, [[os/linux]] becomes pages/os-linux.md because a slug is a single path component and page::is_valid_slug in crates/outl-actions/src/page.rs rejects / outright. Folding is the point. The name the user typed survives in the title:: property, which is where the hierarchy for nested pages is read from.

Then there is the other thing a slug does, in crates/outl-core/src/id.rs:

pub fn from_slug(slug: &str) -> Self {
    Self::from_seed(b"outl-page:", slug)
}

from_seed is sha256(prefix ++ seed) truncated to sixteen bytes and read as a ULID. A page root’s identity is its slug, so that two devices opening the same day’s journal offline create the same node and merge instead of splitting. I added that derivation to close a split brain, and then paid for it again when both devices wrote the title into one node’s text CRDT.

Hold the two facts together. A function that deliberately maps many names onto one string is the input to a function that decides who a page is. Every pair of names that slugify folds together is not two files racing for one path. It is one page, by construction, everywhere, with no file involved at all.

the fold table was the alphabet

Here is what the old rule kept: ASCII letters and digits, plus a hand-written table of Latin letters that fold to an ASCII base, upper and lower case. á to a, ç to c, ñ to n. Everything else became -, consecutive dashes collapsed, leading and trailing ones were trimmed, and a slug that came out empty fell back to untitled.

Read that against a page named 你好. Neither character is in the table and neither is ASCII, so both become -, the dashes collapse and trim, the result is empty, and the slug is untitled. 世界 takes the same path. Two names, one slug, one NodeId::from_slug("untitled"), one node.

The same arithmetic ran on anything that was not English. Проект, こんにちは, 한글, ΛΟΓΟΣ. And it was not only non-Latin scripts: the table has no entry for ł, ř, ş or ộ, so Polish, Czech, Turkish and Vietnamese names lost those letters too, which means miał and mia were one page and Řeka and eka were one page. The half-collapse was the quietest version. [[Rust 笔记]] kept the part the table understood and threw away the part that made it a distinct name, so it slugified to rust and opened the Rust page that already existed.

The new loop keeps a letter or digit the table has no answer for:

match fold_char(ch) {
    Some(folded) => out.push(folded),
    None if ch.is_alphanumeric() => out.extend(case_fold(ch)),
    None => {}
}

Three arms, and the middle one is the fix. fold_char still wins where it has an entry, so café is still cafe. Emoji are not letters, so ship 🚀 is still ship. A letter or digit with no ASCII fold now spells itself: [[你好]] projects to pages/你好.md, [[Проект]] to pages/проект.md.

the merge happened twice, on two unrelated channels

The node id is one channel. The backlink index is the other, and it collapsed independently.

crates/outl-actions/src/backlinks_keys.rs indexes a #tag under its slug, so #Avelino and the page avelino meet:

keys.push(TargetKey::Tag(outl_md::slug::slugify(t)));

A page looks itself up under the slug form of its own slug and its own title. So every tag written in a script the table did not know keyed to Tag("untitled"), and the one page whose slug was untitled collected all of them, regardless of what any of those tags meant. Fixing the slugifier fixed that channel at the same time, because both sides of the lookup go through the same function. That is the one piece of luck in this bug: there were two merges, and one owner of the rule that caused both.

NFC, because one name spelled two ways is two pages

Keeping the characters introduces the mirror problem immediately. 你好 has one spelling, but plenty of names do not. macOS hands filenames back decomposed, a Korean IME composes Hangul out of jamo, and a git checkout without core.precomposeunicode does its own thing. One name on screen, two byte sequences, two hashes, two pages that are visually identical and that nothing will ever merge.

So slugify normalises its input, for ch in name.nfc(), and the read-back direction normalises too. crates/outl-md/src/reconcile/page_root.rs derives a page id from the filename stem when a .md arrives with no sidecar:

fn slug_from_md_path(md_path: &Path) -> String {
    md_path
        .file_stem()
        .map(|s| s.to_string_lossy().nfc().collect())
        .unwrap_or_default()
}

That .nfc() was safe to leave out for as long as every slug matched [a-z0-9-], where NFC and NFD are the same bytes. It is load-bearing now, and a_decomposed_stem_names_the_same_page_as_a_composed_one pins it with the Hangul spelled both ways. Without it, a stem that comes back decomposed mints a second page root for a page that already exists, which is the split brain that had two “today” pages flickering on my phone arriving through a new door.

lowercasing through uppercase, because that is how the disk compares

This is the part I did not see coming, and it is the one I would have shipped wrong.

A slug is lowercased. The obvious way to lowercase a character Rust already hands you is char::to_lowercase. That is correct as a Unicode operation and wrong as a filename operation, because a case-insensitive filesystem does not compare lowercase forms. APFS and NTFS, both case-insensitive by default, compare by uppercasing. And there are characters that are each already lowercase, stay distinct under to_lowercase, and share one uppercase form.

The micro sign µ and the Greek letter μ are the readable example. Under the old rule both were dropped, so µs latency and μs latency were the same page in the same way everything else was. Keep them and lowercase them the obvious way and they become two names, two distinct lowercase forms, two slugs, two page ids, and one file on a Mac to hold both of them. One page silently overwrites the other, or both freeze on PageMarkdownAheadOfLog and stop syncing, and the rebuilt sidecar makes the loss undetectable afterwards. So case_fold routes through the uppercase, and the slug gives the same verdict the filesystem gives:

fn case_fold(ch: char) -> impl Iterator<Item = char> {
    ch.to_uppercase()
        .flat_map(char::to_lowercase)
        .filter(|c| c.is_alphanumeric())
}

Its doc counts 21 groups of characters that do this: s against long ſ, final sigma against sigma, the Turkish dotless ı against i, the Greek symbol variants, the Old Cyrillic letters. None of them was reachable while every slug was [a-z0-9-], because only the ASCII member of each group ever survived. Keeping the characters is what armed them.

The filter on the last line is not tidying. An uppercase-then-lowercase round trip can produce a combining mark, İ gives i plus a combining dot, and a mark inside a slug would make the next slugify of that slug return something else.

the property is pinned over every codepoint, because the breaking character is never the one you think of

Both rules are tested exhaustively, over all 1,114,112 codepoints, rather than over a list of the cases anyone thought of. no_two_slugs_land_on_one_file_on_a_case_insensitive_disk groups every alphanumeric character by its uppercase form and asserts that each group produces one slug. every_character_slugifies_to_a_fixed_point asserts that slugify(slugify(c)) equals slugify(c) for every character.

The second one earns its runtime the first time it runs, which was inside this change. The table folds ß to s and had no entry for ẞ, the capital sharp s, so with the new keep-verbatim arm in place ẞ fell through it, lowercased to ß, and the next slugify of that result found the table and returned s. One German word, two slugs, two node ids, depending on how many times the string had been through the function. And the resolution ladder does slugify a string that may already be a slug, so that path is live rather than theoretical.

There were three hand-written idempotence cases in the suite. All three passed. All three landed on characters the table has no entry for, which is precisely the shape that cannot catch this. ẞ is in the fold table now, with a comment saying it is the only uppercase letter whose lowercase form the table folds and whose own form it did not.

the 255-byte cap became reachable, and truncating alone is this bug in miniature

MAX_SLUG_BYTES is 255, because every mainstream filesystem caps a path component there. Under the old rule that limit was close to unreachable: a name long enough to hit it in ASCII is not a name anyone types, and a name in another script collapsed to the eight characters of untitled long before it could get near. Keeping the characters changes the arithmetic. 255 bytes is 255 ASCII characters and 85 CJK ones, which is one ordinary sentence.

So slugify has to shorten, and the shortening is the same bug again at a smaller scale: two long names that share a prefix, truncated to that prefix, are one slug and therefore one page.

let digest = hex::encode(&Sha256::digest(slug.as_bytes())[..8]);
let mut cut = budget - digest.len() - 1; // room for `-<digest>`
while !slug.is_char_boundary(cut) {
    cut -= 1;
}
format!("{}-{digest}", slug[..cut].trim_end_matches('-'))

Three decisions in five lines. The digest is of the slug and nothing local, so two devices that each type the same long name still converge on one page. It is 64 bits rather than 32, because the suffix is the only thing keeping two long names apart and 32 bits is roughly 2^16 tries to collide two names on purpose, which is cheap for a forged [[ref]] inside an imported vault. And the budget is 245, not 255, because the slug is not the filename: every write renames .<slug>.outl.tmp over its target, ten bytes longer than the slug. A slug filling the whole component would commit a page to the op log that no device can ever project, which is worse than the refusal it replaced.

A hundred-character CJK name therefore keeps its first 76 characters and gains a sixteen-hex-character digest, landing at exactly 245 bytes. a_long_name_in_another_script_still_opens in crates/outl-actions/src/resolve.rs checks the resulting slug against is_valid_slug, and then checks that a name sharing those first characters is still a different page.

what the repair reopens, and what it cannot

Prevention does nothing for the pages that already merged, and this is where I have to be straight about what shipped.

A name that collapsed is still reachable. The resolution ladder in crates/outl-actions/src/resolve.rs tries the literal slug, then the slugified form, then a case-insensitive title match before creating anything, and the merged page carries a title:: property holding the first name that was typed. So if [[你好]] got there first, [[你好]] still opens that page, through the title rung, and finds everything in it.

[[世界]] does not. It now slugifies to 世界, matches no slug and no title, and gets a fresh empty page. Whatever was written under that name is still sitting in the page titled 你好, at pages/untitled.md, and nothing can move it, because nothing recorded that it belonged somewhere else. A block’s creation in the op log is an Op::Create carrying a node id and a parent node id. The parent was the merged root. No operation anywhere says which name the writer had on screen when they typed it.

That asymmetry is worth sitting with, because outl already has the repair for the mirror case. merge_duplicate_slug_roots in crates/outl-actions/src/page_merge.rs handles two roots that share one slug: group by slug, pick the canonical root, re-parent every child, trash the empty duplicate, all through ordinary ops so it converges to every device. That direction is recoverable because each root carries its own identity and the merge is a choice. This direction is not, because the merge already happened inside the id, losslessly in the wrong direction. The only separation tool available is a human reading blocks and moving them, which is exactly the work the automatic repair exists to avoid.

outl plugin init keeps a refusal that used to be free. A plugin’s slug becomes a reverse-DNS label in its id, where outl_plugins::manifest::validate_id accepts [a-z0-9-] only, and npm has its own rules for the package name. Before this change, a name in another script slugified to untitled and was rejected by the fallback check. Now it slugifies to itself and sails through, so crates/outl-cli/src/cmd/plugin_init.rs asks the question directly and says what it needs, at scaffold time rather than at install time.

what it still costs

Three things, stated rather than fixed.

Emoji still collapse, which means the bug survives in a narrower domain. A name made only of emoji or punctuation still slugifies to untitled, so [[🚀]] and [[📝]] still resolve to one node the same way [[你好]] and [[世界]] used to. a_stem_that_slugifies_to_nothing_still_links_back, in crates/outl-actions/src/open_with/tests.rs, exists because that path is still live: a file named 📝.md opened from the OS still lands on a page whose slug does not derive from its title. Emoji are genuinely not letters, and the alternative is encoding codepoints into filenames, so the fallback stays. It stays with its collision intact.

The keep-or-drop decision now follows the build’s Unicode tables. char::is_alphanumeric, char::to_lowercase and NFC all come from the toolchain, so a codepoint assigned between two Rust releases slugifies differently on binaries built either side of it, and the page id goes with it. Freezing a table inside the crate does not remove this, since two versions of a frozen table disagree the same way; it only adds that a newly assigned script stays broken until somebody ships an update, which is the bug this change exists to fix. The exposure is one name using a codepoint assigned since the older binary shipped, and on that binary the result is what every name used to get.

Accent folding is still Latin-only, and that is now visible. CAFÉ and cafe are one page, because the table holds é. ΛΟΓΟΣ and λόγος are two, because it holds no Greek. Case is handled for both, down to final sigma, so this is the table’s scope and not a case bug. Widening it would be a latinising decision for scripts where the accent carries meaning, and that is a separate argument with different people in it.

the question to ask when you make an id deterministic

Deriving a page id from its slug was the right call and I would make it again. It is what lets two offline devices create the same journal and land on one node, with no coordination and no server.

What I did not do is re-read the function whose output I had just promoted. slugify was correct for the whole of its stated job, and its stated job was the first line of its own module doc: filesystem paths. Lossy is a feature there. The day its return value started seeding sha256("outl-page:" + slug), every collision it was designed to produce became a collision in the identity space of the document, and nothing in the change that did the promoting asked which names it folded together.

So the question, for the next time something becomes an id: what does the function that produces this value deliberately throw away, and who was relying on that being harmless? A hash of a lossy string is a lossy identity. It converges beautifully. It converges two different pages onto one node and reports success, on every device, at the same time.

The slugifier is crates/outl-md/src/slug.rs and the original report is issue #405.