blog · · engineering

The release notes named an installer nobody had built

outl's release pipeline used to decide what to publish from needs.<job>.result, which made every build job a single point of failure for every other job's assets: one bundler failed and a release holding 25 finished artifacts published none of them. PR #414 replaced that with scripts/release_assets.py, a table of every asset a release carries and which of them are required, read against the draft's own asset list. The first thing the new gate did was publish several releases that said, in their own notes, that the Windows ARM64 installer was missing. It was missing because the desktop job asked dtolnay/rust-toolchain for the cross target instead of running rustup target add (#421), while the CLI job for the same triple on the same image was green. The crates.io publish was failing at the same time for an unrelated reason, a caret requirement on 0.13.0-beta.270 also matching the stable 0.13.0 that sorts above it (#422), and nothing in the release notes said so, because crates.io is not an asset on a draft.

A
10 min read

build desktop Windows (aarch64) failed on every release from the day the job was added.

Every one of those releases published anyway, carrying the macOS dmg, the Linux bundles, both Windows CLI zips and the x86_64 installer. And every one of them said, in its own release notes, that outl-desktop-windows-aarch64-setup.exe was not there.

That is the pipeline change working. It is also the bill for it, because docs/getting-started.md had a row in a table promising that file, and I wrote it.

the gate read job results, and a job result is not a file

ARM64 became a published platform for outl in PR #414: Linux desktop bundles and CLI tarballs on native ARM runners, a Windows ARM64 CLI zip and desktop installer cross-linked from the x64 runner, a Homebrew arm64_linux bottle, a Scoop manifest serving both architectures, and linux/arm64 on the server image.

Adding architectures is what forced the other half of that PR. The release workflow creates a GitHub release as a draft before the build matrix starts, every job uploads its own assets to it with gh release upload, and a final publish_release job flips the draft to published. That job’s condition used to read needs.<job>.result == 'success' for each build job.

One build job failing meant no job’s assets published. That was already a bad trade before this change, and it had happened: one bundler failed and a release holding 25 finished artifacts published none of them. Every architecture added multiplies the ways one leg can take the rest of the release down with it.

The repo’s own workaround for the Android APK was the opposite extreme. build_android is signed with a throwaway debug key and is sideload-only, so it was simply left out of the conditions, which means a release with no APK published in silence. The same shape produced a v0.12.0 draft with zero assets: a job whose if omits always() carries the implicit “every ancestor succeeded”, that reads the whole ancestor tree rather than the direct needs, and on a GA run the tag job is skipped by design. Nothing went red. The release simply had nothing in it.

Neither extreme is a question about job results. The question is which files are on the release, and publish_release now asks only that:

    if: >-
      !cancelled() &&
      needs.prepare.outputs.dry_run != 'true' &&
      needs.create_release.result == 'success'

It still lists every build job in needs, because it still has to wait for them. It never asks what happened to any of them.

one table, because six places each held part of the answer

What it reads instead is scripts/release_assets.py, which is one Leg per build-matrix leg:

    Leg(
        job="build_desktop_windows",
        key="aarch64",
        assets=("outl-desktop-windows-aarch64-setup.exe",),
        required=False,
    ),

expected(version) turns the twelve legs into the full set of names a complete release carries: the assets, their .sha256 sidecars, and the Homebrew bottles. For a given version that is 40 names, 28 of them required. The twelve that may be missing are the Windows ARM64 CLI zip, the three Linux aarch64 desktop bundles, the Windows ARM64 installer and the Android APK, with their sidecars.

The gate itself is three lines, in scripts/check-release-assets.py:

    wanted = table.expected(version)
    missing_required = sorted(n for n, leg in wanted.items() if leg.required and n not in assets)
    missing_optional = sorted(n for n, leg in wanted.items() if not leg.required and n not in assets)

assets there comes from gh release view --json assets on the draft. A missing required asset exits 1 and the release never publishes. A missing optional one exits 0 and returns a note.

The same table answers a second question, which is why it exists as a file rather than as a condition. Six places have to name the same platforms and none of them can see the other five: the three build matrices, the two update_tap bump loops, Formula/outl-beta.rb, Casks/outl-desktop-beta.rb and bucket/outl-beta.json. Every way they can disagree is silent. A sed that matches no line exits 0, and the tap bump finishes on git diff --quiet, which only asks whether something changed, so the first thing to notice a stale anchor is somebody’s brew install one release later, with a hash mismatch or the previous build’s binary under the new version number. scripts/check-release-assets.py --manifest runs in hygiene.yml, parses rather than builds, and fails on the PR instead.

The checker’s own test suite is 28 cases that each break one file on purpose. test_a_matrix_leg_missing_from_the_table, test_an_asset_renamed_in_the_workflow, test_a_scoop_url_pointing_at_the_wrong_asset, test_continue_on_error_removed_from_a_job. Each one asserts the specific complaint, because a checker that passes because its own parser broke is worse than no checker, and this one parses YAML with regular expressions.

the gap goes in the release body, because that is the last writable moment

A published GitHub release in this repo is immutable. Assets cannot be added to it, and that is the whole reason for the draft: the flip is the last moment anything can be attached, and the last moment the notes can say what is not attached.

So the note is written there:

    lines = ["", "---", "", "**This build is missing some optional artifacts.**", ""]
    for (job, key), names in sorted(legs.items()):
        lines.append(f"- `{job}` / `{key}`: {len(names)} asset(s) did not land, including `{names[0]}`.")

One line per leg, not per file, pinned by test_the_note_names_one_line_per_leg_not_per_file: six missing files from one bundler leg is one fact about one architecture. The publish_release step appends that to the body the draft was created with, and copies it into the run summary, before flipping the draft.

A gap that publishes must not move a package manager. update_tap runs a third reading of the same table, per manifest, all or nothing: tap_assets(version, tap) is the set of digests one manifest points at, and the bump is skipped when any of them is absent. The rule it encodes is that a manifest carrying a new version and the previous release’s digest is a hash mismatch at install time, which is strictly worse than a manifest still pointing at a release that exists. test_an_optional_gap_still_blocks_the_tap_that_points_at_it is the case: the Windows ARM64 CLI zip is optional, so the release publishes without it, and the Scoop manifest does not move. Homebrew, which points at nothing on Windows, keeps bumping.

One detail there was not obvious to me until I wrote the step. That gate reads shas.json, built from os.listdir('shas') after the download loop, not from the release API. A digest that failed to download cannot bump anything whatever the API says the release has.

the first gap it named was a leg that had never once built

The pipeline shipped on 23 October, and the leg it was gating had never worked. Every beta from that day until 26 October published with the build_desktop_windows / aarch64 line appended to its body, because that is what the two steps above do with a missing optional asset and those releases did publish.

tauri build was dying with this:

Target aarch64-pc-windows-msvc is not installed (installed targets: x86_64-pc-windows-msvc)

The job had asked for that target. It passed the triple to dtolnay/rust-toolchain’s targets: input, and the run log shows the flag going in. The pinned action at 89b12181 runs rustup toolchain install stable --target aarch64-pc-windows-msvc, and on an image that already has stable for the host, what that run did was update the host toolchain: stable-x86_64-pc-windows-msvc updated, with no rust-std downloaded for the cross target.

The evidence that settles it was in the same workflow run, twice. The CLI build job builds aarch64-pc-windows-msvc on the same windows-2025 image, and on both red runs it finished green and uploaded outl-windows-arm64.zip. Its toolchain step is three lines and does not use the action at all:

      - name: Install toolchain
        run: |
          rustup show
          rustup target add ${{ matrix.target }}

Two jobs, one triple, one runner image, one run. The green one adds the target to the toolchain rust-toolchain.toml selects. The red one asked an installer to install a toolchain it already had. PR #421 makes the desktop job do what the CLI job does.

What that PR could not do is prove it on a branch. workflow_dispatch of release.yml with dry_run: true came back could not create workflow dispatch event: HTTP 403: Resource not accessible by integration, and the machine it was written on has no MSVC ARM64 toolchain, so there is no NSIS build behind the fix. What stands in for it is the CLI leg of the same triple on the same image, which installs the target the same way and finishes green. The leg stays optional in the table: promoting it is flipping one word, and that flip is the record that a release has actually shipped the installer.

a caret on a prerelease also matches the release that follows it

The second failure the same week was louder and reached nobody. Every push to main since the slug cap landed failed the “Publish lib crates to crates.io” job while verifying outl-actions:

cannot find value MAX_SLUG_BYTES in module outl_md::slug

MAX_SLUG_BYTES is a real const, pub const MAX_SLUG_BYTES: usize = 255; at crates/outl-md/src/slug.rs:58, and the caller is is_valid_slug in crates/outl-actions/src/page.rs, which refuses a slug that would not fit a path component. It is on the outl-md that same job uploads, 0.13.0-beta.270 and later. The verify build was compiling outl-md 0.13.0, the stable release from the v0.13.0 tag, which predates the const.

The publish loop was already careful about ordering: one -p at a time, outl-core, outl-md, outl-exec, outl-actions, outl-ws, and cargo waits for each upload to reach the index before the next crate’s verify resolves it. The ordering was not the problem. The requirement was. cargo publish turns a workspace path dependency into a version requirement, and the version it wrote was a bare 0.13.0-beta.270, which Cargo reads as a caret. The Cargo reference is explicit about what that matches:

Cargo will also upgrade automatically to semver-compatible released versions from prereleases. The requirement foo = "1.0.0-alpha" will allow updating to foo = "1.0.0" as well as foo = "1.2.0".

A stable release sorts above its own prereleases, so resolution picked 0.13.0 and the verify build compiled an outl-md from before the feature. The same page names the fix: when depending on a pre-release package and wishing to avoid breaking changes, a fully specified "=1.2.3-alpha.3" might be warranted.

PR #422 adds scripts/pin_workspace_deps.py, which runs after the version rewrite and turns each workspace-internal path dependency into version = "=<version>". [workspace.package] version stays a bare version, because =0.13.0 is not a package version. The script refuses rather than guesses:

        if bare not in line:
            raise PinError(
                "workspace-internal dep is not at "
                f"{version!r} and not already exact: {line.rstrip()}"
            )

A path dependency carrying some other version means the release sed did not run or did not match, and silently leaving a caret there is exactly how the verify build picks the stable crate again.

The dry run that proves the pin works is the one that fails. Running cargo publish --dry-run in publish order after the pin, outl-actions compiled outl-md 0.13.0-beta.270 and reached aborting upload due to dry run. outl-ws stopped before verify:

failed to select a version for the requirement `outl-actions = "=0.13.0-beta.270"`
candidate versions found which didn't match: 0.13.0, 0.13.0-beta.263, 0.13.0-beta.260

That is the exact pin refusing a version that is not on the index, because the red job had died on outl-actions and never published it. In the real job the previous crate is uploaded and waited for. On a laptop it is the resolution refusing to fall back, which is the entire point.

The cost of the pin is paid by embedders: from the next publish on, each published library crate depends on its siblings at exactly one version, so someone naming two of them uses one requirement. Versions already on crates.io stay as they were published, and the betas that failed were never republished.

the table answers for the release, and crates.io is not on it

That crates.io failure is the honest limit of this design.

publish_crates waits only on prepare and appears in nothing else’s needs. Neither does publish_npm. They are self-gating: cargo publish verifies by building, so a broken commit fails there on its own. Which means a red crates.io publish has never held back a GitHub release, and never will, and it also never appeared in a release note, because crates.io is not an asset on a draft. The table answers one question completely and says nothing at all about the other two publishing surfaces. For several days the signal that library consumers were a version behind was a red job in a list of green ones, which is precisely the kind of silence the asset table was built to end, one layer over.

The rest of the bill:

An optional leg means a release can ship incomplete, on purpose. Someone on a Snapdragon X machine who followed the Windows table in docs/getting-started.md found a zip and no installer, several releases in a row. That doc now carries the blind spot in writing: the ARM64 Windows builds are marked optional, a release can publish without them, and the release notes say so in that release’s own body. I would still rather ship a documented gap than a pipeline that quietly withholds 25 working artifacts because the twenty-sixth did not build, but the gap was real and it lasted three days.

The note is in the release body, which is not where everybody looks. A user who installs through a package manager never reads it. That is why the update_tap gate exists, and it is the half of this change that protects somebody who never visits the releases page.

The table is hand-written text held true by a parser. check-release-assets.py reads release.yml with regular expressions: it locates a job block by indentation, splits the matrix on the list marker, and reads gh release upload commands line by line, stopping at the first line that does not end in a backslash. It exits with a specific failure when it cannot find a job it expects, rather than passing, and the 28 mutation cases exist for the rest. It is still a parser reading a file whose format it does not own.

what the gate is for

The thing I got wrong was not the Windows toolchain. It was believing that a pipeline where every job goes green is the same as a release that carries what it promised. Those are two different claims, and only one of them is about a file. build desktop Windows (aarch64) had never produced an installer, and under the old rule the two possible outcomes were a release that published nothing because of it, or a release that published without it and said nothing. Both would have left the same hole and told me the same amount about it.

A green job is a statement about a process that has finished. An asset is a fact about a file that exists. When the two disagree, the file is right, and the only place the file can be observed is the draft, in the seconds before it becomes immutable.

That is the second time this month the same argument has rewritten part of this pipeline. The first was scripts/check-artifact-stripped.py, which opens every artifact a release ships and reads the strings inside, after every binary published since May turned out to carry the environment of the CI job that built it. The builds were green for all 28 of those releases. A check that only reads the pipeline cannot see inside the file, and a gate that only reads job results cannot see whether the file is there at all.

The table is scripts/release_assets.py and the gate that reads it is scripts/check-release-assets.py, both in the outl repository.