Internal engineering brief
An honest map of how a change gets from your request to production today, the five places it actually breaks, and a phased plan to make the path visible and reliable end to end.
Section one
There are three separate delivery paths, each with its own rules. They do not share gates, they do not share test coverage, and only one of them runs the test suite at all.
Every job runs on the self-hosted runner on AS3 -- the container on your own server, not GitHub's machines. That part is correct and consistent across all workflows.
No unit test suite anywhere on this path. The smoke test loads the app in a browser; it does not assert that anything was written to the database.
There is no test step and no end-to-end step on the staging path at all. Staging is gated entirely on schema and structural guards.
Step 2 changes the live production database. Step 4 can then fail and stop the pipeline before step 5. That ordering is the single most consequential design decision in the whole system.
| Surface | Full unit suite | Browser E2E | Database round-trip proof |
|---|---|---|---|
| PR preview | No | Yes -- smoke + WebKit load | No |
| Staging | No | No | No |
| Production | Yes -- after schema publish | No Playwright; an authenticated canary hits one admin endpoint | Partial -- the canary proves one read path is alive |
| Local git push | Yes -- still runs in the hook | No | No |
The schema-safety machinery is strong and hard-won. The bindings-drift gate (does the committed client code match the database schema) runs before every publish on both staging and production. The reducer-drop gate catches a function disappearing out from under the client. The feature-inventory guard reads the built output and fails if a known feature vanished. The authenticated production canary is a real end-to-end check that catches the "200 OK shell over a dead app" class of outage that a plain health check misses. The staging pipeline correctly queues rather than cancels. None of that should be weakened by anything in this plan.
The system is over-invested in schema safety and under-invested in behavior safety. Staging deploys with zero tests; production runs all 1948 of them, but only after it has already changed the production database.
Section two
Six findings. Five are confirmed against the actual files and the live run history. One I could only partially confirm and I say so rather than assert it.
Every single push to staging-next since roughly 11:46 this morning has failed, and it fails at the first step -- publishing the database schema. Six consecutive red runs. Meanwhile pull requests kept merging into that branch.
The cause is a genuinely unsafe migration: the schema in the branch removes columns and a whole table that exist on the live staging database. SpacetimeDB correctly refuses. Because the publish is step one, the build and deploy never run -- so the staging site is still serving an old build and nobody was told. This is exactly the "did my change go live, I have to guess" problem, in its most literal form.
You asked me to verify this specifically. It is true, and the step is even named for it. In deploy.yml, the order is: publish the production database schema, then build, then run the 1948-test suite, then deploy the client.
So if the suite fails, production is left with a new schema and the old client. Because SpacetimeDB encodes rows positionally, an added column that the deployed client does not know about produces the byte-overflow crash that took the site down on 2026-04-22. The pipeline exists to prevent that outage and, in this specific failure window, recreates it.
Two honest qualifiers. The bindings-drift gate runs before publish and catches the common version of this, so the window only opens when the schema change is legitimate but some unrelated test fails. And publishes use --delete-data=never, so additive columns are usually tolerated by an older client. The window is narrow. It is not closed, and the blast radius when it opens is the whole site.
Production is configured to cancel any in-progress deploy when a new one starts. Combined with finding 2, the half-done state is well defined and bad: the schema publish is fast and near the front, the client deploy is slow and near the end. A cancellation almost always lands after the database changed and before the site was updated.
Worth noting the staging pipeline gets this right and says so in a comment -- it queues instead of cancelling, explicitly to avoid this. Production has the opposite setting.
The commit hook was disabled on 2026-07-22 exactly as ruled. The push hook was not -- it still runs the entire 1948-test suite, on your desktop, on every push, roughly five minutes each time. Its own comment even advertises the cost.
This is the "huge test suites killing our systems" complaint, still live in this working tree. It is also the mechanism behind the orphaned-suite leak: interrupting a push kills the foreground git process, not the suite it already spawned.
Several steps pass unconditionally. Each was a deliberate, documented decision at the time; together they mean a green check mark means less than it appears to.
if: "false" in both deploy.yml and pr-preview.yml. Per-PR databases are never deleted. They accumulate on the AS3 SpacetimeDB host indefinitely.ensure_module_version on production is suffixed || true, so a failure to stamp the deployed version is invisible.continue-on-error: true, which is load-bearing rather than sloppy -- but it means merge decisions hinge on step outcome comparisons rather than a hard gate.Confirmed structurally. Each pull request gets its own newly provisioned, freshly seeded database. Any bug whose trigger is accumulated state -- an auto-increment counter that has run far enough to collide, a tier gate that only bites an existing paid user, a sequence that has advanced -- has no way to fire on a preview. The 2026-05-18 incident was exactly this shape.
The caveat, and the reason it stays dangerous: the preview end-to-end tests are browser smoke tests. I found no step on any path that writes a row, reads it back, and asserts its contents. So the preview both lacks the data state and lacks the assertion that would notice. Your standing rule that write-path E2E must prove a database round-trip is documented in the project instructions but is not enforced anywhere in CI.
The staging.yml in this working tree triggers on pushes to dev/live-tunnel. The runs actually firing today are triggered by pushes to staging-next. So the copy of the workflow in this branch is behind the copy on staging-next. I am reporting what I read plus what the live history shows, rather than guessing which is authoritative. It matters because reading a workflow file from the wrong branch gives a confidently wrong answer about how the system behaves -- which is itself a form of the guessing problem.
The failures are not random. Every one of them is the same shape: an irreversible or expensive step runs before the cheap check that would have stopped it, and when it goes wrong nothing announces it.
Section three
Five phases, ordered so each one closes a specific failure above. Everything here sits inside your locked rules: no tests in hooks, tests run in the swarm, full suite only at the production boundary, focused tests for staging and dev, ship-on-green to staging automatic, production promotion initiated by you.
npx vitest run block from the push hook, leaving the cheap guards it also runs: type check, bindings drift, and the ASCII commit-message check the Cloudflare API requires.|| true from the version stamp so a failed stamp is visible.if: "false", and on any continue-on-error: true that lacks a dated expiry annotation. Disabling a gate becomes possible but never silent.Phases 0 and 1 are the ones that stop outages. Phases 2 through 4 are what stop the slow drift back into guessing.
Section four
You should be able to answer "where is my change right now" in one glance, without opening GitHub, without asking, and without reading a workflow file from the wrong branch.
The information already exists. It is scattered across GitHub Actions run pages, Cloudflare Pages deployments, the SpacetimeDB module version stamp, and the pull request list. Nothing needs to be invented -- it needs one surface.
Extend the existing pull-request dashboard, which already runs locally and has systemd and a watchdog behind it. Add a single delivery pipeline view: one row per change, columns for each stage, refreshed on a short interval. It lives at the same local address you already use.
| Column | What it shows | Where the truth comes from |
|---|---|---|
| Change | Plain-English description plus branch, never a bare number | Pull request title and branch |
| Tests | Focused or full, pass or fail, and which swarm node ran them | Swarm job result record |
| Adversarial | Confirmed, refuted, or not yet run | Verification record |
| Round-trip E2E | Yes, no, or not applicable for read-only changes | The Phase 3 check |
| On staging | Whether this exact commit is the build the staging site is currently serving | Version stamp fetched live from the staging site, compared to the commit |
| On production | Same comparison against the live production site | Version stamp fetched live from lab.joyos.global |
| Blocked on | The single reason it has not moved, in plain words | Failing step name from the run |
The two deployment columns are the important ones, and the reason is finding 1. Today a change can be merged, green in the merge workflow, and still not on the staging site because the publish step failed hours earlier. Reading merge status tells you nothing about that. Asking the live site what version it is serving is the only answer that cannot lie. Both sites already stamp a build version, and production already verifies its own stamp after deploy -- the dashboard just has to poll them.
One rule, not a notification stream: if a change has been merged for more than fifteen minutes and the target site is not serving it, that is an alert. That single rule would have caught this morning's staging outage on the first failed run instead of after six. Everything else can stay quiet -- silence should mean the pipeline is flowing, and today it does not.
Status is not "did the workflow pass." Status is "is the site serving my commit." Measure the second one and the guessing ends.
Section five
Four genuine forks. I have a recommendation on each, but these are yours to call -- guessing past any of them is how the unreliability comes back.
The job queue now fails closed when a job cannot be placed on a swarm node -- it refuses rather than quietly running on your desktop. But there is one exception: when your local working tree has uncommitted changes, it currently only warns and runs locally anyway. The reasoning was practical -- most test runs during active development happen against a dirty tree, so failing closed there would refuse a large share of every project's runs.
Making it strict is literally one word added to fail_closed_reasons in /home/ethan/bin/jobq.
Option C. It honors the rule as written -- no fallback to the desktop by accident -- without blocking the genuine mid-edit case, and the log makes the exception visible rather than habitual. Option B is the honest reading of your ruling and I would not argue against it, but it will bite on the first day and I would rather you choose that deliberately than discover it.
The workflow file in this working tree says dev/live-tunnel. The runs firing today come from staging-next. Your standing rule names staging-next as the branch serving the staging site. Two branches carrying different copies of the same pipeline is a live hazard -- it means any answer about how CI behaves depends on which tree you read it from.
Option A, and add a check that fails if the staging workflow's trigger branch list does not match the declared staging branch. But I will not act on this without your word, because deleting the wrong trigger silently unhooks a deploy path.
Staging is refusing to publish because the branch removes columns and a table that exist on the live staging database. That is the schema-safety system doing its job correctly. But it needs a decision, not a workaround.
Option A. Reset is the shortcut that cannot be taken on production, and staging should rehearse the production path, not a path that will not exist when it matters.
Production promotion is yours to initiate -- that is settled. The open question is whether the pipeline should refuse a promotion whose evidence is incomplete, or trust that you only promote when it is ready.
Option B. It does not take the decision away from you -- it takes the remembering away from you, which is where the misses come from. But it means production can tell you no, and you should agree to that before it happens rather than during an incident.
Nothing in the plan requires new infrastructure. It requires reordering one pipeline, finishing one cleanup, moving tests where you already ruled they belong, and pointing one dashboard at the two live sites so the answer to "is it live" stops being a guess.