The in-house deploy platform

What this is

LogriteMasterControl/ops/ is a deploy platform that replaces Coolify for the services that have been moved onto it. One pipeline, tracked end to end:

fetch from GitHub  →  build on the target host  →  push to that host's registry
                   →  pin the tag  →  docker compose up -d  →  verify

Every run is a job with a queue position, a streamed log, a duration split into waiting and running, a Discord message and a rollback target.

The control surface is management.logrite.ai (ops/management/), behind Cloudflare Access. The service binds 127.0.0.1 only; there is no path to it that does not go through the tunnel.

Note

This is now the deploy path for both estates. Dev runs no Coolify container at all (2026-09-23) and prod is 17 of 18 compose-owned, with jobsearch left behind deliberately — see below. Nothing was deleted on either host: the Coolify data directory, its database volumes and its images are all intact, so the rollback is a compose up, not a rebuild.

What has moved (dev)

All 20 dev services are compose-owned as of 2026-09-23. Coolify owns no Logrite service on dev — only its own coolify-sentinel and coolify-realtime, which belong to the Coolify installation itself.

ai-service · analyzer · backend · data-prepper · demo-seeder · devdocs · downtime-tracker · elasticsearch · frontend · ingest-shim · jobsearch · kibana · logrite-postgres · logrite-redis · mcp-server · opensearch · opensearch-dashboards · semantic-search · warden · webhook

Note

logrite-postgres and logrite-redis were invisible to the catalogue. Coolify keeps them in standalone_postgresqls / standalone_redis, not in applications — so a catalogue generated from the applications table did not list them, they had no is_auto_deploy_enabled row, and being absent from the catalogue they were also absent from DATA_PATHS. Both are in all three now. Postgres moved with its table counts checked either side (agent_db_access_log 103,073, usage_event 44,701, scan_run 1,604 — identical, with one new audit row from live traffic).

Coolify is gone from dev entirely as of 2026-09-23 — docker ps -a | grep -i coolify returns nothing, and dev-cicd.logrite.ai has no DNS record. Nothing was deleted: /data/coolify/, the coolify-db / coolify-redis volumes and every coollabsio/* image are intact.

Caution

Two things on dev are still named coolify and are load-bearing. The /data/coolify bind mounts (17 of them, across 9 services) and the coolify docker network that carries all 20 containers. Coolify’s own uninstaller does rm -rf /data/coolify — never run it. The name is not the owner.

What moved on prod, and what did not

17 of prod’s 18 services are compose-owned as of 2026-09-23 (evening). Promoted one at a time, each verified before the next was started: devdocs, ai-service, warden, ingest-shim, analyzer, semantic-search, mcp-server, webhook, frontend, backend, data-prepper — joining downtime-tracker — then the five stateful ones: logrite-redis, opensearch-dashboards, logrite-mongo, logrite-postgres, opensearch.

jobsearch is the one service still on Coolify, deliberately. Its 689 MB is throwaway embedmongo-db-<uuid> and its database has never survived a restart — there is nothing to preserve and nothing to gain by moving it.

Every takeover left the original container behind as …--preadopt-<stamp>, so each service rolls back without a rebuild. That is also why prod’s Coolify still runs and still holds :8000.

The stateful five were verified on their data, not their exit codes

An exit 0 is what a service that came up on an empty volume also reports.

service	evidence either side of the cutover
logrite-redis	same named volume re-attached (mounts compared against the --preadopt- container); PONG; genuinely empty — it is a cache, 89-byte dump.rdb
opensearch-dashboards	serving real traffic (app.logrite.ai → /api/alerting/alerts 200); local probe 401 = up + auth
logrite-mongo	identical volumes both sides, ping ok:1, 495 MB on disk, auth enforced — i.e. it read its own user store off that volume
logrite-postgres	exact row counts identical: integrity_manifest 1218, analyzer_rules_config 203, user_sessions 159, 85 tables
opensearch	877 indexes, 894 primaries identical; docs 19,572,272 → 19,572,923 (+651 = live ingest, not loss)

Caution

A prod container is not named for its service — it keeps the Coolify uuid. towogues614evh8hvznceoiy is postgres, o4430i5zw4waytcvllrkco9a is redis. docker ps | grep logrite-redis returns nothing for a service that is running perfectly, which reads as “the deploy destroyed it”. Match on the uuid, or ask GET /api/services.

Caution

Restart-window errors are not failures, and the window is the only way to tell. Recreating postgres logged 10 Hikari “Failed to validate connection” in a single 7 ms burst at the swap, then nothing — the pool evicting stale connections, which is correct behaviour. Always grep with a window (--since 60s): a count taken over ten minutes cannot distinguish recovered from still broken.

Caution

opensearch goes red for ~170s on prod and that is recovery, not a failure — it reopens 894 shards, climbing 30 → 894 primaries before reaching yellow. yellow is the correct resting state for a single node (559 unassigned replicas). Waiting for green waits for ever.

Prod still has no automatic deploy path, deliberately

This is unchanged by the migration and is not an oversight:

• Coolify’s per-app auto-deploy is false for every app it still holds.
• All three docker-deploy GitHub Actions workflows are disabled_manually.
• Prod’s watcher carries an empty auto_deploy list — it announces and never deploys.
• Prod actions are refused to any caller without a proven identity, so there is no scripted path. A localhost API call is seen as local and rejected.

Prod moves when a human promotes it from the console. And promote is the path, not deploy — prod has no SSH key at all, so a branch deploy fails Permission denied (publickey). That is not a fault to fix: promotion copies the exact image tested on dev into prod’s registry, so prod runs the bits that were tested rather than a rebuild of the same commit. Giving prod a deploy key would silently turn promotion into a rebuild.

Danger

That rationale has a boundary, and crossing it broke production. It holds only for a service configured at runtime. For one configured at build time it inverts — “the exact bits tested on dev” and “correctly configured for prod” become mutually exclusive, because build args resolve from the build host’s compose environment. A promoted frontend shipped dev’s VITE_* baked into the bundle, so prod’s “Sign in with GitHub” sent real users to dev-leapfrogapi.logrite.ai and all 29 /analytics/* routes read dev’s OpenSearch. Nothing failed: healthy container, 30/30 routes 200, green deploy, satisfied data guard. A correct build of the right commit, aimed at the wrong estate. frontend is the only affected service — measured — and it is now marked environment_flavoured, which is a standing claim about the service rather than a label on an image that may predate the check. Cross-estate builds use build(..., for_env="prod"), and the estate reaches the tag (main-<sha>-prod) so two builds of one commit cannot collide.

Ownership: four answers, not two

GET /api/services?host=<host> reports, per service, owner, container, the live state/status/image, and needs_takeover.

owner	what it means	a deploy will…
compose	this platform created the container	replace its image
coolify	a live container under Coolify’s label	take it over first
external	a container at the stack’s declared name carrying neither label	take it over first
none	nothing is running	simply create it
unknown	the read failed, or two containers claim the service	nothing — fix this first

Caution

coolify and none must never be collapsed. They differ by a live container serving production traffic. This is the same confusion that made adopt answer “nothing to do — a first deploy will simply create it” while Coolify’s container kept serving, and that gave the data guard an empty list of declared paths. A failed read is unknown, never none, and unknown never arms the takeover control — pre-arming it on a guess is how a live container gets renamed by accident.

This comes from one labelled docker ps, not from docker compose ps. compose ps filters on the compose project label, so on a host where another control plane owns the containers it returns nothing — and every field came back empty, which renders as blank cells that read as “nothing is deployed here” about eighteen live production containers.

Ownership is not a matter of opinion — ask the platform:

curl -s "http://127.0.0.1:8090/api/safety?service=analyzer&host=dev" \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["adoption"]["owner"])'
# compose

The three writable-layer services, resolved

These were held back because the data guard refused them. Two now keep their data somewhere a redeploy cannot reach, which was never true under Coolify.

Service	Was	Now
data-prepper	/usr/share/data-prepper/pipelines in the writable layer	bind → /var/lib/logrite-data/data-prepper/pipelines
webhook	/config in the writable layer	bind → /var/lib/logrite-data/webhook/config
jobsearch	702 MB, nothing declared it stateful	investigated — not data

Both binds were verified byte-for-byte against the live container (md5 on every file) before the cutover and again inside the new container afterwards.

Caution

jobsearch’s 702 MB is not data. It is eight abandoned embedded MongoDB databases. The app uses flapdoodle embedmongo, which picks a fresh /tmp/embedmongo-db-<uuid> on every boot — the directories are dated one per container start between 2026-08-26 and 2026-09-18, beside eight separate 52 MB extractions of the same mongod binary. Its database has never survived a restart, under Coolify or anywhere else. Nothing was moved onto a volume because there is nothing a volume could hold: the path is a new UUID every boot.

Services with no source at all

Not everything here is built from a repository. opensearch, its dashboards, elasticsearch and kibana are upstream images — Coolify calls that a dockerimage build pack — and jobsearch is a nixpacks build whose output is already a tag.

Deploying one of these does not build anything. The platform deploys the image its compose file already names:

[image] kibana has no source on dev; deploying the image its compose file
        already names: docker.elastic.co/kibana/kibana:9.5.2

The Fetch source, Build image and Push to registry steps report skipped, not pending, so the log does not imply work that never applied.

Caution

A branch cannot be deployed to one of these, and asking is an error rather than a silent no-op: there is no source to resolve dev against, so quietly ignoring it would deploy something other than what was asked for. The console disables the branch selector and says why.

Caution

An image reference with no / is never pulled. A Coolify nixpacks build leaves a tag like u136yx81y4x9l1jr350kl293:396e179f2558. Docker resolves that to docker.io/library/u136yx…, so a pull does not merely fail — it asks a public registry for a name this estate does not own.

Until 2026-09-22 this path did not exist: deploy called the builder unconditionally and answered “is a pre-built image, not something built from source”. Every word of that is true and the outcome was wrong — it reads like a misconfiguration to go and fix, and nothing is misconfigured. Four of dev’s services were unreachable through the platform for that reason alone.

opensearch, opensearch-dashboards and elasticsearch have now been moved too, and because they hold the estate’s log analytics data the move was verified on the data rather than on the deploy’s exit code:

	before	after
OpenSearch indexes	386	386
OpenSearch documents	48,741,623	48,742,458 (ingest continued)
OpenSearch primary shards	416 active, 42 unassigned, yellow	identical
Elasticsearch index dirs	66	66

Both keep their data on named volumes declared external: true in the generated stack, which is what makes this safe: compose attaches the existing volume rather than creating a project-prefixed empty one. Recovery took about 90 seconds for OpenSearch, during which the cluster reported red while it opened 416 shards — expected, and worth knowing before somebody reads it as a failed cutover.

The cutover recipe

Order matters. Each step exists because skipping it broke something.

1. Turn Coolify’s auto-deploy off for that app — FIRST

While it is on, a push makes Coolify recreate the container under the name compose has just taken, and the two fight over it.

-- on coolify-db. NOTE the table: this flag is NOT on `applications`,
-- which is what the API returns.
update application_settings s set is_auto_deploy_enabled = false
  from applications a where s.application_id = a.id and a.name = '<service>';

Do it for the WHOLE estate rather than per cutover. Seven apps were still true when the last eight services moved, including one that had been compose-owned for a day.

2. Deploy through the platform, allowing adoption

curl -sX POST http://127.0.0.1:8090/api/deploy \
  -H 'content-type: application/json' \
  -d '{"service":"<svc>","host":"dev","ref":"dev","allow_adopt":true}'

Adoption renames Coolify’s container to <name>--preadopt-<timestamp> and stops it. It is never removed: that container is the way back.

Omit ref for a service with no source — there is nothing to resolve a branch against, and asking is an error rather than a silent no-op.

3. Verify ownership

GET /api/safety?service=…&host=dev must report adoption.owner == "compose".

4. Only once everything is across: route pushes to the console

deploy_backend now defaults to "console", so a host that has never been hand-edited behaves correctly. Override it in /etc/logrite-ops/config.json only to go back to Coolify, and restart logrite-deploy-watcher.

Caution

The old default was "coolify", and once a service is compose-owned a Coolify deploy of it cannot succeed — the container name is taken. Measured: devdocs had four consecutive failed Coolify deployments from the watcher, each visible only in Coolify’s own application_deployment_queues table. Not a neutral fallback: a path that can only fail while the push looks deployed.

Build arguments

services.json records which build args each service needs. Values are never stored there — they are tokens and signing keys — and are resolved at build time from two places, in this order:

1. /opt/logrite-stack/build-args.json — operator-declared, for build-only variables.
2. The merged compose environment — the same environment the running container has.

Only args the Dockerfile DECLARES are required

build_arg_keys was harvested from Coolify’s env table, where a human ticks “build time”. Nothing checked that against the Dockerfile.

logrite-ai-service’s Dockerfile.ci declares no ARG at all, yet Coolify records OPENSEARCH_CA_CERT and OPENSEARCH_CA_PATH as build-time — so the guard refused every ai-service build over two variables that cannot reach the image. The builder now reads the Dockerfile’s ARG lines and requires only those. A declared arg with no value still fails the build, which is the case worth stopping.

Build-only variables need the overlay

Coolify flags is_buildtime and is_runtime independently. warden’s AI_SERVICE_IMAGE_TAG is build-only: it selects the base image in

ARG AI_SERVICE_IMAGE_BASE=ghcr.io/deployd-io/logrite-ai-service
ARG AI_SERVICE_IMAGE_TAG=latest
FROM ${AI_SERVICE_IMAGE_BASE}:${AI_SERVICE_IMAGE_TAG}

and is deliberately never injected into the container. Resolving it from the compose environment therefore looked in the one place it could not be.

The overlay can name the relationship instead of a literal tag:

{
  "services": {
    "warden": {
      "AI_SERVICE_IMAGE_BASE": "@pin-repo:ai-service",
      "AI_SERVICE_IMAGE_TAG":  "@pin-tag:ai-service"
    }
  }
}

@pin-repo:<svc> and @pin-tag:<svc> resolve at build time to the image that service is currently pinned to on this host, so warden always builds on the ai-service image the platform actually deployed. A literal tag would go stale on the next ai-service deploy, silently — and if ai-service is not pinned at all, the build is refused rather than guessing a base image.

Note

The default in ARG AI_SERVICE_IMAGE_TAG=latest is not a licence to proceed. Building warden from ai-service:latest is precisely the wrong-environment image the guard exists to prevent.

Building on one estate FOR another

Promotion ships the exact image that was tested on dev, which is right for a service configured at runtime. For one configured at build time it inverts: build_args() resolves values out of the build host’s compose environment, so a dev-built frontend carries dev’s VITE_* values no matter where it is later deployed. That is the 2026-09-23 incident, and it left prod’s frontend unpromotable — the guard was correct, and there was nothing it could let through.

for_env is the way forward. It builds on one host for another: the source and the build machinery are the build host’s, the configuration is the target’s.

curl -X POST http://127.0.0.1:8090/api/build \
  -H 'Content-Type: application/json' \
  -d '{"service":"frontend","host":"dev","ref":"main","for_env":"prod"}'

The target’s values live in a per-estate block of the same overlay:

{
  "environments": {
    "prod": {
      "frontend": {
        "VITE_API_BASE_URL": "https://leapfrogapi.deployd.io",
        "VITE_OPENSEARCH_API": "https://analyticsapi.logrite.ai"
      }
    }
  }
}

The per-estate block wins over the plain services block, so a value that genuinely is the same everywhere is still written once.

Caution

The build host’s environment is not a fallback here — it is the bug.

For an ordinary same-estate build, reading a missing value from the compose environment is correct. The moment the target differs it stops being a fallback and becomes a silent substitution: it produces a correct build of the right commit, configured for the wrong estate, which nothing downstream can detect. The containers are healthy, every route answers 200, the deploy is green and the data guard is satisfied.

So a for_env build refuses any declared arg the target’s overlay does not answer, and names it. A missing value must be a stopped build, never a borrowed one.

Two further consequences, both deliberate:

• The target estate reaches the image TAG, not just a label — main-<sha>-prod. Two builds of the same commit for different estates are different images whose tags would otherwise be identical, so the second push would silently replace the first in the registry.
• Which args are required is the TARGET’s statement, so a for_env build reads build_arg_keys from the target’s catalogue entry. The two lists differ in practice — dev’s frontend carries one key prod’s does not.

Install a fragment onto a build host with the merge tool rather than by hand; it merges key by key, previews every change, and never removes anything:

python3 ops/deploy/merge_build_args.py build-args.prod-frontend.json          # preview
python3 ops/deploy/merge_build_args.py build-args.prod-frontend.json --write  # apply

Hand-editing /opt/logrite-stack/build-args.json risks dropping the @pin-repo: / @pin-tag: entries above, and that failure is not loud: the base image resolves from the compose environment instead, to a tag the previous control plane kept re-pushing and nothing has pushed since.

Reading back what was actually baked in

Every check around the incident asked about the container — healthy, 200, green. None could ask what was compiled into the JavaScript, because Vite inlines VITE_* at build time and the result is string literals in a minified file. A frontend pointed at the wrong estate is not a degraded service; it is a perfectly healthy service wired to someone else’s backend.

python3 ops/deploy/verify_bundle_estate.py --url https://app.logrite.ai --estate prod
python3 ops/deploy/verify_bundle_estate.py --image <ref> --host dev --estate prod

It reads the literals back out and classifies every hostname, with three distinct exit codes because the fixes differ: 2 a hostname belongs to another estate (wrong image), 1 a hostname does not resolve (right image, stale value), 0 clean.

Note

Run against dev-app.logrite.ai as though it were prod, it reports FOREIGN dev-leapfrogapi.logrite.ai — the exact hostname that sent real users to the dev backend. That is the check the estate did not have.

It also resolves every name it finds, because an unresolvable hostname is its own failure and is indistinguishable on a status page from a service that is down: curl returns 000 in about a millisecond either way. Measured 2026-09-24, prod’s live bundle bakes two names with no DNS record — opensearch.logrite.ai (its Analytics Dashboards iframe) and api.logrite.ai.

Deployment steps and progress

The console shows a deploy as an ordered step list with a percentage and a remaining time.

Step	Entered when the log emits
Queued	[queued]
Resolve commit	[started]
Fetch source	[git]
Build image	BuildKit’s own #N … progress lines
Push to registry	[push]
Take over the container	[adopt] / [unadopt]
Data-safety check	[safety]
Move data onto the host	[stage]
Pull image	[pull]
Pin the tag	[pin]
Recreate container	[up]
Done	[succeeded] / [failed] / [cancelled]

The step labels are the console’s, not Coolify’s: it says “take over the container” rather than naming the product it replaced, and a test fails on any user-visible string that mentions it.

The steps are derived from the job’s own log, not reported separately by the handlers. A second state machine can disagree with the log, and when it does the operator has two stories and no way to choose which is true.

Progress is forward-only — the markers are not perfectly ordered (a late [state] line follows [up]) and a bar that goes backwards reads as a bug in the deploy rather than in the display.

The percentage names its own basis:

• history — elapsed against the median of this service’s last five successful runs of this kind on this host. Median, not mean: one cold-cache build would poison a mean for days.
• steps — how far down the list the job has reached. Drawn as a hatched bar and labelled as carrying no timing estimate, so it cannot pass for a measurement.

It never reaches 100% while a job is still running, and never states a remaining time it cannot know.

A deploy watches the container before it says “succeeded”

docker compose up -d returns the moment the container starts, so a container that dies on its first line looked exactly like a healthy one. Since 2026-10-10 the driver watches the recreated container for 20 seconds ([settle] in the log; LOGRITE_DEPLOY_SETTLE_SECONDS overrides) and fails the deploy if it is restarting, exited or dead, if its restart count grows while watched, or if its healthcheck reports unhealthy. A state it cannot read is not a crash: the watch stops and says so.

When it fails, the new image is already running and pinned — the message says so. Old-tag pruning is skipped, so the previous image is still in the registry for a rollback. On its first day it caught a dev webhook image missing a module (MODULE_NOT_FOUND, previously a green deploy over a 35-minute crash loop) and a prod webhook that could not write its config into a root-owned bind (EACCES).

Prod’s healthchecks come from /opt/logrite-stack/docker-compose.healthchecks.yml (source ops/deploy/docker-compose.healthchecks.yml), loaded the same way as docker-compose.health.yml. The dev file must not be copied to prod: it names the dev-only Elasticsearch lab services, and compose then refuses every command.

Two traps that bite on every cutover

container_name in the generated stack is Coolify’s timestamped name

<uuid>-<deploy-epoch> goes stale the moment Coolify redeploys that service. A stale one used to make adopt answer “nothing to do — a first deploy will simply create it”: a false all-clear while Coolify’s container kept serving. compose_driver.container_name now verifies the name exists, falls back to the Coolify label, then to the uuid half.

adopt preserves the container, not the data LOCATION

The platform binds /var/lib/logrite-data/<svc>/…; Coolify binds /data/coolify/applications/<uuid>/…. Merge the data across at cutover, or the service comes up on an older copy without saying so.

$ in a copied value is data, not a template

coolify_to_compose.py generates the stack by reading running containers, so every value in it is already final.

Compose interpolates ${VAR} in the YAML itself, before docker sees any of it, against the environment docker compose was run in — not the container’s.

The analyzer’s image declares:

HEALTHCHECK CMD curl -f "http://localhost:${PORT:-8080}/health" || exit 1

Docker keeps that verbatim and expands it inside the container, where PORT=9001. Copied into the stack unescaped, compose resolved it at config time — PORT is not set in the deploy user’s shell — and baked http://localhost:8080/health into the container. The analyzer listens on 9001 and answers healthy there, so the probe could never pass: 53 consecutive failures against a service that was serving fine, with nothing anywhere saying the port had been rewritten.

yaml_scalar() now escapes $ to $$. It is fixed in the one function every copied value passes through, because command and entrypoint carry exactly the same exposure.

The generator now also refuses to write a stack containing anything compose would still interpolate, naming every line it found:

REFUSING TO WRITE: 1 value(s) compose would still interpolate:
  line 412: test: "curl -f http://localhost:${PORT:-8080}/health"
Every value here is already final, so each of these is a copied value about to
be rewritten.

--allow-interpolation is the escape hatch, named so that using it is a decision somebody made.

Note

The escape is the fix; the refusal is the proof. The escape lives in one function precisely because command, entrypoint, healthcheck and environment all carry the same exposure — so one new line that forgets to call it reopens the hole silently, and the symptom is not a broken file. It is a service that comes up configured differently from the one that was copied.

Both dev stack files scan clean. Prod’s stack has never been generated, so the check will run the first time it is — which is why it lives in the generator rather than in a checklist.

Which commit a deploy actually builds

A deploy is given a ref (main) and, from the watcher, the commit that ref pointed at when the job was queued. Two things resolve between that and the image, and on 2026-09-23 both were wrong in the same direction: they shipped older code and reported success.

git ls-remote <repo> main is a PATTERN, not a name

It matches every ref whose last slash-separated component is main. The pre-squash archive created that day gave eight of nine repos a second one, and it sorts first:

3313cc6b0d2f  refs/heads/archive/pre-squash-20260923/main   ← taken
43c579b203eb  refs/heads/main                               ← wanted

resolve_commit() took the first parseable line, so every ref: "main" deploy resolved to the archive snapshot.

Nothing errors along the way, which is why it went unseen. The sha is real, the checkout is clean, the image is tagged main-<that sha>, compose recreates the container on the image it is already running, and the job records:

{ "ok": true, "requested": "…/ai-service:main-3313cc6b0d2f", "changed": false }

Discord says ✅ and the ledger says the commit shipped. Measured three times — ai-service twice, devdocs once. And it is permanent, not transient: archived branches are kept deliberately, so the ambiguity never ages out.

The exact ref now wins — refs/heads/<ref>, then a dereferenced tag, then the tag object, then the ref as spelled. A single match passes through. Two inexact matches refuse and name both: “which main did you mean” has no safe default, and guessing is what produced this.

A deploy that follows a prebuild is pinned to the prebuilt commit (F-31)

The backend’s dev image is a retag of whatever the watcher’s prebuild last published, so an image can only truthfully carry the commit that prebuild compiled. With the branch as the ref, a push landing during the ~9-minute prebuild made the console follow the branch and stamp the newer sha on the older artifact. builder.verify_built_source() refuses that (StaleArtifactError) — correct, but it surfaced as a red job nobody could act on.

Since 2026-10-01 the watcher triggers a post-prebuild deploy with ref = the prebuilt sha (_reap_prebuilds → _trigger(…, pin_commit=True)). freshen_commit() leaves an exact sha alone, so the deploy ships exactly what was built, and the newer push is picked up on the next tick as its own prebuild. Deploys without a prebuild still send the branch. Pinned by ops/notify/test_trigger_target.py, which drives _reap_prebuilds itself rather than the trigger with the flag set by hand.

A tip that is BEHIND the queued commit is a stale read

A queued deploy follows its branch rather than the tip it was queued with — one worker per host serialises the queue, and a job can wait minutes behind its own siblings. freshen_commit() implemented that by comparing the two shas for inequality, which holds because a branch only moves forward — until the read itself is stale. The same job’s log carried both halves:

[git] main has moved since this deploy was queued: 43c579b203eb -> 3313cc6b0d2f
...
3313cc6..43c579b  main -> origin/main

That reads exactly like a push landing mid-job. It was not: re-running it returned the identical answer, and a race that reproduces exactly is not a race.

tip_is_behind() now asks the one question a squashed history cannot fake — is the sha I just read a parent of the sha I was asked to build. Three answers, not two: unknown (no clone, an object neither side has, a host that will not answer) keeps the queued commit as well, because none of those are evidence the tip is newer. A genuine move forward is still followed, so the base-image mismatch this guard exists for is not traded away for the new failure.

Reading a green deploy

When a deploy succeeds and the estate did not change, compare result.requested against params.commit before believing either. changed: false on a deploy that was asked for a new commit is the signal.

Which commit a service is running

The section above is about what a deploy builds. This one is about what a container can tell you afterwards, which is a different question and had a different answer for two days.

Every image is labelled at build time:

docker inspect --format '{{index .Config.Labels "ai.logrite.commit"}}' <container>

Since 2026-09-25 the deploy also stamps the commit into the container’s environment, because that is where services already look:

# docker-compose.images.yml — generated, rewritten on every deploy
services:
  analyzer:
    image: "127.0.0.1:5000/logrite/analyzer:main-570856e0eb32"
    environment:
      SOURCE_COMMIT: "570856e0eb32bb8efb3ee1eebe1e3e2a0867c3a5"
      SOURCE_BRANCH: "main"

Danger

SOURCE_COMMIT was Coolify’s job, and for two days nothing took it over. Coolify injected it per deploy; the analyzer’s /health prefers it over its own build stamp, with a comment recording that Coolify is “the only place the truth exists”. Coolify left on 2026-09-23.

The value did not go missing — it froze, which is worse. The base stack is generated by reading running containers, so Coolify’s last value was captured as a literal and re-applied on every deploy since. The dev analyzer reported one commit through three image changes in a single day, and a perfectly good deploy was written off as a failure on the strength of it.

An absent SOURCE_COMMIT would have made /health say “unknown” and misled nobody. A frozen one is a real sha from a real deploy, so the wrong answer looks exactly like the right one. That is why an unknown commit now overwrites with the literal string unknown rather than being omitted — omitting the key lets the frozen literal show through, which is the bug.

Caution

A --build-arg is not an environment variable, and the grep makes it look like one. builder.py passes --build-arg SOURCE_COMMIT=<commit> on every build, so searching the ops tree finds a hit and the plumbing looks done. It is not: a build arg reaches the image only if the Dockerfile declares the matching ARG, and it never reaches the container environment at all. The analyzer also builds from a source tarball with no .git, so its git rev-parse fallback cannot work either — which is why build-info.json says commit: "unknown". One half was wired, the other was not, and the wired half is the half that shows up in a search.

Each service picks the stamp up on its next deploy. Containers running from before the fix keep the frozen value — nothing is silently rewritten underneath a running service. Until then the trustworthy source is the image tag:

curl -s http://127.0.0.1:8090/api/services | \
  python3 -c 'import sys,json;[print(s["service"], s.get("image")) for s in json.load(sys.stdin)["services"]]'

Note

COOLIFY_BRANCH in a running container is dead, not broken. It names a branch the main-only consolidation deleted, and it is frozen by the same mechanism. The generator no longer captures it, so it disappears when the base stack is next regenerated. SOURCE_BRANCH is the replacement that is actually maintained.

What is prod behind on, and how does it move?

/api/drift compares the commit each service runs on dev against the one it runs on prod, per service, and says whether the gap is promotable:

curl -s http://127.0.0.1:8090/api/drift | python3 -c '
import sys, json
for r in json.load(sys.stdin)["services"]:
    if not r.get("in_pipeline"):
        continue
    dev = (r.get("dev") or {}).get("commit")
    prod = (r.get("prod") or {}).get("commit")
    print("%-20s %-14s %-14s %-10s %s" % (
        r["service"], dev, prod, r["relation"], r["promotable"]))'

relation is one of same, behind (with ahead_by), diverged, or unknown. A row that is not promotable carries blocked saying why in words — prod’s stack does not declare the service, or dev is not running an image this pipeline built (the stateful five and the upstream images), or prod still runs it under the previous control plane (jobsearch).

Promoting is a button on that row in the console. It copies the exact image already tested on dev into prod’s registry over an ssh tunnel and deploys that — prod runs the bits that were tested, not a rebuild of the same commit.

Deployable only, and promoting several at once (since 2026-09-28). The Dev → prod page opens with Deployable only ticked. It hides every row production cannot be given an image for right now. A row is shown only if it is promotable and its dev container is running. That leaves out services prod does not declare (demo-seeder, the Elasticsearch lab), the stateful and upstream-image services, and jobsearch. The hidden rows are counted in a hint with a Show them button, so nothing disappears silently. Untick the filter to see every row.

Each row has a checkbox. It is armed exactly when that row’s own promote button would be: promotable, carrying a promote_image, and not already on the same commit. The header checkbox selects every armed row. Two buttons sit above the table:

• Deploy selected (n) promotes the ticked rows.
• Deploy all (n) promotes every armed row that is visible, except rows where production is ahead. Promoting one of those rolls production back, so tick it and use Deploy selected when that is what you mean.

Both buttons open one confirmation that lists each service and its drift. Each service is then queued as its own ordinary POST /api/promote job. Prod’s job worker runs one job at a time per host, so a bulk promote is N promotions shipped in order. There is no separate bulk code path on the server. A failed enqueue is reported per service, and the rest still go.

Caution

Bulk promote does not fix frontend: its job is still refused. If frontend is behind, it counts as armed and Deploy all includes it. Its job then fails with the refusal described below, and the other services promote normally. Untick it, or promote the …-prod build separately.

Caution

For frontend, do not use the row’s button — it can only fail. It always sends the image dev is running, and dev’s frontend image has dev’s VITE_* values baked in, so the promote is refused with “this image was built for ‘dev’” (correctly: see “Building on one estate FOR another” above). Build with for_env: prod, then use Deploy → Promote to production, pick frontend and choose the …-prod tag in the Image field. The field is searchable since 2026-09-28: type -prod to filter to the prod builds. The failed job’s dialog now offers exactly that as buttons (see below).

Why did a deploy fail, and how do I fix it?

Open the failed job, from the Deployments ledger or from the toast. Since 2026-09-28 its dialog shows one panel for any failed deploy, build, promote or restart, on dev or prod:

• Why it happened gives the cause in plain words, not the guard’s raw message.
• How to fix it gives the steps in order, plus the platform’s own advice when the error contains some.
• One-click actions start the fix as an ordinary job. Each one goes through the same confirmation as every other button, and a prod action sends DEPLOY-TO-PRODUCTION.
• Evidence lists the error lines that prove the diagnosis. For a docker build these are the failing step’s own output, found inside BuildKit’s #N step. Chatter that merely contains the word “error” is left out.
• Copy (panel header) copies the whole diagnosis as plain text: title, job, location, why, the fix steps, links and evidence. It is ready to paste into a ticket or a chat. Copy evidence copies just the evidence lines.

The panel comes from GET /api/deployment?id=<job>. For a failed job that response carries a diagnosis object, built by ops/management/diagnose.py. It is computed each time the job is opened, never stored, because the right fix depends on live state. When a frontend promote is refused because the image has dev’s VITE_* values baked in, the panel checks dev’s registry:

• If a …-prod build of that commit exists, it offers Promote … to prod.
• If not, it offers Build <sha> for prod. Run that, reopen the same failed job, and the panel now offers the promote.

No rule names a service. Each rule matches the shape of an error the platform emits, taken from the 49 failed jobs in the ledger on 2026-09-28. The service, host, commit, image and missing values all come from the failed job’s own text.

Rule	Recognised by	What it offers
foreign-build	promote refused: image built for another environment	promote the …-prod image, or build one
confirm-token	production confirmation missing	run it again with the token
git-access	Permission denied (publickey) resolving a ref	build on dev and promote; or add a deploy key
base-pinned / base-guard	the base image (ai-service) is not this commit’s build	deploy the base at that commit, then retry
missing-build-args	build args with no value in the compose env	where to add them, then retry
writable-layer	the data guard refused a recreate	inspect with docker diff; retry with the guard overridden
registry-push	pushing to 127.0.0.1:5000 failed	check the registry and disk, then retry
interrupted	the console restarted mid-job	check the running image, then run it again
no-new-privileges / read-only-fs	the console’s systemd sandbox blocked a step	the unit setting to check, then retry
network	unreachable network, DNS failure or timeout during a pull	retry
source-error	a build step failed at a file and line in our code (mdx/vite/astro, tsc, javac/maven, python, webpack, go, …)	Fix it in GitHub, which opens that file at that line in GitHub’s editor, plus View the line at the commit that failed. No retry
build-step	a Dockerfile RUN failed with no file and line to point at	a gate (lint/test/tsc): fix the code, no retry offered; otherwise retry without cache
generic	anything else	the failed step, the log’s error lines, the error’s own advice, retry

Fix it in GitHub takes the repo and commit from the build log’s own [build] <svc> on <host>: <repo>@<sha> line. The file path the tool printed is a path inside the container (/app/src/…). The console checks every suffix of it with git cat-file -e <sha>:<path> in the builder’s checkout (/var/lib/logrite-deploy/src/<repo>), so the link points at a file that exists. Without a checkout it strips the usual WORKDIR (/app) and says the path was not confirmed. Committing in GitHub’s editor pushes to main, and that push deploys dev on its own, so fixing a broken build does not need a second trip to the console.

A retry re-sends the job’s own request pinned to the commit it recorded, not to its branch. A branch moves, and dev has been deleted, so “the same job” has to mean the same commit.

Caution

A diagnosis for an old job is about that job. The fix for a refused frontend promote offers a prod build of that job’s commit. If dev has moved on since, promoting it moves production only partway. Check Dev → prod for the commit dev runs now.

Every confirmation in the console — deploy, promote, take over, reclaim — is the console’s own themed dialog (since 2026-09-28), not the browser’s confirm() box. OK runs the action; Cancel, Esc or a click outside runs nothing. The separate typed DEPLOY-TO-PRODUCTION dialog in a service’s own Deploy tab is unchanged.

Danger

Prod refuses any caller without a proven identity, and that is the control, not a bug. A loopback curl on the box is seen as local and answered 403 — “production requires a signed-in operator”. Being on the host is exactly the thing an attacker gets first, so it buys nothing here; dev stays reachable that way on purpose.

What satisfies the gate is a name: a console session from the login form, a Cloudflare Access assertion, or an operator token (which buys a name, and nothing else — it does not widen the loopback case). So a promote is a human action by design, and automation cannot quietly acquire the ability to move production.

Caution

Nothing from ops/ runs on the prod host, and assuming otherwise stalls work. pins.render() executes in the console on dev; pins.write() hands the YAML to hostio.write_file(), which base64s it across the ssh hop and mvs it into place. Every other remote call is docker or sh -c.

This was written down backwards once — “install the fixed ops/ on prod first” — and the cost was not a wrong command but a wrong shape: it put an imaginary prerequisite in front of the only real action, so a one-click item read as blocked on access nobody had.

Prod’s base stack does still carry the frozen SOURCE_COMMIT literals, and does not need regenerating: docker-compose.images.yml is last in the -f order and the overlay overwrites them. That is precisely why render() stamps "unknown" rather than omitting the key.

Note

Check the result with /health or /api/drift — never with /api/services. That endpoint has no commit key at all, on either host: dev answers null for all thirty of its services while their /health values are demonstrably correct. It was once cited as the way to confirm a prod promote had landed, which would have read as “the promote did not work” no matter what happened. A check that cannot distinguish fixed from broken is worse than no check, because someone will act on it.

Is every service running the newest code?

python3 /opt/logrite-management/ops/deploy/freshness.py    # exit 1 if anything is behind

One row per service: the commit it is pinned at, the newest commit that can affect its image, what that tip was scoped by, and a verdict.

Caution

“The tip of the repo” is the wrong question for a service that builds from a subdirectory, and for one that shares a repo.

data-prepper builds from /data-prepper inside LogriteMasterControl, so every commit anywhere in that monorepo — a backlog edit, a docs gitlink bump — moved the tip it was compared against. It reported *** BEHIND *** permanently and could only read LATEST in the instant after a deploy. The tip is scoped by the service’s base_dir now, which is why the check reads the builder’s source cache rather than git ls-remote: a path-scoped tip needs a clone.

Scoping alone was still not enough. data-prepper deploys at a repo-wide commit, so what it runs is routinely newer than the last commit to touch its own directory — an equality test calls that “behind” too. Verdicts are ancestry-based, and tri-state: the daily history squash orphans merge parents, so “git cannot say” is reported as unknown, never as the alarming answer.

no pinned commit is honest and expected for jobsearch (nixpacks, no repo this platform builds from). A repo-wide BEHIND on one of the four services sharing logrite-ai-service.git names how many share it, because a repo tip cannot attribute the change to one of them.

One commit per day

main in every service repo carries exactly one commit per calendar day, back to 2026-07-01. ops/backup/daysquash.py is what maintains it:

python3 ops/backup/daysquash.py <repo> --from 2026-09-24 --to 2026-09-24 \
    --emit-plan plan.json                 # what each day contains
python3 ops/backup/daysquash.py <repo> --from 2026-09-24 --to 2026-09-24 \
    --messages msg.json                   # dry run — verifies, writes nothing
python3 ops/backup/daysquash.py <repo> … --messages msg.json --apply --push

It walks first-parent, not topological order. These repos carry real merges, and a topological linearisation interleaves days — a side branch started on the 3rd and merged on the 11th puts 3rd-of-the-month commits after 11th-of-the-month ones, so “the last commit of day D” stops meaning “what main looked like at the end of day D”. The first-parent walk is the mainline and is monotonic by construction.

Each new commit reuses that day’s final tree byte for byte. No merge is recomputed and no patch is re-applied, so nothing can conflict, and the tip tree is identical to the tip tree before. verify() asserts exactly that and refuses to move the ref otherwise — a build reads the tree, not the history.

Nothing is discarded: every original commit stays reachable from an attic tag pushed before the branch moves, so the rewrite is only ever reversible.

The attic tag must not end in main

git ls-remote <repo> main is a glob (above). The tag this tool writes ends in pre-daysquash deliberately, and check_ref_names() refuses to run at all if the tag it was given would tail-match main. Pre-existing ones warn rather than refuse — eight repos still carry refs/archive/pre-squash-20260923/main on purpose, and refusing on those would block every repo.

Ancestry is the wrong test here

Because history is rewritten daily, git merge-base --is-ancestor reports drift that does not exist. Compare trees. That applies to MasterControl’s submodule gitlinks too: measured 2026-09-24, ten of eleven were behind, but five were behind only in the sense that they named a sha a rewrite had replaced — same tree, so bumping them changed nothing at all.

Caution

--force-with-lease=refs/heads/main:<sha> is only a lease if <sha> is the REMOTE’s. The tool passed the local tip. One unpushed commit is enough to break it: the lease names a sha the remote has never held, git refuses with stale info, and by then update-ref has already moved main and the attic tag has already been written. The rewrite is half-applied and the error message is about something else. The remote is now read before anything is written, and only when --push is given — a local-only rewrite has no remote to lease against.

Caution

The invariant is not self-maintaining. Squash at the end of a day, or re-run. On 2026-09-24 a sixth commit landed on logrite-ai-service after its day had already been collapsed, so the day held two again and MasterControl’s gitlink pointed at the intermediate. Both are cheap to correct — the tree is preserved either way — but only if someone looks.

Data safety

ops/deploy/safety.py refuses to recreate a container whose data exists only in its writable layer, and fails closed: an unresolvable path, an unmeasurable layer and an unknown container are all treated as unsafe.

It resolves a declared path through an in-container symlink before calling it uncovered, because a guard that cries wolf on a safe deploy is one an operator learns to override.

Caution

os.replace() does not follow a symlink — it replaces it. That is how the downtime tracker’s notifications.json detached itself from its volume and lost its history on every redeploy while always looking present and readable. Any bind-mounted file has the same exposure. Write in place.

Data that exists only in the layer is moved BY the deploy

A guard that refuses is not a migration. Copying the data out by hand and then deploying has a race nobody can close from outside the deploy:

copy the data  ->  ...the service keeps writing...  ->  deploy

Everything written in that gap exists only in the old container’s layer, and the recreate destroys it. On prod’s OpenSearch that gap runs at about 4.6 documents a second.

deploy() contains exactly one instant with no writer:

adopt()        the old container is renamed and STOPPED
>>> here <<<   nothing is running; the layer is frozen and consistent
compose up     the new container is created with the bind mounted

ops/deploy/staging.py copies there, and nowhere else. It appears as a [stage] step in the job’s own log, so it is something an operator watches rather than a procedure they follow.

• It copies only into an empty destination. A destination that already holds data belongs to a service that has already migrated, and the source in that case is a frozen --preadopt- shadow — copying it over today’s data is a restore-from-old-backup wearing the clothes of a deploy.
• A failure aborts the deploy. The alternative is a container that starts perfectly and is empty: a green deploy, a broken service, and a cause three layers from the symptom. An abort leaves the previous container stopped-but-intact, and unadopt() restores it.
• The copy is verified by manifest — size and relative path for every regular file on both sides, compared as sets so a mismatch names the files.

Caution

docker cp does not preserve ownership. Every file is 1000:1000 inside the OpenSearch image; all 19,338 of them landed root:root on the host. OpenSearch runs as uid 1000, so that cutover would have produced a container that starts cleanly and cannot write one byte to its own data directory.

The copy is a tar stream extracted with --numeric-owner --same-owner -p. --numeric-owner is not optional: without it tar maps through the host’s /etc/passwd, where uid 1000 is the deploy account — a different identity that happens to share a number, which is exactly the kind of mistake that looks correct for the services that collide and breaks the ones that do not.

Caution

A declared data path can be a FILE. downtime-tracker declares /app/notifications.json. mkdir -p on a file path creates a directory, and the app then fails to read its own notifications with EISDIR. Staging probes what the path IS — through the tar stream, because docker exec needs a running container and this one is deliberately stopped — and creates the parent for a file.

Every command here is a pipeline out of docker cp, so they all run under set -o pipefail. A broken docker cp produces an empty stream, tar reports nothing, and the listing comes back empty — which reads identically to “this directory has no files” and leads straight to “nothing to copy”. That is a silent skip of the one step protecting the data, so an unreadable source raises instead.

“Nowhere to put it” and “not there yet” are different verdicts

assess() measures the container that EXISTS, which is right: recreating today’s container really would destroy what is in its writable layer. But when the stack already declares a bind for the uncovered path, the report says so, and says the deploy will copy the data there before the replacement starts.

The verdict stays safe: false — the bind is not what makes it safe, the deploy’s own adopt-and-stage sequence is. What changes is that the operator is no longer shown a flat refusal for a service the deploy is about to migrate, with the guard override as the only visible way forward. That is how an operator learns to override a guard that is telling the truth.

A guard that was never asked about two stateful services

DATA_PATHS encodes what the platform KNOWS is stateful. elasticsearch and kibana were not in it. Both are stateful, so the only thing between them and a recreate was the writable-layer measurement — a real signal, but it says “nothing has been written to the layer yet”, not “this service is safe to recreate”. The two signals cover different blind spots and neither is sufficient alone.

elasticsearch is declared now (/usr/share/elasticsearch/data and its snapshots bind), and the guard confirms both paths are covered by a volume. When adding a service to this platform, check it against DATA_PATHS rather than assuming the measurement will catch it.

When the refusal is about bytes that are NOT data

The other half of the guard is the writable-layer measurement, and it fires on bytes nothing declared. On backend those bytes are toolchain caches: it restores NuGet, npm and Maven packages while it instruments a customer repo, and they land in its layer.

That is a real refusal and the guard is right to make it — a recreate does destroy them — but the fix is not to override it and not to add them to DATA_PATHS. They are not data. The fix is to give them a bind, so the recreate destroys nothing and the measurement the guard takes is true rather than tolerated. They live in docker-compose.volumes.yml, apart from the declared data paths and labelled as caches.

Caution

The list of caches is discovered, never predicted — and it is never final. Backend was blocked by this twice. On 2026-09-23 it was 3.02 GB of NuGet and npm; those four paths were bound and the problem was declared solved. On 2026-09-25 it was 259 MB of Maven at /root/.m2, which had been absent from the list for one reason only: no Java repository had been scanned yet, so docker diff showed nothing there. An empty cache is indistinguishable from a path the service never uses. Expect the next toolchain to do the same.

Because it recurs, the refusal now names the bytes. It used to end “these are some other bytes — find out which before proceeding”, and finding out took a docker diff, a prefix-dedupe, a cross-check against the container’s mounts and a du, by hand, while the service sat undeployed:

275.0 MB sits in the writable layer and is destroyed by a recreate. Its
declared data paths ARE on volumes, so these are some other bytes — find out
which before proceeding. The largest unbound paths in that layer are
/root/.m2 (271.6 MB).

safety._layer_hotspots() produces that, and layer_hotspots carries the same list into the job record for the console. Mount-covered paths are excluded before measuring — their bytes are not in the writable layer at all, and du over backend’s 2.4 GB bound NuGet tree is slow enough to matter on a path somebody is waiting on. It runs only on the refusal path, and returns nothing at all on any failure: a diagnostic that could turn a refusal into an approval would be a guard that fails open.

Adding a bind is only half the fix

The bind tells the next container where to write. It does nothing about the bytes already in the current one, and the guard measures the current one — so a deploy attempted after only the first half is refused with exactly the same message, which reads as though the edit did not take.

Both halves, in this order:

1. Stage the bytes, then verify the count on both sides. A bind over a directory the image does not populate mounts empty, so an unstaged bind silently throws the cache away and the next scan re-downloads it.

   docker cp <container>:/root/.m2/. /var/lib/logrite-data/backend/maven/
   docker exec <container> sh -c 'find /root/.m2 -type f | wc -l'
   find /var/lib/logrite-data/backend/maven -type f | wc -l

2. Declare the bind in /opt/logrite-stack/docker-compose.volumes.yml (repo copy: ops/deploy/docker-compose.volumes.yml).

3. Remove the in-container copy, which is what the guard is measuring:

   docker exec <container> rm -rf /root/.m2

Only then does the guard fall below its 64 MB threshold and the deploy proceed.

Note

Step 3 looks like the dangerous one and is the safest: the bytes were verified onto the host in step 1, and the recreate the deploy is about to perform would destroy them anyway. After it, the bind mounts them straight back.

Afterwards, verify on the file counts, not the exit code:

docker exec <container> sh -c 'find /root/.m2 -type f | wc -l'
find /var/lib/logrite-data/backend/maven -type f | wc -l
docker ps -s --filter "name=^<container>$" --format '{{.Size}}'

Measured on the 2026-09-25 cutover: 2296 files on both sides, and the writable layer 275 MB → 123 kB.

Caution

A green deploy cannot tell you the bind worked. A container that came up with an empty /root/.m2 is healthy, runs the right image and reports the right commit — identical in every respect until the next scan silently re-downloads the cache. The equal file count is the only signal that separates them.

Note

Clearing the guard is not the same as resuming deploys. The watcher gives up after three attempts and nothing resets that counter but a new commit, so a service blocked long enough needs one deploy triggered by hand even after the cause is fixed.

The guard said “safe” for a reason that was not why

At first deploy the guard was answering the wrong question, and answering it confidently.

deploy() passed container_name(host, service) or service to the guard. container_name() only returns a name it has PROVEN is live, and straight after adopt() there is none — the Coolify container has been renamed to --preadopt-… and compose has not created its own yet. So the service name was substituted, docker found no container by that name, and assess() returned early:

{"container": "webhook", "declared_paths": [], "safe": true,
 "reasons": ["no container named 'webhook' on dev — nothing to destroy"]}

declared_paths is empty for a service whose DATA_PATHS entry is ("/config",): the early return skipped the loop entirely.

Nothing was lost, because adoption renames rather than removes and the data had been staged onto host binds by hand first. Had it not been, the report would have been identical — webhook would have come up with an empty /config and said nothing.

At first deploy the question is not “what does a recreate destroy” (nothing — there is no container). It is “will the replacement find its data?” The guard now answers that one:

• configured_name() — the name compose WOULD create, no existence check. A container that is really about to exist, unlike the service name.
• configured_mounts() — what compose WOULD mount, in docker inspect shape so one coverage rule serves both cases instead of two that can drift.
• assess(..., incoming=…) — on an absent container, DATA_PATHS is checked against those mounts. Uncovered means unsafe, and the reason says the new container starts EMPTY and will not mention it.

Caution

Three states, not two. incoming=None means “asked, and the stack could not be read” — unknown, therefore unsafe. A caller that never asked is a third state and must not inherit that verdict, or every existing call site starts refusing. Collapsing them either way is a bug: one direction blocks every deploy, the other restores the false all-clear.

/api/safety in the console reads the same way, so the page and the deploy path answer from one guard. Two readers that disagree is two stories and no way to choose.

Known limit: a path reachable only through an in-container symlink cannot be resolved before the container exists, so it reads as uncovered. That is failing closed, and the reason text says so — an operator about to override deserves to know which of the two cases they are looking at.

HEAD is answered, not refused

BaseHTTPRequestHandler has no do_HEAD, so every HEAD reached the default 501 — including HEAD /, which is what a plain uptime check sends. A console that reports 501 to its own monitor looks down while serving perfectly.

do_HEAD sets a flag and routes through do_GET: one route table, so a monitor and a browser can never disagree about whether a page exists. Content-Length is still the length the GET would have had, per RFC 9110 — a monitor that read it as 0 would conclude the page is empty.

Caution

http.client cannot see a HEAD body. It knows a HEAD has none and returns b"" from read() whether or not the server sent one, so body == b"" passes against a deliberately broken server. That assertion has to be made over a bare socket. Same lesson as ETag-versus-content-hash: the client is not the authority on what the server did.

Assets are versioned, so half a deploy is impossible

index.html and app.js must be the same vintage: the current app.js clears #svc-view-body, and one from before the services rewrite clears #services-rows, which the markup no longer has. A browser holding one from before a deploy and one from after gets clear(null) and a blank page — reported as a crash in code that is demonstrably correct on the server.

ETag + no-cache asks a browser to revalidate, and that is not enough: revalidation is a request, and a browser or an intermediary can decide not to make it. The console rewrites its own asset URLs to carry a hash of each file’s bytes:

<link rel="stylesheet" href="/style.css?v=0d0356603fa0">
<script src="/app.js?v=9ba6a157374a"></script>

A new app.js is a new URL, so no cache anywhere can answer it with the old bytes. A request naming the current version is served immutable; a version we do not recognise never is, or that browser could never be updated again.

What the console shows about a deployment

Every job carries its own record, opened from any row in the Deployments table:

• A timeline — Queued, Started, Finished, each with its wall-clock time, how long ago it was, and the gap that followed it (waited / ran / total).
• What was asked for beside what happened: the branch or image requested, whether the build cache was bypassed, whether ownership was taken over as part of the deploy, and whether the data guard was overridden.
• What resulted: commit, image deployed, image replaced, the container, and already on this image — nothing was recreated when a deploy was a no-op.
• The data guard’s own verdict, in the record of the deploy it guarded. It is written to the log and then scrolled past; this is where somebody looks a week later when asking what protected a service.
• The steps, derived from the job’s own log, and the full log itself.

Caution

The facts block was unreadable until 2026-09-23 and the data was never the problem. #job-facts was itself a <dl class="lr-kv"> — a two-column grid — and the helper that fills it returns its own <dl class="lr-kv">. The inner grid landed in the outer grid’s 190px label column, so its value column collapsed to nothing and every value wrapped one character per line. A test now asserts that nothing receiving that helper’s output is itself a .lr-kv.

The Deployments table is scoped by the host switcher

The host switcher in the sidebar scopes this table like every other section, and the filtering happens on the server, not in the browser.

Caution

That is not a style preference. /api/deployments applies limit after host, and on this estate the difference is the whole answer:

/api/deployments?limit=60            -> 60 jobs, ALL dev
/api/deployments?limit=60&host=prod  ->  1 job  (downtime-tracker)

Prod’s only deployment is nowhere near the top 60. Filtering the payload in the browser would have reported “nothing has been deployed to prod” about a host that has a deployment — turning a visible bug into an invisible one. Filter the DATA, not the payload.

Three consequences worth knowing when changing this page:

• The URL is a second door into the same state. applyRoute() writes state.hostFilter directly, so logic that lives only in the switcher’s handler leaves a bookmark, the back button and a shared #/deploys/prod link silently broken. Both doors call one hostFilterChanged().
• A payload for the previous host is dropped, not rendered. It is not a slow answer to the new question, it is an answer to a different one — and the next poll quietly correcting it looks like a flicker rather than the bug it is.
• A correct empty answer must not read as a broken filter. “Nothing has been deployed through this console yet” is true of the whole estate and false of a filtered view, so the empty state names the host it is about.

Resources: measured, or it says so

Three views over one measurement, so they cannot disagree:

Where	What it answers
Resources page	every service on every host, sorted
A host card on Overview (click, or Enter)	every service on that host, with the host’s own capacity
The Resources tab on a service	that service alone, and what share of the host it is

The numbers come from one docker stats --no-stream sample joined to docker ps -s and /proc — the same joined rows for all three.

Caution

A container docker did not sample reads “not measured”, never 0. A stopped container reports nothing, and a resources page whose whole purpose is saying where the memory went cannot afford to invent a zero — it would sort quietest and read as the thing to leave alone. The same rule governs sorting: the unsampled rows go last in both directions rather than bottoming out the numeric order.

A percentage with no denominator is not information — 180% CPU is alarming until you know the box has 12 cores — so every share names what it is a share of: cores, the container’s own memory limit, and the host total.

The “no volume” badge is measured now

It used to come from a hand-written tuple of five service names. It went stale the moment two of those services were given binds, and a stale warning is worse than none: it says a recreate destroys this about a service where it no longer does, and the next real warning is read the same way.

Each row now checks its own container’s mounts against the service’s declared data paths, parent-aware — a mount at /app/data covers /app/data/db. Three outcomes, three renderings:

• no volume — a declared path with nothing mounted there, named in the tooltip;
• mounts unknown — docker inspect did not answer, which is not a clean row;
• no badge — measured, every declared path covered.

Note

collector.py keeps a COPY of DATA_PATHS rather than importing it: that module is deliberately stdlib-only and loads on hosts with no deploy engine installed. test_resources.py fails if the copy and ops/deploy/safety.py ever disagree, which is the only thing that makes a copy safe.

Search data: OpenSearch’s cost is heap, not disk (2026-10-19)

The console’s Search data section has one card per host for its OpenSearch cluster. Disk was the only view of OpenSearch the console had, and the least useful one: the cost of these clusters is cluster state held in heap, which grows per shard whatever the shard holds.

Each card shows shards per GB of heap against the usual guideline of 25 (with the heap that would support the current shard count), heap use against the 85% at which ml-commons refuses embeddings, shards / indexes / segments, data size, how many indexes are empty or under 1,000 documents and how many shards they hold, the share of wall time spent in GC, and the ML circuit breaker’s trips. OpenSearch’s own host-memory figure is shown with a warning, because it counts page cache as used. Inspect N indexes opens a paged index list (filter by name and by empty / tiny / real; sort by size, shards, documents or name).

Route	Answers
GET /api/opensearch?host=dev|prod	the card’s figures; present: false when the host runs no OpenSearch
GET /api/opensearch/indices?host=&sort=&only=&contains=&limit=&offset=	one page of indexes; total, store_bytes and shards describe the whole filter, not the page

Both are read-only and read the cluster inside its container on each host (prod over the same SSH hop as every other prod read), so there is no published port and nothing extra to configure. Entering the section and Refresh both re-read every visible host. Measured at install, 2026-10-19: dev 182.5 shards/GB (730 shards, 4 GB heap), prod 368.5 (737 shards, 2 GB heap) — both “critical”, 7x and 15x the guideline.

The Archive: every report, with when it was created and updated

The console’s Archive section lists two shelves: Claude Artifacts (from ops/management/archive-artifacts.json) and the local HTML served off the dev VM (/opt/logrite-reports, /home/deployd/www).

Date filter. The same filter row as Deployments: a Range segment (Today, Yesterday, This week, This month, Custom, All), From / To date pickers, and Date (Updated or Created) to choose which time the window applies to. Picking a day in either box is the custom range. Periods are calendar periods in the viewer’s own zone; a week starts on Monday.

GET /api/archive?q=&field=updated|created&since=&until= — since and until are epoch seconds the browser computes from the viewer’s local midnight and 23:59:59, both inclusive. The server never resolves a bare date, because only the browser knows the viewer’s midnight. A bound that is not digits, a since after until, or an unknown field answers 400. A row with no value for the chosen field is left out of a window and counted in window.undated, and the page says how many.

Times on each row, as YYYY-MM-DD HH:mm:ss in the viewer’s zone:

	Created	Updated
Local HTML	the file’s birth time (statx btime on ext4)	file mtime
Artifact	not available: claude.ai exposes no creation time	publish time of the version being served, to the second

The artifact time comes from the version id a scope: "files" listing prints (1789736809-3c3a leads with its epoch). A manifest entry without updated_ts shows the date alone, never an invented 00:00:00. Created is never filled from mtime: a “created” equal to “updated” on every row would look right and be wrong.

How each shelf stays current

Local HTML is scanned live (cached 120 s), so a new file appears within two minutes.

Artifacts are kept current by a Claude Code hook, ops/management/archive_artifact_hook.py, registered as PostToolUse with matcher Artifact in the operator’s ~/.claude/settings.json (ops/management/install-archive-hook.sh adds or --removes it). After every Artifact call in any Claude Code session on the dev VM it updates the live manifest /var/lib/logrite-deploy/archive/archive-artifacts.json, which the console re-reads whenever it changes, with no restart or reinstall:

Call in a session	What the shelf learns
publish a page	that artifact, indexed from the very file published, timed by its version
list (complete)	the whole shelf: new ones added, deleted ones removed
list (a page, “Showing 5 of 44”)	upserts only; nothing is removed
list scope: "files" / read	the exact publish time of the version served
delete (no asset path)	that artifact removed

The merge (refresh_archive_artifacts.py) matches a row by any id it goes by (short id, old UUID, URL) or its exact title, keeps indexed text for an unchanged version, writes atomically under a lock, and refuses an empty listing or one under half of a shelf of four or more: a failed fetch, not mass deletion. The hook always exits 0 and prints nothing. Log: /var/lib/logrite-deploy/archive/hook.log. The card on the page says when the shelf last changed.

Caution

Why a hook and not a timer. The Artifact tool is off in claude -p and the Agent SDK, and claude.ai has no personal listing API, so no scheduled job on this VM can read the gallery. The limit that leaves: an artifact published outside a Claude Code session on this VM (the claude.ai web app, another machine) appears at the next full listing any session here makes. Sessions that were already open when the hook was registered keep the hooks they started with; new sessions get it. The shipped archive-artifacts.json is now only the seed for a host where no session has written yet.

Production is gated on a human

Prod deploys are refused to any caller without a proven identity. A localhost API call is seen as local and rejected — including from the deploy watcher, deliberately.

Use the console’s Deploy button while signed in. Promotion ships the exact image built and tested on dev rather than rebuilding the same commit: base images move and package registries serve different bytes, and the first time that matters is the time it breaks in production.

A caller that is not a browser: operator tokens

A login-form session and a Cloudflare Access assertion are both browser things. Everything else fell through to local, so a deploy driven by a script, a runbook or an agent had no way to say who it was — and the only route to a prod action was a human with a mouse. That is a gap, not a policy.

python3 /opt/logrite-management/server.py --mint-token release-bot --token-ttl-days 30
python3 /opt/logrite-management/server.py --list-tokens      # labels, never digests
python3 /opt/logrite-management/server.py --revoke-token release-bot

curl -X POST https://management.logrite.ai/api/promote \
  -H "X-Logrite-Operator-Token: <token>" \
  -d '{"service":"devdocs","image":"…","confirm":"DEPLOY-TO-PRODUCTION"}'

This does not widen the list of unproven identities; it adds a way to prove one. Two properties are what make it a credential rather than the same hole with ceremony:

• Only sha256(token) is stored, exactly as sessions are. Someone who reads operators.json holds a verifier, not a key, and cannot mint a request with it — which is what stops “root on the jump host” collapsing back into “may ship prod”.
• The identity is token:<label>, never an email. token:release-bot and victor@… are different actors and the audit log has to tell them apart afterwards. A token that logs as a person destroys the only record of who actually shipped. Labels are restricted to a-z 0-9 - _ so one can never be mistaken for an address.

Caution

Mint it with sudo and the file lands owned by root, while the console runs as deployd. The open then fails EACCES, every token silently stops working, and the caller is reported as local — which looks exactly like a token that was never minted. _load_operators() distinguishes absent (a legitimate empty answer) from unreadable (a misconfiguration) and says so on stderr. It still fails closed; it just fails closed out loud.

Operating it

# what is running, and who owns it
docker ps --filter label=com.docker.compose.project=logrite \
  --format '{{.Label "com.docker.compose.service"}}\t{{.Status}}'

# the platform's own view, both hosts
curl -s http://127.0.0.1:8090/api/overview | python3 -m json.tool | head -40

# a job's live log
curl -s "http://127.0.0.1:8090/api/deployment/log?id=<job-id>"

Install or update the console (idempotent; restarts the unit):

rsync -az ops/ dev:/tmp/logrite-ops-src/
ssh dev 'cd /tmp/logrite-ops-src/management && sudo ALLOW_ACTIONS=1 ./install.sh'