CI Runners

Overview

Every workflow in a private Deployd-io repo runs on a self-hosted GitHub Actions runner installed on the dev VM (deployd-dev). GitHub bills hosted runner minutes for private repositories only; self-hosted minutes are free and unmetered, and that is the whole reason these exist.

Measured on 2026-09-18, before the move: the org is on GitHub Free for organisations — 2,000 included Actions minutes a month — and stood at 1,803 / 2,000 with 13 days of the cycle left. Per-job wall-clock for that cycle:

repo	workflow	minutes	billable
Logrite-UI	checks	880	yes — private
logrite-ai-service	checks	620	yes — private
logrite-analyzer	Analyzer CI	71	yes — private
JobSearch	LogRite Gatekeeper	38	no — public repo
LogriteMasterControl	ghcr-cleanup	18	yes — private
logrite-semantic-api	checks	11	yes — private
Logrite-AI	checks	10	yes — private
logrite-mcp-server	checks	2	yes — private
		1,612 billable

1,612 billable minutes a cycle → ~0. None of that work needed a throwaway VM: it lints, runs unit suites and builds images, all of which this box already does for the dev environment — with the BuildKit layer cache and the ~/.m2 it built up doing so.

This page is the operator’s view

ops/runner/ in LogriteMasterControl owns the installer, the host toolchain script, the systemd template and the deregistration script; its README.md is the maintainer’s reference. The runner is not the deploy path — see Discord Notifications for how a push reaches dev.

The label contract

Every migrated job says exactly this, and nothing else selects a runner:

runs-on: [self-hosted, logrite-dev]

Both labels, always. self-hosted on its own would also match any future runner someone registers on another box; logrite-dev on its own is not a valid selector. Nothing outside this pool carries logrite-dev, which is what makes the failure mode unambiguous: a job that sits in Queued is a runner service that is down, not a mislabelled job and not a billing problem.

systemctl status 'actions.runner.*'        # on deployd-dev — the health check

The eight instances

All eight are online, all labelled self-hosted, Linux, X64, logrite-dev, all owned by deployd under /opt/actions-runner/<name>/:

runner	serves
logrite-dev-ui	Logrite-UI
logrite-dev-ai-service	logrite-ai-service
logrite-dev-analyzer	logrite-analyzer
logrite-dev-ai	Logrite-AI
logrite-dev-semantic-api	logrite-semantic-api
logrite-dev-webhook	logrite-webhook
logrite-dev-mcp-server	logrite-mcp-server
logrite-dev-mastercontrol	LogriteMasterControl

They are repo-scoped — one per private repo with an active workflow — because the operator’s gh token carries repo but not admin:org, and only the org endpoint can register a pool that serves every repo.

Repo-scoped runners do not share work

A repo-scoped runner serves only its own repo, so Logrite-UI’s two jobs (checks and tour-browser) serialise on logrite-dev-ui instead of running side by side. That is the cost of the fallback, and it is the symptom to expect before anyone reaches for “the box is too small”.

Org-wide registration is the preferred shape and ops/runner/install.sh switches to it automatically once the token has admin:org, registering a logrite-dev-1..N pool against https://github.com/Deployd-io. That has not been done yet. When it is, drop the repo-scoped instances first — install.sh skips any instance that already has a .runner file, so re-running it after the scope change adds an org pool alongside the old one — eleven listeners on a twelve-core box at the installer’s default count:

gh auth refresh -h github.com -s admin:org     # as an org owner
sudo ops/runner/deregister.sh
sudo ops/runner/install.sh

Why JobSearch stays on ubuntu-latest

JobSearch is a public repo, so its 38 minutes of LogRite Gatekeeper cost nothing and there is no saving to chase. The reason it is excluded rather than merely unmigrated is security:

A self-hosted runner must never serve a public repo

A pull request from a fork runs its own workflow code as the runner user. Here that user shares a host with the Coolify stack, the deploy keys, the loopback registry and every service’s data. One fork PR would be arbitrary code execution on the dev VM, dressed as CI. No public repo goes on this pool — not JobSearch, not a future one.

Rules a workflow on this box must follow

These are not style preferences. Each is a way this host differs from a throwaway ubuntu-latest VM, and each names a failure that has a real cost here.

No fixed host ports. This box shares its ports with ~35 Coolify containers — 9001 is the dev analyzer — and with any concurrent job. A hard-coded port fails at “wait for server to be ready” for reasons that look nothing like a port clash. Every job that starts a server allocates one at runtime, the same idiom as scripts/quality/run-harness.sh:

- name: Allocate a free port
  run: |
    port="$(python3 -c 'import socket;s=socket.socket();s.bind(("127.0.0.1",0));print(s.getsockname()[1]);s.close()')"
    echo "CI_PORT=$port" >> "$GITHUB_ENV"

Everything downstream reads $CI_PORT — the server’s PORT, the readiness poll, the BASE_URL of every test step.

No sudo, and no --with-deps. A job’s apt-get install — or the sudo rm -rf /usr/share/dotnet /usr/local/lib/android … “reclaim runner disk” step that is harmless on a hosted VM — mutates the real machine, on every run. Host-level setup lives in ops/runner/provision-toolchain.sh, which provisions /usr/local/bin/esbuild (0.21.5) and /usr/local/bin/tsc (5.7.3), the exact paths scripts/quality/run-harness.sh exports as ESBUILD_BIN/TSC_BIN, plus the Playwright system libraries. harness-verify.yml now does a 40 GB headroom precheck instead of deleting directories.

No cache: npm / cache: maven / actions/cache@v4. ~/.npm and ~/.m2 persist on this host between jobs, so the cache service is a WAN round-trip that also spends the 10 GB-per-repo cache allowance on bytes already on the disk.

No docker/setup-buildx-action and no type=gha cache. The default builder’s BuildKit cache persists in the daemon between runs, which is the point of building on a machine we own.

No actions/upload-artifact. The Free plan’s 500 MB artifact allowance is shared with GitHub Packages — the same quota ghcr-cleanup.yml exists to defend. Reports are copied to the host instead, and the job’s step summary names the path:

/var/lib/logrite-ci/artifacts/<repo>/<run_id>/

They are already on the machine the team SSHes into.

Installing and removing

sudo ops/runner/install.sh          # idempotent — provisions the toolchain, registers, starts, verifies
systemctl status 'actions.runner.*' # the health check
sudo ops/runner/deregister.sh       # all instances
sudo ops/runner/deregister.sh logrite-dev-ui   # just one

install.sh is root-only, safe to re-run, and calls provision-toolchain.sh itself before config.sh — the runner’s PATH is frozen at configure time, so the tools must exist first. It refuses to report success until GitHub reports every configured runner online, printing the failing unit’s journal instead; an installer that exits 0 on an offline runner is how a pool silently shrinks to nothing.

Deregistration has to happen on both sides. Stopping the service only makes the runner offline — GitHub keeps the registration and keeps counting it as a member of the pool, so config.sh remove is the half that actually retires it. That is what deregister.sh does.

A credential a job needs is already on this box

A repo secret buys no isolation on a self-hosted runner. GitHub decrypts it onto this same VM, into that same job’s shell — so for a credential the box already holds, a secret is not a safer copy, it is a second copy, living outside the estate and outlasting the container it was taken from.

The W5 judge benchmark in logrite-ai-service is the worked example. It needs a live LLM provider key, and the ai-service container on this host already has one as NEURALWATT_API_KEY. So the key is provisioned to the host, not to GitHub:

sudo ops/deploy/install_judge_eval_key.sh        # re-run to ROTATE — it re-reads the container

It writes /etc/logrite-ops/judge-eval.env, 0640 root:<runner-user>, and the workflow sources it only when secrets.LLM_API_KEY is unset — so moving to a hosted runner later needs no change here. The job log names which source answered and never the value:

judge key source: runner-host-file (67 chars)

Two properties are load-bearing, and both exist because of failures this estate has already had. The script matches the container on its uuid prefix, never its service name — the name carries a deploy timestamp that changes on every deploy, so a pinned name would silently provision an empty key, which the workflow then reports as “not set”, i.e. as though nobody had ever run it. And it proves the runner user can read what it wrote before reporting success: a file written unreadable is indistinguishable, from the workflow’s side, from never having been written.

A key is only half of a gate

Setting the credential does not arm a benchmark that compares against a recorded baseline — the baseline is the other half. logrite-ai-service’s judge baseline had drifted three months from its own case set, so the first scheduled run after a key appeared would have failed on “2 case(s) in the baseline are absent from this run”, every night, for ever. A red nobody can act on is how people learn to ignore red. Check what the gate compares against before you give it the means to run.

A workflow that cannot parse reports nothing

security-gates.yml carried if: hashFiles(...) on three jobs. That function is only available in jobs.<job_id>.steps.* — there is no checkout when a job-level if is evaluated — and GitHub rejects the entire file for it. It had therefore never run, since the day it was written, while showing a red ✗ on every commit to main.

The distinguishing evidence is check runs, not jobs.

	jobs	check runs	looks like
billing-blocked (2026-08-19 → the migration)	0	≥1	a failing gate
unparseable workflow file	0	0	a failing gate

A billed-out job is still created before it is refused, so it can report. An unparseable file creates nothing, so the only thing GitHub can show is “this run likely failed because of a workflow file issue” — and from the run list the two are identical. This one was first diagnosed as the billing block, moved to the self-hosted runner on that basis, and failed identically; that second failure is what said the diagnosis was wrong.

# what named it, in one run
docker run --rm -v "$PWD":/repo -w /repo rhysd/actionlint:1.7.12

checks.yml now runs exactly that, pinned, immediately after checkout, because nothing in a repository can otherwise see an unparseable workflow — the only thing that reads workflow files is the thing that refused to read it. .github/actionlint.yaml declares logrite-dev so the custom label stops reporting as an error on all eight runs-on lines; a linter that is wrong eight times is a linter nobody runs, and then it is not a gate.

A parsing workflow is also visible in the run list without opening it: GitHub cannot know a workflow’s name: until the file parses, so a broken one is listed by its path (.github/workflows/security-gates.yml) and a working one by its name (security-gates).

What this did not change

• Deploys still use no Actions. A push reaches dev through logrite-deploy-watcher.service, which polls each branch tip and posts to the management console; the console builds on this VM and deploys with compose. (Before 2026-09-23 this read “GitHub webhook → Coolify builds from source on the VM” — Coolify has since been removed from dev entirely.) Either way it is not an Actions path and not a billing consequence.
• All eight docker-deploy.yml workflows stay disabled_manually. Re-enabling one would give dev a third concurrent deploy path alongside the Coolify webhook and logrite-deploy-watcher.service. Their runner and cache settings were brought in line anyway, so a future re-enable cannot silently reintroduce billed minutes.
• GHCR is still written only by those disabled workflows, by harness-verify.yml and by ops/registry; the loopback registry:2 at 127.0.0.1:5000 remains the deploy path.
• scripts/quality/run-harness.sh remains the primary, attended way to run the quality harness; harness-verify.yml remains the on-demand clean-room run.