Quality Harness

The harness answers one question the unit tests cannot: if a customer connects a repository in language X, do we find their methods, insert good logs, and leave the file valid?

It emulates a repository per language, drives the real flow — sweep → analyze → build DTOs → addLogs → syntax gate → re-scan → execute → replay — and scores the result across thirteen dimensions. It is a pipeline, not a fixed test: every language in LanguageDetector.instrumentableLanguages() is enumerated, and one with no fixture yet is reported as NO FIXTURE rather than quietly omitted, so the report always shows the true denominator.

Running it

Use the runner script. It encodes every step that has already cost a full measurement run — the mandatory -Dharness.report.dir, a fresh analyzer container per run, the clean build, the keepalive that stops Coolify pruning the 6 GB image mid-run, and reclaiming the root-owned target/ a previous in-container run leaves behind.

./scripts/quality/run-harness.sh                     # the full board, in the verification image
./scripts/quality/run-harness.sh --determinism full  # + the pipeline replay
./scripts/quality/run-harness.sh --languages scala   # diagnostic: no report, no ratchet
./scripts/quality/run-harness.sh --no-image          # host-side, FAST and PARTIAL

The raw invocation, if you need it:

# From logwrite-api/ — the harness profile is required
mvn test -Pharness

# Against a deployed analyzer instead of a locally-built container
mvn test -Pharness -Dharness.analyzer.url=http://192.168.200.102:9001

# ONE language, for a debugging loop — ~110 s instead of ~550 s
mvn test -Pharness -Dharness.languages=scala

# Keep the file a gate is about to revert, so the failure can be read
mvn test -Pharness -Dlogrite.gate.keepReverted=/tmp/kept

The analyzer runs in Docker under NODE_ENV=test (the keyless mode the LLM provider supports). If the container cannot start, the harness aborts rather than fails — an unavailable Docker daemon is not a quality regression.

The verification image

Ten checker rows self-skip on a machine without their toolchain, so a run on a normal laptop silently grades a subset. A full, comparable run must happen inside logrite-harness-verify:local, built from Dockerfile.harness (~6 GB, 29 toolchains).

# 1. the analyzer, mounted live, on a port of its own
docker run -d --name logrite-verify-analyzer \
  -v "$PWD/services/analyzer/dev/logrite-analyzer:/app" -w /app \
  -e NODE_ENV=test -e PORT=9000 -p 19311:9000 \
  node:22-bookworm bash -c "npx ts-node src/server.ts"
until curl -sf http://localhost:19311/health >/dev/null; do sleep 3; done

# 2. the harness, against it
docker run --rm \
  -v "$PWD/services/backend/dev/Logrite-AI:/workspace" \
  -v "$PWD/docs/quality:/report" -v ~/.m2:/root/.m2 \
  -w /workspace/logwrite-api --add-host=host.docker.internal:host-gateway \
  logrite-harness-verify:local bash -lc 'mvn -o -B test -Pharness \
    -Dtest=LogGenerationQualityHarness \
    -Dharness.analyzer.url=http://host.docker.internal:19311 \
    -Dharness.report.dir=/report'

Takes ~3–6 minutes.

Two more things the image teaches:

esbuild is REQUIRED, not nice-to-have. Without it JsSyntaxGate silently falls back to node --check — a fallback that hid a TypeScript F for an entire session. verify-toolchains.sh probes 28 required toolchains and runs as the last layer of the build, then again in CI against the pulled image.
prove-gates.sh is deliberately not baked into the image. CI mounts it read-only so the proof matches the checked-out gate definitions. Run it after any gate change: it asserts both directions — a pristine file is ACCEPTED and a deliberately misplaced log is REJECTED at our line. A gate that cannot fail is worse than no gate.

`parseOnly` — which checkers may revert

Only a parse-only checker may revert a file that already failed before instrumentation. A type-checker reports where it lost the thread, routinely the line after the real break, which is what LanguageSyntaxGate’s upward walk exists to rescue. Getting this backwards makes the gate blame our logs for the fixture’s own pre-existing errors.

What it scores

Thirteen dimensions per language. Detection, validity, execution, determinism and placement carry double weight because they are the five that make output unusable; instrumentation, insertion and rulePlacement carry 1.5.

Dimension	Weight	Asks
detection	2.0	Did we find every construct the fixture marks `shouldLog`?
validity	2.0	Did the syntax gate accept every file, or did any get reverted?
execution	2.0	Does the instrumented file still run? (added 2026-08-06)
determinism	2.0	Does the same input give the same answer? (added 2026-08-12)
placement	2.0	Is the line in a position that works — not after a `return`, not outside the method whose variables it reads? (added 2026-08-14)
instrumentation	1.5	Of the constructs found, how many actually received a log?
insertion	1.5	Of the distinct insertable recommendations, how many landed?
rulePlacement	1.5	Did each expected rule put a log in the file? (added 2026-08-12)
ruleCoverage	1.0	Did each construct get the rules its manifest expects (entry, exit, database, external calls, exceptions, variable tracking)? ⚠️ recommendation, not placement — see below
consistency	1.0	Is the emitted format uniform across files and constructs?
falsePositives	1.0	Did we leave the constructs marked `excludedConstructs` alone?
idempotency	1.0	Does a second scan of an already-instrumented repo add zero lines?
fidelity	1.0	Do the emitted lines match the surrounding file’s own conventions (indent character and width)? (added 2026-08-14)

Composite → grade: A ≥ 0.95, B ≥ 0.85, C ≥ 0.70, D ≥ 0.50, else F. The report also states the composite as a score out of ten, with its denominator whenever a language was measured over fewer than thirteen dimensions — 9.9/10 (7 of 13) and 9.9/10 are not the same claim.

placement and rulePlacement are different questions. rulePlacement asks whether each rule the manifest expects put a log somewhere in the file — it needs annotated ground truth. placement asks whether the line we actually emitted works where we put it, and is ground-truth free: it reads the diff, not the manifest, which is what lets it score a repository nobody has annotated. Both can be 1.0 while the other is 0.

placement and fidelity are computed together by PlacementAudit, which classifies each injected line against the statement it follows — dead code after a terminal statement, an exit log outside the method whose variables it names, a timer read on the line before it is started. Ruby surfaced all three at grade C, and eight of its nine defects were invisible to every other dimension; the ninth was caught by execution only because it happened to throw first.

determinism — three layers

The product’s promise is that connecting a repository twice does the same thing twice. Until 2026-08-12 nothing measured it: run-to-run stability had been checked exactly twice, by hand.

Layer	What it catches	Cost
A — analyzer replay	The analyzer asked the same questions again, once in order and once reversed. The reversed pass is the interesting one: it catches an engine whose answer for file B depends on having seen file A.	seconds
B — pipeline replay	The whole flow re-run from pristine into a second directory and byte-compared. The only layer that can see the backend’s unsorted sweep, its thread pool, or gate ordering. `-Dharness.determinism=full`.	~doubles the run
C — output digest	A SHA-256 of the instrumented tree, recorded in `baseline.json`. Detects drift across processes, machines and analyzer versions.	free

rulePlacement — the question no dimension was asking

ruleCoverage reads the analyzer’s response. insertion subtracts safety drops from its own denominator. instrumentation asks whether a construct got any log. So a rule dropped every single time cost a language nothing at all — and exception_handling produced no log in 11 of 28 languages while the report called several of them 1.000.

rulePlacement asks, per rule: did at least one of that rule’s recommended statements reach the file? One landed log is enough — a rule that places 5 of 6 is working, and charging for occasional drops would re-create the bug insertion was corrected for. The report also carries a per-rule placed/recommended matrix, so 0/6 (dropped) reads differently from - (never asked).

An unmeasured dimension is excluded from the mean, never counted as 1.0. A language with no execution driver is therefore scored over eight dimensions rather than nine — honest, but it means its composite is not directly comparable to a language that was executed.

The gap between `ruleCoverage` and the file — read before trusting a 1.000

No dimension currently asks “did this rule put a log in the FILE?”, and a census on 2026-08-12 found what that costs: exception_handling emits nothing at all in 11 of the 28 languages, and the report calls several of them perfect. java scores 1.000 on all nine dimensions with one finding, and that finding is the exception log being dropped.

Two sound choices meet, and the gap is between them:

ruleCoverage reads the ANALYZER’s per-rule response. The rule recommended a log, so it counts as fired — whatever happened to it downstream.
insertion deliberately subtracts safety drops, so that “declined to place a log where placing it would break the code” is not scored as failure. Correct for an occasional drop; it means a rule dropped every single time costs nothing.
instrumentation asks whether a CONSTRUCT got any log, and riskyOperation did — its entry and exit logs landed.

The placement half is fixed (2026-08-12): 17 → 24 languages now emit an exception log, and not one score moved. The key was ex.line + 1 for every language and Util adds OFFSET on top, so the log was written one line PAST the handler’s first statement — after the return that a catch body almost always is. It is now keyed at the handler’s first statement read from the source, which is ex.line for K&R and Python-style clauses and ex.line + 1 for C#‘s Allman brace, exactly as before.

On execution — read this before touching it

Every other dimension is satisfied by a log that compiles. None can see a log that compiles, resolves every symbol, and then throws at runtime.

ExecutionProbe runs each fixture twice — pristine, then instrumented — and scores the delta per case, because several fixtures throw by design (C#‘s Database.Query is throw new NotImplementedException()). A construct that threw before and throws the same way now is not our defect; one that ran clean and now throws is exactly our defect.

Drivers live in logwrite-api/src/test/resources/harness/drivers/<language>/ and print LOGRITE-EXEC <case> <ok|throw> <detail> per case, then LOGRITE-EXEC-END. The runner table is written out by hand in ExecutionProbe.java — deliberately, so adding a language forces an explicit decision about how its output gets executed, exactly like CheckerCoverage does for compilers.

Two things the Rust driver had to get right, and both generalise to the compiled languages that still lack one:

Use the toolchain the image actually has. It carries rustc 1.75 and no cargo, so a cargo-based driver would have meant rebuilding and republishing the 6 GB image. It does not need one — the fixture is crate-free, which is the same property that lets the validity gate check it with rustc --crate-type lib --emit=metadata on a single file.
Splice the fixture in at the crate root, do not import it. The fixture declares its logger shim as pub mod log at its own file root and the instrumentation emits the absolute path crate::log::debug!(…). Pulled in as mod user_service, the shim would sit at crate::user_service::log, every injected log would fail to resolve, and the probe would report a compile error it created itself as though the instrumentation had broken the file. include! puts it where the emitted paths expect it.

The second point is the general trap: a driver that changes the module path, package or namespace the fixture is compiled in will blame the product for its own scaffolding. Run the driver against the pristine fixture first — if that does not execute clean, nothing measured afterwards means anything.

A fixture can ask a question no correct answer exists for

rust reached 1.000 on all nine dimensions (2026-08-12) — the fifth language to do so, and the change that got it there was to the fixture, not only the engine.

Its risky_operation had been written Err(_) => String::new(),: a single expression, with no statement position anywhere inside it. exception_handling therefore had nowhere legal to put a log, so the Rust placement pass walked it up to the top of the method, where it merged into the entry log’s key — and the rule was reported as RULE SILENT: exception_handling produced nothing (ruleCoverage 0.833). Worse than the score: the emitted log announced an exception on every successful call.

Rewriting the arm as the block Rust actually writes — and BINDING the error, so the log has a value to report — made the position legal, and exposed two engine defects that were only ever hidden by the fixture being too small to reach them: the placement window stopped at a tail expression’s first line, treating a whole trailing match as untouchable when each of its block-bodied arms is a legal statement position; and the merged-group carrier always preferred the exit log, which deleted the file’s only exception_handling entry.

The rest of a production-shaped Rust fixture — an axum handler over sqlx and reqwest, async fn, ? propagation, a trait impl — is written, compiles under the gate’s own command and runs clean under the driver, but instruments to 0.569 with the file reverted. What blocks it turns out to be the harness, not Rust: the gate type-checks each .rs file alone, so the fixture must declare its own sqlx/reqwest/axum stand-ins IN the scored file, and the engine is then meeting a shape no real service file has. Artifact, pre-gate evidence and the architectural decision it forces: docs/quality/rust-production-fixture.md.

The detection matrix — which client libraries each language actually recognises

The harness above measures the whole pipeline over one fixture per language, and every fixture asks its database question once, in one dialect. A Node service therefore scores against pool.query('SELECT …') and nothing else — while that stack’s default database is MongoDB, and the ecosystem around it is Mongoose, Prisma, Sequelize, Knex, Redis, DynamoDB and Firestore.

DetectionMatrixTest asks that narrow question on its own, over 141 cases — one small file per client library — and reports two answers where the main harness reports one:

	asks
detected	did the parser record the call in `databaseOperations`?
logged	did a recommendation actually come back for it?

Three rounds of work came out of it, and every case in the matrix now both detects and logs:

                       first run     round 2      round 3
database  detected      54 / 96     99 / 113    113 / 113
database  logged        40 / 96     99 / 113    113 / 113
http      logged        39 / 45     39 /  45     45 /  45

objc  0.909 → 1.000   shell 0.769 → 0.906   perl/php/powershell +0.077 each
mean composite 0.8903 → 0.9067, nothing down

Round three (2026-08-13) closed the last 20. They were not exotic: dart, elixir, kotlin and scala had no database arm at all and were carried entirely by the language-agnostic raw-SQL pattern, so Ecto — the only way an Elixir service reaches Postgres — was invisible. On the HTTP side the gaps were each stack’s default client: Feign, Ktor, Alamofire, Dio. Two of those six cannot be recognised by a call shape at all, only by a receiver name: Feign’s entire purpose is that the call site looks like an ordinary interface method, and Ktor’s is client.get(url), a bare .get( that must never match on its own.

It cost nothing in placement — the full 28-language board came back byte-identical, zero digests changed.

JavaScript and TypeScript had no database detection of their own — the pattern list is Java/Spring/.NET shaped, so only an inline SQL literal ever matched an Express service.
The Java DB timer was keyed AFTER the query it times, so a method whose query is its first statement had the timer land on the return and both halves deleted. Java recognised 7 of 10 clients and logged 1. The fixture hid it: its query has statements after it, so the key landed on ordinary code and the backend’s key chain happened to drain the timer to the right place.
The file-level gate duplicated the detector, and now asks it instead.

The four languages that gained have nothing to do with Node: their fixtures were being classified “general file”, so database_logging was filtered out before it ran and the log the second pass then added scored as non-idempotent.

The second round closed the rest. The gate was being asked its question without the language, so every ORM the detector keeps in a language-gated branch — Django, SQLAlchemy, peewee, Tortoise, MongoEngine, ActiveRecord, Sequel, Mongoid, GORM, Eloquent — was skipped at the gate and the rule filtered out. It only reproduced with a second rule in the request, which is why it read as nine unrelated per-language gaps. Swift, which had no database detection at all, gained CoreData, SQLite.swift, GRDB and Realm; Rust gained diesel, the mongo driver and redis-rs; Python gained model persistence (User(...).save(), user.update(**kwargs)) and redis-py.

mvn -B -Pharness test -Dtest=DetectionMatrixTest \
    -Dharness.analyzer.url=http://127.0.0.1:9001 \
    -Dmatrix.baseline=/keep/detection-matrix.json    # REQUIRED inside the image

Fixtures are the instrument, and a toy fixture measures a toy

A production-shaped Python fixture was written the same day — decorators, wrapped annotated signatures, docstrings, real imports, the Django ORM, with transaction.atomic():, a bound except … as exc. It found five engine defects, including one that quietly destroys docstrings in a customer’s repository: the entry log went above the docstring, which demotes it to an ordinary expression statement, so __doc__, help() and every IDE hover go empty and nothing reports a problem.

On idempotency

It is binary — any duplicate line scores 0 — and it is the dimension that most often costs a language its A. The report names the actual added lines, not just a count, because a count cannot be diagnosed:

NOT IDEMPOTENT: re-running added 2 line(s)
  ↳ added on re-scan: long start = System.currentTimeMillis();

The scan is re-run in full against the already-instrumented tree — not replayed from cached line numbers, which would measure nothing.

The twelve languages that scored idempotency 0.00 when this was investigated share one cause — do not triage them individually. (Two have cleared since; see below.) Placement is decided against the file as it currently stands, not against the pristine source, so pass one’s own insertions create the anchor pass one was looking for and pass two places what pass one dropped.

It surfaces at two stages, fixed in different repositories:

M1 — pass one drops the log as no placeable anchor and pass two places it. The added line appears verbatim in the same run’s SAFETY DROP list. Backend, FileUtil. 14 of 19 re-scan additions, 9 of 12 languages.
M2 — pass one’s analyzer response never recommends the log at all. Analyzer. Accounted for python and javascript entirely; python is fixed, and javascript is the same shape.

The dropped logs are real misses on the first pass, which is the only pass a customer repository ever gets — so this suppresses coverage as well as idempotency, and a fix whose mechanism is “emit less” is not a fix.

Full investigation, the per-language table and the reproduction: docs/quality/idempotency.md.

First one cleared (2026-08-12): r reached idempotency 1.00 (0.761 → 0.838), with python findings 9 → 5 and shell 15 → 13 and nothing regressed. The branch handling “this line is the last statement in the method” emitted a method’s insertions before appending that line — right when the line is a return, wrong when it is ordinary work, so python’s __init__ read self.tenant one line before it was assigned. Keys past the last original line now go to an EOF flush that steps up over trailing block terminators and one trailing terminal statement, which places both shapes from one rule.

Second cleared, and the first M2 (2026-08-12): python reached idempotency 1.00 — 0.908 → 0.985, B → A, findings 5 → 4, with the other 27 languages byte-identical. Python was routed through bodyKeyPlacement, the analyzer pass that re-anchors from each log’s natural key downward and shares a key rather than bumping a loser out of the body — the pass that already owned VB and Kotlin. Ten languages still score 0.00.

That one change made three latent backend defects reachable, and all three are fixed at their own root rather than worked around: LanguageDetector was applying a Java/JS “ends with ) but not ); → unterminated” heuristic to Python, where a balanced self.db.query(…) is a complete statement; FileUtil’s consecutive-key chain drained a duration log past the query it times; and the cross-pass timer suppression is a counter that the chain could spend twice, so a re-scan wrote a second import time; __log_start = time.time() into a method that already had one.

-Dharness.keep.dir writes a fourth phase: pass1-pregate — the tree as the inserter left it, before any gate pruned or reverted. A PRUNED: finding names the statement removed but not the file that provoked it, and the gates delete exactly the evidence needed to see why: py_compile reports one error at a time, so a single bad indent takes a neighbouring log down with it and leaves no trace of either. Two wrong theories were reasoned out before this tree was kept; it answered the question in one diff.

<keep-dir>/<language>/{pristine,pass1-pregate,pass1,pass2}/…

Two cases are not duplication and are reported as themselves, because scoring them the same 0 made both undiagnosable:

The file was reverted
The re-scan destroyed the work

A reverted file is restored to pristine, so a second pass instruments and reverts it identically and the diff is empty. That is a perfect score for the one behaviour the language never performed — and because the ratchet pins any dimension reaching 1.0, the fix for the revert then reads as a regression and fails the build. The ratchet was defending the broken state; Scala, F# and Elixir all recorded a false 100% this way.

A run that added nothing and reverted something is now NOT MEASURED. A run that did duplicate lines keeps its 0 — that is a real observation whatever else happened.

A negative delta means the second scan removed what the first one did, and the only thing that removes lines is a gate. Vue scored -18 with zero duplicate lines while reporting validity 100%, because the pass-2 GateOutcome was discarded and the revert that erased 18 correctly-placed logs was recorded nowhere.

RE-SCAN REVERTED: src/UserService.vue — the second scan destroyed pass one's instrumentation
· idempotency 0.000 (18 line(s) LOST on re-run — the re-scan gate reverted src/UserService.vue)

Pass one’s own validity is still reported separately and may legitimately read 100%, which is why the finding says so — that combination is otherwise unbelievable.

Every lost point is explained

insertion counts distinct insertable recommendations, and misses are classified as DUPLICATE_RECOMMENDATION, SAFETY_DROP, or UNEXPLAINED. Correct de-duplication and correct safety drops are not failures, and scoring them as such was a harness bug, not a product one. Only UNEXPLAINED should ever be non-zero, and when it is, that is the thing to go and fix.

A language nobody checked is UNGRADED

This is the harness’s most important rule, and it was learned the hard way.

validity used to score 1.00 whenever nothing had been pruned. That is the same answer to two completely different questions — did the output parse? and did anything ever parse it? The consequence was not subtle once separated out: every A grade on the report belonged to a language with no checker, and no verified language scored above a B. The report was ranking languages by how little was known about them.

Worse, for a period the gates had never run at all: they find our injected lines by diffing the working tree against git, and the harness materialised fixtures into a plain temp directory, so Git.open threw, each gate caught it and returned an empty result. The run log carried [syntax-gate] skipped — repository not found 28 times, once per language, on every run.

Both are fixed. The harness commits a pristine baseline before instrumenting, CheckerCoverage records which gate checks each language, and a language with none reports UNGRADED rather than a letter. CheckerCoverageTest fails the build when a language is added without an explicit verification decision, and pins the verified count so it can only be raised deliberately.

Fixtures must compile before they are instrumented

Every fixture is a working project that its own compiler accepts before any log is inserted. This is not a nicety. If the pristine fixture does not compile, the gate blames our injected logs for pre-existing errors and every number it produces is noise — and when compile gates were first switched on, not one fixture compiled. The Java one alone had five errors: Database and HttpClient undefined, and restTemplate called but never declared.

Support files (a go.mod, a .csproj, stub type declarations) are listed in the manifest with "constructs": []. Fixture.materialise only writes files the manifest names, so an unlisted support file never reaches disk; an empty construct list keeps it out of the graded denominator.

Fixtures

One directory per language under logwrite-api/src/test/resources/harness/fixtures/<language>/, each with a manifest.json and the source files it describes:

{
  "language": "go",
  "label": "Go",
  "files": [{
    "path": "cmd/user_service.go",
    "constructs": [{
      "name": "FetchRemote",
      "role": "fetchRemote",
      "kind": "method",
      "shouldLog": true,
      "expectRules": ["method_entry_logging", "method_exit_logging",
                      "external_service_calls_response_times"]
    }]
  }]
}

Directory names use the analyzer’s routing token, not the backend’s display label — csharp, cpp, objc. LogGenerationQualityHarness.fixtureId() holds that mapping, in one place.

Coverage grows by adding directories, not code. A language with no fixture fails the build (enforceFixtureCoverage) precisely so that a newly-supported language cannot arrive unmeasured and quietly turn every number on the report into a score over a subset.

The ratchet

Nothing here asserts a hand-picked expectation. The build fails on exactly two things, both about the numbers staying honest rather than any one language being good enough:

a language scoring below its recorded baseline in docs/quality/baseline.json — either its composite falling, or a dimension that had reached 1.0 coming back below it;
an instrumentable language having no fixture.

That makes the bar a ratchet raised deliberately, not a wall that blocks landing honest results.

Fixed 2026-08-12: baseline.json now records what it was measured against.

"measuredAgainst": {
  "analyzerCommit": "512eb98",
  "analyzerTarget": "source",
  "backendCommit": "3c63465"
}

A dirty tree records as <sha>-dirty, deliberately ugly — a baseline nobody can reproduce should be obvious in review rather than discovered later. The runner script states the analyzer revision explicitly via -Dharness.analyzer.rev, because the usual shape defeats inference: the harness sees an external analyzer whose code is nonetheless the working tree next door.

The warning below is what made this necessary, and is kept as the reason.

A baseline only means something against a stated analyzer commit. The harness drives the analyzer over HTTP, so the analyzer is an input to every score — and the ratchet cannot tell an analyzer change from a backend one.

Measured on 2026-08-12: the baseline recorded against analyzer 5c2cd67 was compared against a run using 8b0cfe4, and five languages moved in both directions with the backend held constant — fsharp 0.855 B → 0.500 D, scala 1.000 A → 0.855 B, shell 0.846 → 0.769, c 0.923 → 0.945, lua 0.968 → 0.987. fsharp had lost a third of its composite and nothing surfaced it, because the drift was being attributed to whatever backend change was under test.

Record the analyzer commit whenever you re-record the baseline, and pair any change-run with a same-conditions control run rather than comparing against the committed baseline.

The waiver is two-tier, and conflating the tiers costs you a green build on a regression

When a new dimension arrives, existing baselines predate it and something has to be waived. What gets waived depends on which dimension:

New dimension	Waives
a validity checker	the whole language — a gate that starts running also starts reverting, and reverts cascade into `instrumentation` and `insertion`
any other dimension	the composite only, and nothing else

Output

docs/quality/ in LogriteMasterControl:

File	Contents
`log-generation-quality.pdf`	The board, every language graded x/10, plus the rule × language matrix and every finding. The artefact to send someone
`log-generation-quality.html`	Branded report — per-language table, dimension breakdown, findings, real emitted samples
`log-generation-quality.md`	Same, for reading in the repo
`baseline.json`	The ratchet’s recorded scores, plus `measuredAgainst` and a per-language `outputDigest`
`detection-matrix.json` / `.md`	The narrow instrument: which DB and HTTP client libraries each language recognises

Anything that moves a score for measurement reasons goes in the report’s “Measurement corrections” section, not silently into the baseline. Three such corrections are recorded there already; the discipline is what makes the trend line mean anything.

Adding a language to the harness

Add its generator and a LANGUAGE_CATALOG entry in the analyzer (see Language Support).
Add it to LanguageDetector.INSTRUMENTABLE_LANGUAGES in the backend.
Create fixtures/<token>/manifest.json plus source files exercising each rule you expect to fire — and at least one construct in excludedConstructs that must be left alone.
Run mvn test -Pharness. The first run records its baseline; every run after that defends it.

Reading the board without being misled (2026-08-12)

Board at this revision: mean 0.9287 — 13 A, 10 B, 3 C, 1 D, 1 ungraded, measured against analyzer e39ad39 / backend 0e9b2e6. Four dimensions of hard-won reading advice, each of which cost a wrong conclusion before it was written down.

`validity` is a step function, not a proportion

validity = reverted.isEmpty() ? (pruned.isEmpty() ? 1.0 : 0.7) : 0.0;

Any prune at all scores 0.7. Halving a language’s prunes moves nothing. C++ went from two pruned logs to one and the dimension did not budge — which reads as “the fix did nothing” and is wrong. Judge a prune fix by the prune COUNT in the findings list, not by the dimension. It only moves at zero.

`SAFETY_DROP` is not a diagnosis

LogGenerationFlow labels a miss SAFETY_DROP merely because the file had other logs in it. It is the harness’s benign default when it cannot tell why a log is absent, not a finding that the pipeline declined for safety. Thirteen of fourteen B-grade languages reported one and they did not share a cause.

A composite that rises while `ruleCoverage` falls is a failed experiment

Two languages’ composites rose in an experiment that made them emit fewer rules. Read ruleCoverage before the composite. “Never delete a log to move a number” applies to the score as much as to the code.

The pre-gate diff is the fastest diagnostic in the harness

--languages <lang> -Dharness.keep.dir=<dir> keeps four trees: pristine, pass1-pregate, pass1, pass2. pass1-pregate is the instrumented file before the syntax gate prunes it, so

diff -u <dir>/<lang>/pass1/src/F <dir>/<lang>/pass1-pregate/src/F

shows every rejected log in place, in context. Every fix that landed on 2026-08-12 came from that diff or from a direct probe; every hypothesis reasoned from source alone was wrong. -Dlogrite.gate.keepReverted=<dir> does the same for whole-file reverts.

What the fixtures can and cannot measure

Longest method body per fixture: cpp 4 lines, javascript 4, java 6, c 8. Six rules — entry, exit, variable tracking, database timing, external timing, exception — compete for roughly three insertion keys, and a timing pair needs two keys separated by the work it times.

So SAFETY DROP (no placeable anchor) in those languages is largely not a placement defect. It is what a correct pass does when asked to fit six logs into three slots, and c/cpp/javascript cannot reach 1.000 on these fixtures however good the placement is.

The pattern across a full day of attempts is the tell: every placement change was measured and reverted; every punctuation or scope correctness change landed —

Landed	Reverted
F# block-opener wanted Python’s `:`	bounds at both ends of the body
`ReturnFinder` wanted Java’s `;`	variable-tracking key offset
PHP `%s` on an object is fatal	`OWNED` expansion to c/cpp/js
Ruby emitted an undefined `logger`	Kotlin declaration patterns
C++ namespaced declarations

Before investing further in placement, settle whether a fixture may carry support scaffolding the gate compiles but does not score — docs/quality/wip/{python,rust}-production-fixture/ are waiting on exactly that call. It is a measurement decision, and until it is made, placement work against these fixtures improves the instrument rather than the product.