Collector Agents

Overview

Collector Agents lets a user mint a workspace-scoped agent and get a pre-baked one-liner — workspace, token, ingest endpoint, and the chosen collection capabilities are all embedded, so the user just copy-pastes and runs it. The agent ships into the central OpenSearch store over TLS on a chosen interval. It is the collection / centralization track of the program — an active, heartbeating agent is exactly the live coverage signal the M-26-14 maturity scorer reads (Collection Coverage, Centralization, Inventory, Data Retention, Log Management).

Collection capabilities (configurable per agent)

The create modal lets the user pick what the agent collects before provisioning; it starts from the workspace’s collector defaults (Settings → Log ingestion, collector.defaults.* — see Workspace settings). The choice is baked into the generated collector.sh (overridable per run with LOGRITE_* env vars), and from collector 1.3.0 the capability switches, interval and an absolute custom folder are also refreshed from the server at every check-in, so changing them needs no reinstall — see Live config at check-in:

Capability	Default	What it collects	M-26-14 element
Logs (always on)	on, locked	*.log·*.txt·*.jsonl·*.out from the run directory (or a custom folder)	Centralization · Log Management
Custom folder	run-location	watch a specific absolute path instead of the working dir (collect_path; resolved to absolute so cron runs watch the same folder)	Centralization
Heartbeat	on	a {"type":"heartbeat"} liveness ping each cycle, so an idle agent still counts as active	Collection Coverage (live signal)
System details	on	one {"type":"system_metrics", hostname, os, kernel, arch, cpus, memBytes, disk…} host-metadata doc (no root)	Inventory Visibility
📓 journald	off	the systemd journal via journalctl -o json, resumed per-entry by __CURSOR; PRIORITY→level; journald:<unit> filePath. Optional unit filter + system/user scope	Collection Coverage · Centralization
📁 /var/log	off	a path list (default syslog·auth.log·messages·kern.log·*.log) tailed through the shared inode/offset shipper (source:varlog)	Centralization · Log Management
🐳 Docker	off	the Docker socket — docker logs --since <cursor> --timestamps per container (non-root; the docker group grants the socket); raw-file fallback /var/lib/docker/containers/*/*-json.log for rootless/Podman/no-CLI. source:docker, container, docker://<name>	Inventory · Collection Coverage
☸️ Kubernetes	off	/var/log/containers/*.log (CRI) + the API-server audit log; source:k8s, namespace/pod/container, k8s://…	Inventory · Collection Coverage
📡 Syslog receiver	off	a background socat/ncat/nc -k listener (default UDP :5514) → spool → RFC3164/5424 parse (source:syslog, facility/severity/level)	Centralization · Collection Coverage

System-wide sources (2026-07). journald//var/log/Docker/Kubernetes/syslog are opt-in host capabilities that each stamp a source + a synthetic filePath so the existing read stack (/logs, /sources, /stats, Warden scan, Chat) works unchanged.

Seamless permission acquisition (2026-07). Reading /var/log, the systemd journal, or the Docker socket is impossible for an unprivileged process on stock Linux (they’re DAC-protected root/syslog:adm/systemd-journal/root:docker) — the same one-time sudo every agent (Datadog, Vector, Fluent Bit, …) requires. So when a privileged source is enabled, the mint install one-liner auto-inserts a single sudo step — sudo usermod -aG <only the needed groups> "$USER" (journald → systemd-journal, /var/log → adm, Docker → docker; sorted/deduped; single source of truth AgentService.CAP_GROUPS) plus a best-effort socat install for the syslog receiver. It’s non-blocking (; not &&): decline sudo and the collector still runs; the blocked source then reports loudly (below). The collector keeps running unprivileged — a group added by usermod reaches it on the next cron cycle (a running process never gains a group), so a freshly-granted source reads “activating on the next cycle” (benign, not an error). Logs-only agents get the unchanged, sudo-free one-liner.

Truly zero-touch (2026-07 hardening). Three refinements make installs need no manual step: (1) Docker collects over the daemon socket (docker logs --since <RFC3339Nano cursor|-Nm> --timestamps per container, incremental, dedup the inclusive boundary) — like Vector/Grafana Alloy — so any member of the docker group collects containers with no root (the old approach read root-only /var/lib/docker json files, which the docker group can’t read; that path is now a rootless/Podman fallback). (2) The one-liner probes actual readability first (journalctl -n0, [ -r /var/log/syslog ], docker version) and skips sudo entirely when the sources are already readable (e.g. a user already in adm/docker) → zero prompts. (3) The install self-schedules cron BEFORE the first collection and runs that first cycle detached, so a large first backfill never blocks the install; LOGRITE_TAIL_FROM_NOW=1 anchors never-seen files at EOF to bound first-run volume.

Fail-loudly (never silently under-collect). A source that still can’t read — the grant was declined, or K8s needs a DaemonSet, or syslog’s socat is missing — is not quietly skipped. The collector’s perm_state helper distinguishes active (group in this process → collect), pending (in /etc/group, not yet in this process → benign “next cycle” note), and denied (neither → record an error). Denied/dependency errors accumulate as {source, code, group, fix} and ride the next check-in’s sourceErrors array → shim → source_errors on the agent → the agent card shows a red ⚠ chip and the detail page a prominent banner with the exact copy-able fix command. A clean cycle reports [] and clears it. K8s → “deploy as a DaemonSet mounting hostPath: /var/log”; syslog dep → sudo apt-get install -y socat (or dnf/yum/apk). Ports <1024 are rejected (they need root); default 5514 is unprivileged.

Every batch is cut on a line boundary under MAX_BATCH_BYTES (4 MB, below the shim’s 5 MB limit) so an oversize backlog drains in chunks instead of 413-ing the shim forever (a latent bug the pre-2026-07 file path shared). The collector template lives at resources/collector/logrite-collector.sh.tmpl (a real, bash -n-testable file); the backend substitutes @@NAME@@ tokens single-pass at render time.

Host reconciliation (⚖, on by default). Two agents on one host watching the same source would each ship it to their own per-key index (duplicated). A shared claim registry (/tmp/logrite-collector-reconcile, sticky 1777; LOGRITE_RECON_DIR override) makes exactly one live agent own each source: an atomic mkdir <sha1(sourceid)>.claim (keyed by resolved identity — file:<host>:<dev>:<inode>, journald:…, docker/k8s per file, syslog:<host>:<port>) wins; a live owner (fresh within RECON_TTL=3×interval+60s or pid alive) is deferred to; a stale+dead claim is taken over. Claims refresh each cycle and are released on uninstall. The collector reports reconOwned/reconDeferred on each heartbeat → the agent card shows ⚖ owns/defers. LOGRITE_RECONCILE=0 is a byte-identical opt-out. Env overrides: LOGRITE_COLLECT_{JOURNALD,VARLOG,DOCKER,K8S,SYSLOG}, LOGRITE_JOURNALD_{UNITS,SCOPE,SINCE}, LOGRITE_VARLOG_PATHS, LOGRITE_SYSLOG_{PORT,PROTO,BIND}, LOGRITE_{DOCKER_ROOT,K8S_LOG_DIR,K8S_AUDIT_LOG}, LOGRITE_RECON_{DIR,TTL}, LOGRITE_MAX_{BATCH,LINE}_BYTES.

The script is headless by default — no interactive prompt (--menu shows the baked config; --loop runs one cycle for cron/tests).

Readable application names on Docker logs (2026-09-29). A container’s Docker name on this estate is an opaque lgy8d7ypqygqd2rn3glqsuht-123842360521, which nobody can search on. The same docker ps call now also reads the container’s labels — no second fork — and emits the readable name as an app field beside container. Two labels are read and the first non-empty wins: coolify.applicationName, then com.docker.compose.service. The second is not optional: a service adopted out of Coolify into docker compose is recreated without Coolify’s labels, and measured on both VMs every container had an empty Coolify label and a correct compose one — so 957 Docker documents shipped with no app field at all.

Choosing which containers are collected. sourceParams.docker.include and sourceParams.docker.exclude are space-separated shell globs matched against the container name (the Docker name, or the id when it has none). include, when set, is an allowlist; exclude then removes from what is left. Example: "exclude": "carevault-* logrite-demo-*".

• Set them at create time or with PATCH /api/v1/agents/{id} {"sourceParams":{"docker":{"exclude":"…"}}}.
• They are baked into the script as the defaults of LOGRITE_DOCKER_INCLUDE / LOGRITE_DOCKER_EXCLUDE. They are not part of live config, so after a PATCH re-run the install one-liner for the new value to apply.
• To change it on a host without a reinstall, set LOGRITE_DOCKER_EXCLUDE='carevault-* logrite-demo-*' on the collector’s crontab line; a value in the environment overrides the baked default.
• Excluded containers are skipped before the ownership claim, so another agent on the same host can still collect them.

A busy /var/log/syslog starves every source after it

Sources run in order within a cycle. On a host whose /var/log/syslog carries everything journald does — which is the case on both Logrite VMs, with ~35 containers — the /var/log pass consumes the entire cycle and the Docker pass never runs. Measured 2026-09-29: 90,462 varlog documents in ten minutes and 0 Docker documents over three consecutive cycles, while the same script with LOGRITE_COLLECT_VARLOG=0 produced 683 Docker documents across 11 applications in 27 seconds.

Nothing reports an error: .ship says 200 / 0 failures throughout, because shipping is working — it is the sources further down the cycle that never ran. If a source you enabled is simply absent from the index, suspect starvation before you suspect the source. The fix here was narrowing varlog.paths to /var/log/auth.log /var/log/kern.log and letting journald carry the rest.

Note the per-source params (/var/log paths, journald units, Docker name globs, syslog port) are baked into the rendered script and are not part of live config, so PATCH /api/v1/agents/{id} is not enough for them — re-download collector.sh on the host, or the running collector keeps the old paths. Turning a whole source on or off is live from collector 1.3.0.

It extends the existing /agents screen (no new route) and the Agent/AgentConfig scaffold — not a greenfield.

Note

Sprint 3. Storage-first: the ingest path only stores; compliance/anomaly flagging is a separate periodic job (Phase 3 — now shipped, see below). Mint-once: the token is shown exactly once and stored as a SHA-256 hash + last4 only.

OpenSearch Index Architecture

Per-Workspace Index Isolation

Each Logrite workspace gets its own dedicated OpenSearch indices. No data from one workspace can appear in another workspace’s queries.

Index Pattern	Purpose	Retention
log-analytics-{workspaceId}	Collector agent log documents	90 days
agent-flags-{workspaceId}	Compliance & anomaly flag findings	90 days

Where {workspaceId} is the numeric ID of the workspace (e.g., log-analytics-2903).

How Routing Works

1. Agent creation — the backend derives targetIndex = "log-analytics-" + workspaceId from the JWT workspace claim. The agent’s routing is set at creation time and stored in the database.
2. Ingest time — the Collector Ingest Shim resolves the agent token against the backend (/internal/agents/resolve), which returns the pre-computed targetIndex. The shim writes directly to that index.
3. Auto-provisioning — OpenSearch index templates (log-analytics-*, agent-flags-*) automatically create the per-workspace index with correct field mappings and 90-day ISM retention on the first document write. No manual provisioning is needed when a new workspace is created. The templates + ISM policy live in opensearch/exports/templates/ (log-analytics-template.json, agent-flags-template.json, collector-ism-90d.json) and are applied by import.sh step 0 — they must be applied to the cluster before the first agent doc lands, otherwise dynamic mapping makes filePath a text field (breaking the Sources aggregation). createAgent derives targetIndex = log-analytics-{workspaceId} (per-workspace, not per-agent); the off-path scanner reads log-analytics-* and routes each flag to agent-flags-{workspaceId} so the Flags panel (which reads the same tenant index) populates.

Index Template Field Notes

log-analytics-* key fields:

Field	Type	Notes
filePath	keyword	Direct keyword field — not filePath.keyword (no .keyword sub-field exists)
agentId	keyword	Stored as the numeric agent ID; OS coerces Long→string for term queries
@timestamp	date	Primary time field for all range queries
collector	boolean	true for shim-shipped docs; used to scope stats aggregations

agent-flags-* key fields (Phase 3 scanner):

Field	Type	Notes
flaggedAt	date	Sort field used by AgentService.getFlags() — not timestamp. Set once: flags are written with the _bulk create op, so a rescan no longer re-dates an existing flag (analyzer 639e749d, 2026-10-21)
@timestamp	date	Secondary time field; set once, like flaggedAt
docId	keyword	Source doc _id that triggered the flag
serviceName	keyword	Service that produced the flagged document
ruleId	keyword	Rule that raised the flag: p<policyId>.<ruleKey> (deterministic) or the judge policy id (plain-English); null for frameworks and for flags written before 2026-10-16
policyId	long	Owning policy: parsed from a deterministic ruleId, or (since 2026-10-21) the logs bundle’s nlPolicies[].policyId for a plain-English rule; null for frameworks. Part of the doc _id is the ruleId, so two rules on one line are two flags — see Rules & Policies
severity	string	The rule’s own severity, medium when it has none (2026-10-21; every flag read medium before)

Note

The filePath field being keyword (not text) means aggregations must reference it as filePath directly. Using filePath.keyword returns zero buckets — .keyword sub-fields only exist on text fields with default dynamic mapping. This was the root cause of the Sources panel always showing empty (fixed in backend PR#3, 2026-06-16).

Security Model

All agent API endpoints (/api/v1/agents) require a valid workspace JWT. The backend verifies that the requested agent belongs to the authenticated workspace before returning data. Cross-workspace access returns HTTP 403.

Architecture

collector.sh (current dir, TLS one-liner, $LOGRITE_AGENT_TOKEN)
   │  NDJSON, Authorization: Bearer lgca_live_…
   ▼
Ingest shim :9700 (logrite-ai-service)  ── the TRUST BOUNDARY
   validate+hash token → resolve {workspaceId, serviceName, targetIndex}
   from the token ONLY (never an agent-declared field) → stamp collector:true
   → bulk-write to OpenSearch → async heartbeat (backend)
   ▼
OpenSearch (log-analytics-{tenant})   ── query-time isolation by serviceName

The shim does identity/routing only; it is the trust boundary data-prepper cannot be (per-agent token→workspace resolution, mint/rotate/revoke, last-seen). v1 writes via OpenSearch _bulk over the https OS URL; a dedicated data-prepper ssl:true http source is a hardening follow-up. The _bulk write goes through the codebase’s single OpenSearch axios choke point (src/utils/opensearchAxios.ts → osAxiosConfig), so it carries the same scoped TLS trust as every other OS call — OPENSEARCH_CA_PATH (prod, verify on) / OPENSEARCH_INSECURE_TLS (dev self-signed), applied via a dedicated https.Agent, never a process-wide NODE_TLS_REJECT_UNAUTHORIZED. Hand-rolling that request (as it did until 2026-09-11) bypassed the agent and rejected a self-signed cluster cert with “unable to verify the first certificate”, and because only control-plane records skip the sink, every collector’s data silently 503’d while heartbeats stayed 200. Two silent-drop traps on this path: that TLS bypass, and a dangling index.default_pipeline (a parse pipeline lost with a cluster rebuild) — writes then 400 with pipeline does not exist and the shim drops them as permanent 4xx while advancing the cursor. Verify a live collector by CONTENT (_count rises, _search returns the doc), never by the connector’s “shipped N rows” line alone.

Ports

Service	Port	Notes
Ingest shim	9700	Container built from logrite-ai-service (Dockerfile.shim), POST /ingest — ingest.logrite.ai / dev-ingest.logrite.ai
Backend (mint/script/stats/resolve)	8080	/api/v1/agents/*, /internal/agents/*
Syslog receiver (per host, opt-in)	5514	Unprivileged default (UDP/TCP; LOGRITE_SYSLOG_PORT); <1024 rejected. Runs on the collector host, not the platform

API

Method	Path	Purpose
POST	/api/v1/agents	Mint a collector agent — response carries mintedToken + oneLiner ONCE
POST	/api/v1/agents (source:"database")	Create a database source — body carries a nested dbConnection{…}; no token/one-liner is minted (the poller is server-side). Response carries the saved connection (password stripped). See Database sources below
POST	/api/v1/agents/test-connection	Pre-create dry-run — opens JDBC with the supplied dbConnection, SELECT 1, validates the tracking column/query, probes max(cursor). Nothing persisted
POST	/api/v1/agents/{id}/test-connection	Test an existing database agent (decrypts the stored credential; persists status/error)
GET	/api/v1/agents/{id}/setup-sql?egressIp=	Copy-paste least-privilege GRANT SQL for a database agent (per its engine/db/table/user)
GET	/api/v1/agents/{agentKey}/connector	Server-rendered Node push connector for connector-mode DB agents (gated on the agent’s own token via the X-Agent-Token header — not Authorization: Bearer, which the JWT filter would reject; reads $LOGRITE_AGENT_TOKEN+$DB_PASSWORD; saved locally as logrite-connector.js)
GET	/api/v1/agents/{id}	Single agent (ownership-checked; 403 on miss/cross-tenant). Used by the detail-page deep-link instead of fetching the whole list
PATCH	/api/v1/agents/{id}	Edit capabilities — intervalMinutes (1–1440) / collectPath / collectSystemDetails / heartbeatEnabled / the system-wide sources (collectJournald, collectVarlog, collectDocker, collectK8s, collectSyslog) / retentionTier / name / description. key/tokenHash/workspaceId/targetIndex are immutable. A collector 1.3.0+ applies the change at its next check-in (within its interval); an older one needs one reinstall — see Live config at check-in
POST	/api/v1/agents/apply-defaults[?workspaceId=]	Admin/Owner. Writes the workspace’s collector.defaults.* (interval + every capability) onto every collector of the workspace; database agents are skipped. Workspace = the query param, else the caller’s JWT workspace. Answers {"updated": n}; audit action collector.applyDefaults with {updated, keys}. Each 1.3.0+ collector applies it at its next check-in
POST	/api/v1/agents/{id}/rotate	Re-mint the token (shown once); old token 401s at the shim after cache TTL
PATCH	/api/v1/agents/{id}/status?enabled=false	Disable = pause (token still resolves; shim → 403). Distinct from delete
GET	/api/v1/agents/{agentKey}/collector.sh	Server-rendered token-less bash (reads $LOGRITE_AGENT_TOKEN)
GET	/api/v1/agents/{id}/stats	24h lines/bytes from OpenSearch aggregations over collector:true docs
GET	/api/v1/agents/{id}/sources	Per-file breakdown (distinct files + per-source lines + last-seen)
GET	/api/v1/agents/{id}/system?points=N	Host telemetry — the latest type:system_metrics doc (OS, kernel, arch, CPUs, memory, disk, uptime, agentVersion); ?points adds a recent series. Fail-soft available:false when System details is off or no doc has landed. This is the only surface that exposes agentVersion
GET	/api/v1/agents/{id}/logs?page&size&hours&from&to	Individual collected log lines, newest-first, paginated — read live from OpenSearch (must_not type:system_metrics; from+size clamped to the 10k window → truncated:true). from/to (epoch-millis) select an absolute window and override hours
GET	/api/v1/agents/{id}/flags?page&size&hours&from&to&rule&severity&framework&verdict	Compliance/anomaly findings (the post-hoc scanner sink), newest-first, paginated + filterable. Carries verdict(NOTIFY)/action/framework/complianceRef/decisionLayer/reason/confidence/leakedFields/redactedSnippet, and since 2026-10-16 ruleId/policyId/policyName (the name of a policy in the caller’s workspace, else null; the Logs drawer links it to /policies/policies/<policyId>). limit is a back-compat alias for size; hours=0 = all-time; from/to (epoch-millis) select an absolute window and override hours
DELETE	/api/v1/agents/{id}?purgeData=false	Delete + revoke (host collector self-uninstalls on next check-in). Collected data is preserved by default; purgeData=true also _delete_by_querys the agent’s docs from the target index + agent-flags-{ws}. Also deletes the agent’s collector policy assignments
—	/api/v1/warden/agents/{id}/rulesets*, /api/v1/warden/agents/{id}/evaluate-log	Removed 2026-10-15. Which business-rules policies reach a collector is GET /api/v1/policies/effective?feature=logs&collectorId=; a policy’s logs feature applies to every collector unless its collectorIds narrow it. The dry run is POST /api/v1/policies/effective/test {feature:"logs", input, collectorId} → {verdict, findings, wouldBlock} (still proxied to the analyzer’s /internal/scan/evaluate, nothing persisted). Deleting an agent deletes its collector policy assignments. See Rules & Policies
GET	/internal/warden/logs/resolve?workspaceId&agentId	(internal, token-guarded) Resolve the merged bundle of the business-rules policies effective on Collector logs for this agent (no relay key) — the analyzer’s AgentLogScanner calls this
GET	/internal/agents/resolve?tokenHash=…	Shim → backend resolve (X-Warden-Internal-Token guarded). Returns the binding for a disabled agent (enabled:false) so the shim can answer 403; 404/null only for unknown/revoked
POST	/ingest (shim :9700)	NDJSON ingest, Authorization: Bearer lgca_live_…. 401 = revoked/deleted → collector self-uninstalls; 403 = disabled → collector pauses

Time-window params (shared). insights, facets, patterns, flags, and logs all accept a hours window and an optional absolute from/to pair (epoch-millis). When both from and to are supplied they win over hours (gte from · lte to), so the detail tabs can drive an exact custom range; hours alone remains an open-ended now-hours window. hours is clamped to 1h–1y (8760h); a custom span picks its own histogram interval from its length. This backs the analytics-style date dropdown on the Overview / Logs / Patterns tabs (below).

Send logs yourself: mint → copy → verify

The collector script is one client of the ingest endpoint; anything that can POST NDJSON is another. This walkthrough was run end to end on dev on 2026-10-01 (agent created, two lines shipped, read back, agent deleted).

1. Mint an agent. In the app, Agents (sidebar) → the new-agent button, which opens New Collector Agent; or with an API key (lgp_live_…) or session JWT:

curl -s -X POST https://leapfrogapi.deployd.io/api/v1/agents \
  -H "Authorization: Bearer $LOGRITE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name": "payments host", "key": "payments-01"}'

key is required — lowercase letters, digits and hyphens — and becomes the agent’s index suffix and its service on every line. The response carries agentId, targetIndex (log-analytics-<workspace>-<key>), oneLiner and mintedToken (lgca_live_…).

2. Copy the token now. mintedToken is shown once; only its hash is stored, and the app shows the last four characters afterwards. Lost it? POST /api/v1/agents/{id}/rotate mints a new one (the old one stops working within the shim’s 15-second cache). Keep it in an environment variable, not in a script:

export LOGRITE_AGENT_TOKEN='lgca_live_…'

3. Send NDJSON to POST https://ingest.logrite.ai/ingest (dev: https://dev-ingest.logrite.ai/ingest). One record per line; a line that is a JSON object is read as structured (message, level, @timestamp/timestamp honoured), any other line is taken as a plain message and its time and level are parsed from the text:

cat > batch.ndjson <<'NDJSON'
{"@timestamp":"2026-10-01T09:32:00Z","level":"ERROR","message":"payment failed: connection refused to db-01:5432"}
2026-10-01 09:32:01 WARN retrying db connection (attempt 2)
NDJSON

curl -s -X POST https://ingest.logrite.ai/ingest \
  -H "Authorization: Bearer $LOGRITE_AGENT_TOKEN" \
  -H "Content-Type: application/x-ndjson" --data-binary @./batch.ndjson
# {"stored":2,"index":"log-analytics-3103-payments-01"}

The workspace, index and service come from the token only — a service or index field in the body is not honoured. A level in the body is kept unless the message text reads as more severe.

4. Verify by content, not by the stored count alone:

curl -s "https://leapfrogapi.deployd.io/api/v1/agents/$AGENT_ID/logs?hours=24&size=5" \
  -H "Authorization: Bearer $LOGRITE_API_KEY"
# {"agentId":…,"total":2,"logs":[{"message":"2026-10-01 09:32:01 WARN retrying db connection (attempt 2)",
#   "level":"WARN","service":"payments-01",…},{"message":"payment failed: …","level":"ERROR",…}]}

The lines also appear on the collector’s Logs tab and are queryable by Cortex.

Answer	Meaning	What a client should do
200 {"stored":N,"index":…,"dropped":D,"redacted":R}	Written. dropped = lines sampled out, redacted = lines the workspace’s redaction changed — both 0 unless ingest-time redaction/sampling is on	—
200 {"stored":0,"heartbeat":…,"dropped":D,"redacted":R}	Only control records (heartbeat/check-in) in the batch, or every line was sampled out	—
200 text/plain LOGRITE_CFG_VERSION=…	A control-only batch sent with X-Logrite-Config: env — the live config reply	A 1.3.0+ collector filters and stores it
400 {"error":"empty NDJSON batch"}	No non-blank lines	Fix the request
401 {"error":"missing bearer token"}	No Authorization: Bearer	Send the token
401 {"error":"invalid or revoked agent token"}	Unknown, rotated-away or deleted agent	Stop; the collector script uninstalls itself
403 {"error":"agent disabled"}	The agent is paused	Pause and retry later
429 {"error":"rate limit exceeded","retryable":true}	Over SHIM_RATE_LIMIT_PER_MIN (default 600/min)	Back off
503 {…,"retryable":true}	Token resolver or OpenSearch unavailable	Retry the same batch

A batch is at most 5 MB (SHIM_MAX_BATCH_BYTES).

The collector script

GET /agents/{key}/collector.sh returns pure zero-dependency bash (curl + coreutils), bash -n clean. The token is never embedded — it is read from $LOGRITE_AGENT_TOKEN (baked into the mint one-liner with bash, not sh, since the script uses bashisms), so the served file carries no secret and the server stores only the hash. The script:

• is headless — capabilities (COLLECT_LOGS / COLLECT_SYSTEM / HEARTBEAT / COLLECT_PATH) are baked from the agent record (env-overridable). --menu prints the baked config; --loop runs exactly one cycle (cron/tests); --uninstall removes itself (no token required). No blocking prompt,
• tails logs: initial backfill then incremental — per-file inode + byte-offset state in ~/.logrite-collector/<key>.state (chmod 600), surviving rotation, sending only new lines, from COLLECT_PATH (default ., resolved to absolute so cron watches the same folder),
• ships a type:system_metrics host-metadata doc when System details is on,
• checks in every cycle with one auth-gated request and reads its HTTP status: 2xx → proceed; 403 (agent disabled) → pause this cycle (no data sent); 401 (token revoked/deleted) → after 2 consecutive confirmations it self-uninstalls. The check-in carries type:heartbeat when Heartbeat is on (bumps last_seen_at) or type:checkin when off (auth-only — not indexed, no last-seen bump). From 1.3.0 it also asks for its live config (X-Logrite-Config: env),
• self-schedules (cron → nohup loop; no root), carrying LOGRITE_COLLECT_PATH; the cron entry is de-duplicated by the script path so re-running never stacks duplicates,
• batches NDJSON and POSTs to the shim with the bearer.

Clean uninstall (no trace)

Deleting an agent revokes its token. On the host the collector removes itself: on its next check-in the 401 triggers do_uninstall, which deletes its own cron line (matched by script path so other agents are untouched), its ~/.logrite-collector/<key>.state + .revoked files, the state dir if empty, and the script file. For immediate removal the user can run bash logrite-collector.sh --uninstall in the install folder. Collected data is kept unless the delete opted into purgeData=true.

`--uninstall` was broken two different ways until 2026-09-29

Both were found by running it, not by reading it.

1. The script runs under set -uo pipefail, and do_uninstall dereferenced $DOCKER_CURSORS — a name assigned ~200 lines below it. So bash logrite-collector.sh --uninstall, the exact command DeleteAgentModal tells users to run, aborted on an unbound variable after it had already removed the cron line, leaving a half-removed collector.
2. With that fixed it still left <key>.ship behind — SHIP_STATUS_FILE was added after the rm list was written and never added to it — which made the rmdir of the state directory fail, so the whole directory survived under a message that says “no cron entry, script, or state remains”.

CollectorScriptRenderTest.uninstallDeletesEveryStateFileTheScriptDeclares now derives the set: it scrapes every NAME="$STATE_DIR/…" assignment out of the rendered script and asserts each is both declared above do_uninstall and mentioned inside it. A hand-written list would go stale in exactly the case it exists to catch. Verified on the real fleet: both VMs reported uninstalled — no cron entry, script, or state remains with 0 files left.

The shim classifies records by type: heartbeat is never indexed (it only bumps last_seen_at); checkin is an auth-only probe — not indexed and no last-seen bump; system_metrics is stored (it drives Inventory) but stamped byteLength:0 so it doesn’t inflate log throughput, and is never sampled; a control-only batch returns 200 {stored:0, heartbeat:<bool>, dropped, redacted} — or, when the request carries X-Logrite-Config: env, the live-config text below. getStats/getSources/getLogs all must_not type:system_metrics.

Live config at check-in (2026-10-07)

Until this release a collector ran the capabilities baked into its script at install time, so every Setup-tab edit needed the install one-liner re-run on the host. From collector 1.3.0 (AGENT_VERSION="1.3.0" in resources/collector/logrite-collector.sh.tmpl) the collector re-reads its configuration from the server at every check-in, so an edit lands within one interval with no reinstall.

What travels. The backend’s agent resolve binding (GET /internal/agents/resolve) now carries a config object built from the agent row (AgentLiveConfig): configVersion, the capability fields, and env — the exact LOGRITE_* override names the template reads:

env name	From	Value
LOGRITE_INTERVAL_MIN	intervalMinutes (null → 15)	minutes
LOGRITE_COLLECT_SYSTEM	collectSystemDetails	true/false
LOGRITE_HEARTBEAT	heartbeatEnabled	true/false
LOGRITE_COLLECT_JOURNALD	collectJournald	true/false
LOGRITE_COLLECT_VARLOG	collectVarlog	true/false
LOGRITE_COLLECT_DOCKER	collectDocker	true/false
LOGRITE_COLLECT_KUBERNETES	collectK8s	true/false
LOGRITE_COLLECT_SYSLOG	collectSyslog	true/false
LOGRITE_COLLECT_PATH	collectPath	only when absolute and filter-safe; a relative path (or the unset .) is left out because a cron run has no install directory as its cwd

A null capability column is sent as false, the same default the rest of the page documents. configVersion is the first 12 hex characters of SHA-256 over the key-sorted JSON of env, so it changes exactly when what the collector would apply changes. Any value outside [A-Za-z0-9_./,:@*=-]* is dropped on the backend.

Kubernetes is LOGRITE_COLLECT_KUBERNETES in live config

The line filter (shim and collector) allows only [A-Z_] in names, so LOGRITE_COLLECT_K8S would be dropped in transit. Live config uses the digit-free LOGRITE_COLLECT_KUBERNETES; the template reads it first and still falls back to LOGRITE_COLLECT_K8S for environments set by hand.

The exchange, cycle by cycle (run_cycle):

1. config_load sources the config the previous check-in delivered (~/.logrite-collector/<key>.config.env, mode 600) after the baked defaults — unless LOGRITE_LOCAL_OVERRIDE=1 is set in the collector’s environment.
2. The check-in POSTs its control record with the header X-Logrite-Config: env and writes the reply body to a scratch file; the HTTP status still drives revoke/pause exactly as before.
3. The shim answers a control-only batch carrying that header with text/plain: first LOGRITE_CFG_VERSION=<configVersion>, then one NAME=value line per env entry, sorted by name. Any line not matching ^LOGRITE_[A-Z_]+=[A-Za-z0-9_./,:@*=-]*$ is dropped by the shim.
4. On a 2xx, config_apply keeps only lines matching that same regex and additionally rejects LOGRITE_AGENT_TOKEN, LOGRITE_INGEST_URL and LOGRITE_LOCAL_OVERRIDE — the token, the ingest URL and the override switch can never be set from the server. The body is filtered, never executed. An older shim answers JSON, which the filter discards.
5. If LOGRITE_CFG_VERSION differs from the stored file’s, the filtered file is swapped in and the collector logs config updated (<version>); the new config is loaded in the same cycle. Same version → nothing changes.
6. If the interval changed and the collector is cron-scheduled, its crontab line is rewritten (schedule updated (every Nm)). A collector running under the nohup fallback keeps its install-time sleep.

Measured on dev (2026-10-07, a throwaway collector 1.3.0 on the dev VM): PATCH collectDocker=true → the next cycle logged config updated (c7ef0b7527b6) and .config.env held LOGRITE_COLLECT_DOCKER=true. Docker collection ran, but every container was already claimed by that host’s existing collector (reconciledDeferred: 31) — one agent ships each container, by design. PATCH intervalMinutes 15 → 5 → config updated + schedule updated (every 5m) and a */5 crontab line. The collector was uninstalled afterwards and the crontab was byte-identical to before.

Because the file is sourced after the defaults, server values win over LOGRITE_* variables set on the host for the names in the table above. To pin a host to its baked + environment configuration, set LOGRITE_LOCAL_OVERRIDE=1 where the collector runs: the file is still fetched, stored and logged, but never sourced. Per-source params (journald units, /var/log paths, Docker globs, syslog port) are not part of live config and stay baked.

Collectors older than 1.3.0 need one reinstall

A 1.2.x script does not send the header and ignores the reply, so it keeps running its baked config. Re-run the install one-liner once; after that every edit is live. The agent’s Setup → Host / System panel shows the badge “Reinstall once to enable live config” next to any collector reporting an agentVersion below 1.3.0 (unknown versions show no badge). The one-time sudo group step lives only in the install one-liner, so switching on journald, /var/log or Docker live on a host whose user lacks the group still needs that grant — the source reports it through the existing fail-loudly sourceErrors chip with the fix command.

Setup tab toast after a capability save (configSavedMessage):

Agent	Toast
Collector 1.3.0+ (or version not yet reported)	“Configuration saved. Applies at the collector’s next check-in (within N min).”
Collector older than 1.3.0	“Configuration saved. This collector predates live config — reinstall it once to apply changes.”
Database poller	“Configuration saved. It applies automatically on the next scheduled poll.”
Database connector	“Configuration saved. Re-download and re-run the connector for it to take effect.”

Workspace defaults. A new collector takes the workspace’s collector.defaults.* (Workspace settings) for any capability or interval the create request leaves out, and the New Collector Agent modal pre-fills from them. Editing the defaults does not touch existing collectors; Settings → Log ingestion → Collector defaults → Apply to existing collectors calls POST /api/v1/agents/apply-defaults (Admin/Owner; database agents skipped) and toasts “Defaults applied to N collectors. Each applies them at its next check-in.”

Ingest-time redaction and sampling. The same resolve binding carries the workspace’s ingestPolicy, which the shim applies to every line a collector ships — masking values and sampling routine levels before anything is stored. See Ingest-time redaction and sampling.

Data model

agents gains: workspace_id, token_hash (SHA-256), token_last4, source, interval_minutes, target_index, retention_tier, last_seen_at (migration 2026-collector-agents.sql; key stays UNIQUE), plus the capability columns collect_logs, collect_system_details, heartbeat_enabled, collect_path (migration 2026-agent-capabilities.sql, all nullable — read with the documented default: collect_logs null → true, others null → false), and the system-wide source columns collect_journald, collect_varlog, collect_docker, collect_k8s, collect_syslog + a source_params JSONB blob (per-source params: journald units/scope, /var/log paths, syslog port/proto) + reconciled_owned/reconciled_deferred (host-reconciliation counts) — migration 2026-system-sources.sql, all nullable — and a source_errors JSONB list ({source,code,group,fix} the collector reports each check-in so the UI can show the fail-loudly banner; migration 2026-collector-source-errors.sql, nullable, empty/clean cycle clears it). collectorBash’s signature collapsed to a CollectorCaps record so a new source never grows the arg list; params are sanitized (sanitizeSourceParams, the same reject-set as collectPath + clampSyslogPort) before baking. The editable profile (interval/index) also lives in AgentConfig JSONB.

Unified Business Rules (Phase 1): warden_rule_set gains an applies_to JSONB column (["relay"] default / ["logs"] / both) that tags which plane a rule set evaluates on, and a new agent_rule_set join (agent_id, rule_set_id) records per-agent logs attachments on top of the workspace-wide appliesTo:["logs"] sets (migration mirror 2026-warden-ruleset-applies-to.sql; the applies_to backfill runs in WardenRuleSetAppliesToMigrator at boot since Hibernate ddl-auto doesn’t apply SQL defaults). The agent-flags-* index template adds verdict/action/framework/complianceRef/decisionLayer/reason/confidence/scanScope/ruleSetId (all keyword/text — older docs project null). See the AI Warden page for the shared rule model.

Superseded 2026-10-15

applies_to and agent_rule_set were migrated into policies (logs feature; agent_rule_set rows became collector assignments) by PolicyUnifyMigrator. Collectors have no project, so a policy limited to Selected projects never applies to collector logs.

System details feed Inventory Visibility via AgentInventorySyncService — a @Scheduled backend job that reads the latest type:system_metrics doc per enabled agent and upserts a host asset (source:"agent", hasLogging:true) through the existing AssetInventoryService, lifting the maturity scorer’s Inventory coverage %.

Frontend

The /agents screen is a table (AgentsTable.jsx) — collectors and AI-assistant agents in one list with a Type column; collector-only columns (Capabilities, Source/Path, Interval, Last seen) show ”—” for assistants. Columns: Name (+key) · Type · Status · Capabilities (🗂️/💓/🖥️/📓/📁/🐳/☸️/📡 + a ⚖ owns/defers reconciliation chip) · Source/Path · Interval · Last seen · Actions (Configure (AI-assistant agents only — collectors have no relay-style config) · Enable/Disable · Delete/Uninstall). Sortable by Name/Status/Last seen, paginated (common/Pagination), with the existing search/Ctrl-K and the New Collector Agent button preserved.

The New Collector Agent modal leads with what you get (a locked “collect logs where you run it” row + an optional “watch a specific folder” field) and capability toggles (Heartbeat, System details, and the system-wide sources 📓 journald / 📁 /var/log / 🐳 Docker / ☸️ Kubernetes / 📡 Syslog — each with its M-26-14 element chip, a permission/dependency prereq note, and an inline params sub-panel shown when enabled: journald units + scope, /var/log path list, syslog port + UDP/TCP), with name/interval/retention/index tucked under an Advanced accordion (name is optional, auto-defaulted). On mint it shows a single big Copy install command button (the token is embedded in the one-liner; raw token behind a disclosure), a capability recap, and a live verification line that flips ”⏳ waiting for logs” → ”✅ active” by polling /stats.

The detail view is tabbed (AgentDetail.jsx branches collector agents into CollectorAgentDetail.jsx; AI-persona agents keep the original layout), modeled directly on the AI Warden / API Relay page. A persistent hero + KPI strip (Last seen · Status · Lines 24h · Bytes 24h · Findings) sit above a pill tab bar; the active tab is in the URL (?tab=) for deep-linking. The hero’s primary action is Chat about logs (see below). The Findings KPI tile is a shortcut — clicking it deep-links to the Logs tab pre-filtered to Flagged (?tab=logs&flag=1). There are three tabs (the former Findings, Sources, and System tabs were folded in to mirror the relay’s lean layout):

• Logs — AgentLogsTable.jsx, the main “what actually happened” table. Columns: Time · Verdict (VerdictBadge Clean/Flagged) · Level (uses the collector’s real level field where present — journald/syslog/db — falling back to a regex on the message) · Framework (TemplateChip) · Severity (StatusBadge) · Source · Service · Message · copy-row, over a selectable window with pagination. The Source cell is a rich SourceCell — a per-source emoji icon + a colored type badge + the real identifier: 🐳 container name, 📓 journald unit, 📁 /var/log file, ☸️ namespace·pod·container, 📡 app@host, 🗄️ db table, 🗂️ file path (tooltip = full filePath). This is powered by the enriched GET /agents/{id}/logs projection, which now also returns sourceType/filePath/container/unit/namespace/pod/stream/appName/service/level. An All / Clean / Flagged filter folds the old Findings tab in exactly as the relay folds Violations into its Overview: All & Clean render /logs and join agent-flags by docId (a docId → finding[] map) to populate the compliance columns per flagged line; Flagged renders /flags server-side (with framework + severity sub-filters) so the flagged view is complete + paginated. A row click opens LogDetailDrawer.jsx — a structured source-metadata grid (source badge + identifier · service · host · container/unit/pod/stream · path) plus every finding against it (decisionLayer/action/category/framework/severity/confidence/leakedFields/remediation/complianceRef/redacted snippet) with a Mark as false positive suppress action. An empty window shows an animated “waiting for logs” stream (shimmer rows + a live pulse showing the ship interval) that auto-polls every 15s until the first lines land.
• Business Rules — CollectorBusinessRulesTab.jsx (2026-10-15): lists the business-rules policies effective for this collector with Edit (the shared policy editor drawer); “Add a policy to this collector” picks a business-rules policy and adds the logs feature (adding this collector to collectorIds only when the policy’s logs are already narrowed); “Remove from this collector” appears only on narrowed policies that list it, otherwise the row reads “Applies to all collectors — narrow it in the policy”. The paste-a-log dry-run console (LogDryRunConsole.jsx) now calls POST /api/v1/policies/effective/test.
• Setup — SetupTab.jsx: connect/install + rotate, editable Capabilities (PATCH /agents/{id}; the panel says edits apply at the next check-in, and that collectors older than v1.3.0 need one reinstall first), Encryption & Retention, M-26-14 contribution, the Sources per-file breakdown (/agents/{id}/sources), Host / System telemetry (/agents/{id}/system, fail-soft empty state when System details is off), and the Danger Zone. This tab self-fetches its Sources + System panels. For database agents it also renders two edit panels backed by applyDbConnectionPatch, so a query or connection change no longer forces a delete-and-recreate (the agent keeps its cursor/history):
  • Pull source — edit the table ⇄ custom query, tracking column/type, message column; changing the cursor surfaces a re-seed-watermark field. A poller applies it on the next scheduled poll; a connector (push) must re-download + re-run its script.
  • Connection — repoint the agent (host/port/database/username/SSL) or rotate the credential. engine is immutable (a different driver = recreate). For a poller the password is required to apply a change (re-enveloped AES-256-GCM, never echoed) and a Test connection button dry-runs the proposed target via POST /agents/test-connection before saving; for a connector the DB password stays in the customer network ($DB_PASSWORD) so there’s no password field or server-side Test — the new target is baked in on the next connector download. Both save only changed dbConnection fields (null/absent = untouched).

Delete opens DeleteAgentModal — it explains the host self-uninstall, offers a copy-paste --uninstall command, and has an opt-in “Also permanently delete all collected data” checkbox that drives DELETE ?purgeData=true (default keeps the data).

Shared time-range dropdown (Overview / Logs / Patterns). All three data tabs use one selector — AgentTimeRange.jsx — that mirrors the analytics (Discover) date dropdown exactly: the same preset list (Last 15 Minutes → Last 1 Year) plus a Custom Dates entry that opens our own CustomDatePicker (Relative + Absolute tabs, backed by LogriteDatePicker — never the native browser calendar). The selected option is resolved to { hours, from, to } by resolveWindow(): presets/relative ranges become a window ending now (recomputed each fetch so live refresh keeps sliding); an absolute custom range carries fixed from/to. Those params flow to insights/flags/logs/facets/patterns — sub-hour windows (15/30 min) are now exact because they ride from/to rather than an integer hours. The Overview’s live auto-refresh is visibility-gated (paused while the browser tab is hidden, one immediate refresh on refocus) and aborts the prior round’s in-flight insights/flags calls (AbortController) so a slow backend can’t leave overlapping requests racing.

Chat about logs (per-agent Cortex)

The hero’s Chat about logs button opens the Cortex panel scoped to this agent’s own index. This matters because of an index seam: the universal Cortex chat queries the shared tenant index log-analytics-{clientId}, but each collector writes to a per-agent index log-analytics-{workspaceId}-{key} — so the universal chat never sees collector data. The button calls openPanel(agent.key, agent.name, { indexName: agent.targetIndex }); the scope threads through AgentChatContext → CortexPanel → useChatLogic, which pins indexName on every message (collectors’ serviceName is mapped as a plain keyword, so it deliberately does not also send projectName — that would add a serviceName.keyword filter that matches nothing). The collector ingest shim stamps level/severityText/service on each document so Cortex’s severity/service filters work on raw collector lines. Ask concrete questions (“Were any SSNs or card numbers logged?”, “Were any errors logged?”) — these reliably trigger a scoped log search.

Phase 3 — periodic scan + maturity wiring

• Rule-driven post-hoc scan (Phase 1, unified Business Rules). AgentLogScanner in the analyzer (main) runs an opt-in (AGENT_SCAN_ENABLED=true) periodic scan over collector:true documents. It is off-path and read-only — it never mutates the stored documents. Instead of a fixed leak-pattern sweep, it resolves each workspace’s logs-channel rule bundle from the backend (GET /internal/warden/logs/resolve, LRU-cached) and evaluates every log line against the SAME business-rules policies AI Warden uses (those with the logs feature), across three NOTIFY-only layers: (1) framework/compliance (deterministic leak patterns), (2) deterministic custom rules (generic regex / prompt-injection, when rulesEnabled), and (3) the Groq NL AI-judge (plain-English rules — gated on NL policies present + sample-rate, cheap-model ladder, 30-min cache). The verdict is derived from the composed action (never BLOCK). Findings persist to the per-workspace agent-flags-{ws} index with _id = docId:layer:rule (idempotent across layers) carrying verdict/action/framework(canonical HIPAA/SOC2/M-26-14/PCI/GEN)/complianceRef/decisionLayer/reason/confidence. A workspace with no logs business-rules policy produces no findings (coverage is opt-in: a policy with the Collector logs feature in Rules & Policies). The vendored evaluators (complianceCheck.ts/ruleEngine.ts/judge.ts/librarySnapshot.ts/config.ts/types.ts/leakPatterns.ts) live in the analyzer’s src/warden/ and must stay in sync with ai-service. The same evaluators back POST /internal/scan/evaluate (the dry-run console). Ops: the updated agent-flags-* index template (keyword mappings for framework/verdict/…) must be applied via import.sh before new findings land, or dynamic mapping makes them text and the framework/verdict term-filters return zero.
• OpenSearch monitors (shipped). opensearch/exports/monitors/agent-monitors.json (imported by import.sh step 4b) adds two query-level monitors → Discord: Compliance Flag Raised (any agent-flags-* finding in 15m) and Ingest Silence (no collector:true docs in 30m).
• Active-agent coverage → the M-26-14 scorer (shipped + live). An active, heartbeating agent now feeds MaturityScoringService — Collection Coverage / Centralization and the Inventory live signal.
• CDM/HWAM/SWAM connector (shipped). POST /api/v1/inventory/assets ingests an asset feed (upsert by asset id); Inventory Visibility then derives from the % of inventoried assets shipping logs. See the M-26-14 page.
• data-prepper ssl:true http source (code-complete). agent-ingest-pipeline.yaml.example + gen-agent-ingest-certs.sh (self-signed pair) + the shim repoint toggle COLLECTOR_DATAPREPPER_URL (posts docs as a JSON array to the TLS http source instead of OS _bulk; empty = the OS _bulk-over-https fallback). Only dropping in CA-signed certs + flipping the env remains.

OPS — operational / system-health rules (ops framework)

A first-class compliance-library framework, ops, that flags operational signals (not privacy/compliance) in collector logs so admins / SRE / security can triage the box. Seeded by ComplianceLibrarySeeder as ~55 signature rules — whole-message regexes with no field tokens (valuePatterns only, variableTokens: []), which matchPattern already fires on — grouped by category: app_crash (Go/Java/Python/Node crashes, segfaults, fatal), oom (kernel/cgroup/container OOM-kill), auth (SSH failed-password / invalid-user / PAM / sudo / break-in / account change / log tamper), service (systemd start/exit/failed-state/crash-loop, docker exit, health probe, K8s CrashLoopBackOff), disk (ENOSPC, read-only remount, fs corruption, I/O errors), network (connection refused/timeout, DNS, cert expiry, TLS handshake), http (5xx/upstream, 429), permission (EACCES, SELinux/AppArmor), kernel (MCE, hung task, oops, lockup, panic), resource (open-files, ENOMEM, pool/thread/conntrack exhaustion), database (deadlock, too-many-connections, slow query, replication, recovery, auth), config (bad config, NTP skew). Every rule carries a severity (critical/high/medium/low) and an operator-facing remediation (the rule hint), and is NOTIFY-only.

• Delivery. The rules live in the versioned compliance library (compliance_framework_version / compliance_rule_definition) and reach the analyzer through the existing /internal/warden/library/snapshot poll — no relay/analyzer redeploy needed to publish new ops rules. frameworkForRule maps ops_* → OPS; findings persist to agent-flags-{ws} with category + remediation (carried as reason).
• Enable it. Turn on the OPS framework for Collector logs (Rules & Policies → Frameworks, or the “System Health & Security” starter template in the Policies template gallery, frameworks: ["OPS"]), workspace-wide or narrowed to chosen collectors, and the whole ops signature set activates for those agents’ logs. OPS is a selectable framework in FrameworkPicker and has its own teal TemplateChip.
• Scope. Rules are scanScope: request (a log line) and relayApplicable: true so the log scanner runs them; the signatures are specific enough (panic:, Out of memory: Kill, Failed password for …) that they never match normal relay traffic. Windowed rate/burst escalation (e.g. “N failed logins in 5m → higher severity”) is a documented follow-on — the per-line rule still fires today.

Database sources (agentless server-side poller)

A Database source is a second collector kind, chosen via the [ Local | Database ] tab group at the top of the New Collector Agent modal. Instead of installing a script on a host, the user hands Logrite read-only DB credentials and points it at a log table (or a custom SELECT); a resident backend poller reads new rows on a schedule and writes them into the same collector:true envelope + per-agent log-analytics-<ws>-<key> index as the file collector — so the whole read side (Logs / Findings / Sources / Stats, the Warden AgentLogScanner, and Chat about logs) is reused unchanged.

Agentless, no token. A database agent mints no lgca_live_ token and never touches the shim. The poller (DbSourcePollingService) is trusted: it reads the authoritative workspaceId/targetIndex straight off the immutable agent row. Token rotation therefore returns 409 — the DB credential is rotated via PATCH /{id} with dbConnection.password.

How it polls (industry-standard rising-column watermark)

Each cycle (interval_minutes), DbSourcePollingService decrypts the credential in memory, opens a verify-full TLS JDBC connection, and runs SELECT … FROM <table> WHERE <cursor_col> > :watermark ORDER BY <cursor_col> ASC LIMIT n, advancing the watermark only on a clean OpenSearch write (a transient failure just retries next cycle). Same pattern as Splunk DB Connect / Logstash jdbc / Fluentd sql. Details:

• First poll with no cursor seeds the watermark to the current MAX and emits nothing (“from now”); an initial cursor of 0 backfills everything.
• Numeric cursors use strict > (unique monotonic id — recommended). Timestamp cursors use >= + a deterministic _id (row-content hash) so boundary rows are re-read idempotently rather than lost.
• A per-agent in-flight guard means one slow/hung DB never stacks polls or blocks others.
• Disable = pause (poller skips; cursor preserved). Delete cascades the connection row + secret. Rows get the same post-hoc Warden scan as file logs (analyzer sweep / optional collector.db-poll.scan-url trigger).
• Engines (v1): PostgreSQL, MySQL, SQLite (drivers already bundled). SQL Server / Oracle / Mongo validate-reject until a driver + dialect is added.

Security posture

• Credential at rest — the DB password is stored ONLY as an AES-256-GCM envelope (per-row DEK wrapped by WARDEN_KMS_MASTER_KEY, tenant-bound via GCM AAD ws:<workspaceId> — a wrapped DEK stolen from one tenant can’t unwrap in another), reusing WardenCryptoService. password_last4 is the only non-secret tail; the four password_* columns + ca_cert/tunnel_config are @JsonIgnore. Fail-closed: set WARDEN_KMS_REQUIRE_MASTER_KEY=true in prod so the app refuses to boot on the insecure dev-fallback key.

• Injection-safe (DbCatalogService) — identifiers (table/column) can’t be parameterized, so they’re allow-listed ([A-Za-z_][A-Za-z0-9_]*, optional schema.table) and quoted per engine ("…" / `…`); every value (the watermark) is a bound ?; a custom query must be a single read-only SELECT containing {cursor} (rewritten to a bound param) and runs on a read-only connection with a statement timeout. Driver exceptions are scrubbed of the DSN + password before they reach logs/UI/audit.

• Least-privilege DB user — ship the customer a SELECT-only user scoped to one table/view:

  -- PostgreSQL (prefer a redacting VIEW so PII never crosses the wire)
  CREATE ROLE logrite_reader LOGIN PASSWORD '…' NOSUPERUSER NOCREATEDB NOCREATEROLE
      CONNECTION LIMIT 4 VALID UNTIL '2026-10-01';
  GRANT CONNECT ON DATABASE app_prod TO logrite_reader;
  GRANT USAGE ON SCHEMA audit TO logrite_reader;
  GRANT SELECT ON audit.app_logs TO logrite_reader;   -- one table/view only

  -- MySQL (user@host is itself an allowlist for our static egress IP; require TLS)
  CREATE USER 'logrite_reader'@'<our-egress-ip>' IDENTIFIED BY '…' REQUIRE SSL
      WITH MAX_USER_CONNECTIONS 4 PASSWORD EXPIRE INTERVAL 90 DAY;
  GRANT SELECT ON app_prod.app_logs TO 'logrite_reader'@'<our-egress-ip>';

  Never pg_read_all_data / db_datareader / SELECT ANY TABLE.

• Connectivity — verify-full TLS by default (weaker modes warn in the UI); reach private DBs via IP-allowlist + our static egress IP, an SSH bastion, or PrivateLink. Per-tenant isolation is structural: workspaceId is JWT-derived and targetIndex is workspace-fenced.

Data model & config

agents.source = "database" (no token columns). The connection lives 1:1 in a new agent_db_connections table (migration 2026-db-source-agents.sql; created on boot by ddl-auto): engine/host/port/database/username, ssl_mode, the four password_* envelope columns + password_last4, log_table XOR log_query, tracking_column/tracking_type, message_column, cursor_value, last_poll_at, last_row_count, consecutive_failures, status (pending|active|error|disabled|revoked), error (sanitized). The poller is a resident @Scheduled bean (no new process):

Env var	Default	Notes
COLLECTOR_DB_POLL_ENABLED	true	master switch for the poller
COLLECTOR_DB_POLL_TICK_MS	60000	coarse scheduler tick
COLLECTOR_DB_POLL_BATCH_LIMIT	500	rows per poll (bounds prod-DB load + backfill)
COLLECTOR_DB_POLL_SCAN_URL	(empty)	optional near-real-time Warden scan trigger
COLLECTOR_DB_POLL_TIMESTAMP_OVERLAP_SECONDS	5	re-read window for timestamp cursors (out-of-order safety)
COLLECTOR_DB_POLL_ALERT_WEBHOOK	(empty)	per-agent failure Discord webhook (empty = off)
COLLECTOR_DB_POLL_ALERT_THRESHOLD	3	consecutive failures before a “down” alert
COLLECTOR_DB_POLL_EGRESS_IP	(empty)	our static egress IP, rendered into the per-agent setup SQL
WARDEN_KMS_REQUIRE_MASTER_KEY	false	set true in prod → fail-closed on a missing master key

Hardening (shipped)

• Tamper-evident access audit — every outbound DB access (each poll cycle + existing-agent test) appends one SHA-256 hash-chained row to agent_db_access_log (per (workspace, agent), single-writer via a striped lock so it never forks). It records the connection target + query shape (a fingerprint) + rows-read + cursor before/after + outcome — never SQL text, values, DSN, password, or row content. AgentDbAccessLogService.verifyChain(ws, agentId) detects any post-hoc mutation.
• Per-agent failure alerting — on threshold consecutive poll failures the poller fires one “down” Discord embed (config-gated COLLECTOR_DB_POLL_ALERT_WEBHOOK) and one “recovered” when it clears — no spam (the monotonic consecutive_failures counter is the dedup). The fleet-wide Ingest Silence OS monitor remains the backstop.
• Engine-aware column validation — test-connection verifies the tracking/message column actually exists via information_schema.columns (pg/mysql) or pragma_table_info(?) (sqlite), returning code:"COLUMN_NOT_FOUND". This closes SQLite’s double-quote-as-string-literal quirk, where a bad column previously passed silently.
• Timestamp overlap window — timestamp cursors re-read from watermark − COLLECTOR_DB_POLL_TIMESTAMP_OVERLAP_SECONDS so rows committed out of timestamp order (clock skew, long transactions) aren’t lost; the deterministic _id upsert dedups the re-read at zero cost.
• Least-privilege setup SQL generator — GET /api/v1/agents/{id}/setup-sql[?egressIp=] returns copy-paste SELECT-only GRANT SQL scoped to the agent’s engine/db/table/user (never pg_read_all_data/db_datareader).

Connector mode (push) — for private/unreachable DBs

When a customer’s DB can’t be reached from Logrite’s egress, create the agent with dbConnection.connectorMode:true. The agent then mints a token (like a file collector) but keeps source:"database", stores no DB credential server-side, and is excluded from the server poller (a DB agent with a tokenHash is push). GET /api/v1/agents/{key}/connector (gated on the agent’s own token; saved locally as logrite-connector.js) serves a Node connector (pg+mysql2) that runs the same rising-column watermark poll inside the customer network and pushes NDJSON to the existing shim /ingest — the DB password (from $DB_PASSWORD) never leaves their network. It reuses the token → workspace/index routing and the 401(stop)/403(pause)/503(retry) + heartbeat/checkin control machinery. The connector sends the deterministic rowId, and the shim now honors a client-supplied rowId as the OpenSearch _id, so push re-sends UPSERT in place — the same at-least-once idempotency the server poller has (file collectors, which send no rowId, are unchanged).

Running it. node logrite-connector.js --loop runs one cycle and exits — that is the form for cron, despite the name; with no flag it runs for ever on intervalMin. Env: LOGRITE_AGENT_TOKEN, DB_PASSWORD, optional DB_HOST/DB_PORT/DB_NAME/DB_USER overrides, LOGRITE_INITIAL_CURSOR (empty = from now, 0 = backfill, or a timestamp such as 2026-10-15 00:00:00), LOGRITE_STATE_DIR, and DB_TIMEZONE.

Timestamp cursors and event time (fixed 2026-10-15, backend 055ccb92). Two defects, both found on the ERPNext collector:

• The newest row re-shipped every cycle. mysql2/pg turned DATETIME(6)/timestamp into a JS Date, which holds milliseconds, so the saved cursor (…12.365) sat below the row’s real value (…12.365584) and > {cursor} matched it again. The rowId upsert meant no duplicate documents — the symptom was a shipped 1 rows line on every run. Timestamps now come back as the server’s own text, so the cursor is exact. A cursor file written by an older connector (ISO, ending Z) is converted back to the wall clock it came from; the newest row re-ships once under its original rowId (date columns are hashed exactly as before), then never again.
• Every row was filed under ingest time. The connector now sends the tracking column as @timestamp when trackingType is timestamp. A zone-less value is read in DB_TIMEZONE (an IANA name such as Asia/Kolkata, or +05:30), defaulting to the host’s zone. Set it whenever the application writes a zone other than the host’s: ERPNext writes its site zone while MariaDB’s own @@time_zone says UTC, so neither the database nor the host can answer it. A wrong zone shifts events by hours; a future time is clamped to ingest time by the shim.

Caution

A connector-mode agent must be created with dbConnection.connectorMode: true. Without it the agent is a pull agent with no token, and rotate answers 409 — it cannot be switched afterwards; delete and recreate. dbConnection accepts the database as database or databaseName (the spelling every response uses), and a connector agent keeps a requested heartbeatEnabled on create — both were silently dropped before backend 88aa4507.

Roadmap (deferred): CDC/WAL streaming (Debezium) for true DELETE/UPDATE completeness — scoped as a separate dedicated-worker epic (embedding it in the API is unsafe: an abandoned replication slot can fill a customer’s disk). See planning/db-collector-cdc-epic.md. Also: per-tenant KEK migration to Vault/KMS.

Deferred (v1)

Shipped since v1: custom-folder collection, host system-metrics, explicit heartbeat, Database sources, and (2026-07) the system-wide host sources — journald, /var/log, Docker, Kubernetes, and a syslog receiver — plus host reconciliation (one owner per source across agents). These are no longer “coming soon”; they run whenever the collector has read access (degrading gracefully with a note: + skip otherwise — see Collection capabilities above). journald//var/log did not require abandoning the no-root design: they read what group membership grants and skip the rest.

Still deferred: Cloud control-plane (CloudTrail / GCP Cloud Audit / Azure Activity) as an agentless per-agent poller ☁️ Cloud tab (mirrors the Database source); Local Claude folders; and the third-party CDM/HWAM/SWAM data-format adapters.

Security hardening (Jul 2026 audit)

• Ingest shim — a single malformed NDJSON line (e.g. null) can no longer crash the shim: non-object lines are coerced to a log line and the handler runs under an async error boundary + process-level guard. _bulk partial failures distinguish transient (retry) from permanent (drop) so one poison doc can’t head-of-line block a collector forever. A per-source-IP cap (SHIM_RATE_LIMIT_PER_MIN, default 600) blunts fresh-random-token amplification.
• Agent keys are per-workspace — the user-chosen key is unique within a workspace (not globally), so one tenant can’t squat a name or probe another tenant’s keys. The collector.sh / connector download URLs are keyed by an unguessable publicId, not the guessable key, so they can’t be enumerated. Existing installs keep working; re-copy the one-liner from the UI to move to the new URL.
• GET /agents/{id}/logs redacts server-side — raw collected lines are masked (secrets, SSN, card, email, sk- keys) for any member below Maintainer; un-redaction is role-gated, not a client-only toggle.
• AgentsFlowController is authenticated — its wizard endpoints previously ran with no auth under the /api/v1/** permit-all; they now require a JWT and verify workspace membership.
• DB-source connector — the JDBC database field is allowlist-validated (blocks connection-property injection), read-only cursor queries reject data-modifying CTEs, and the sqlite path is traversal-guarded.
• Delete + purge now propagates an OpenSearch failure instead of returning a false 204; host-inventory assets are namespaced by workspace so two tenants’ identically-named hosts don’t collide.

New / changed environment

Var	Default	Meaning
SHIM_RATE_LIMIT_PER_MIN	600	Per-source-IP ingest cap on the shim (0 disables).

An end-to-end harness, with numbers (2026-10-01)

The collector is headless: a clean cycle exits 0 and prints nothing. That is the right default for cron and the wrong one for answering “did it work”, so LogriteMasterControl/ops/testing/loadgen/ holds a two-file, dependency-free harness — a service that writes one JSON line per request into a watched directory, and a runner that issues 100 requests paced across a minute.

Measured on dev:

Check	Value
requests issued / reached	100 / 100, elapsed 62s
log file	101 lines / 20,769 bytes
collector cursor	194 → 20,769 — exactly the file size
ship status	200, 0 failures, .srcerr []
documents in OpenSearch	101
by level	78 INFO / 13 WARN / 10 ERROR, matching the runner’s 77+1 / 13 / 10

The level breakdown is the part that carries weight. “101 documents arrived” would be equally true if every line had landed as opaque text; the per-level counts matching the HTTP codes the runner saw is what says the fields survived the trip.

Read the state files, not stdout — and level is a text field

A silent cycle is a successful cycle, so the evidence lives in ~/.logrite-collector/<publicId>.*: .state is <inode> <offset> <sha> and its offset must equal wc -c of the log file; .ship is <epoch> <status> <failures>; .srcerr is [] when clean; .recon is <owned> <deferred>.

⚠ level is mapped as text, so a terms aggregation on it fails with “Text fields are not optimised for operations that require per-document field data”. Count per level with a match query. The error reads like a broken index and is a mapping choice.

What the harness proves about notifications, and what it cannot

Only discord.com hosts pass the webhook URL check, so a local sink cannot stand in. Using a syntactically valid but deliberately non-existent webhook makes the attempt observable without posting into a real channel:

Event	Observed
agent aged past its stale threshold	notified_health_state ok → offline, POST → 404
agent checks in again	offline → ok, a second POST one minute later
agent held healthy for three further sweeps	last_delivery_at unchanged — no repeats

A healthy agent shipping logs is not a notification

Notifications are about transitions, not states — keyed on state alone a dead host would post once a minute for ever, which is how a channel teaches its readers to mute it. So successful ingestion produces no Discord message, and its absence is correct behaviour rather than a fault. What notifies is stopped-checking-in, recovered and ship-failing.

A test agent left enabled will flip to offline an hour later and page a real channel about a host nobody cares about. findEnabledCollectors() filters on enabled = true, so disabling it takes it out of the sweep.

Close-out changes (2026-10-09)

• collectPath is refused when it contains quotes, backticks, $, backslashes or control characters. The MCP tool warns when a path does not look like a plain absolute folder (no leading / or drive letter, or contains ;, |, &): the agent collects nothing from a folder that does not exist.
• Connector-mode database agents cannot be tested from Logrite: test_connection answers a validation error explaining that the agent reaches its database through the in-network connector on your host; the connector reports dbConnection.status and lastSeenAt on each run.
• Reveal. Log and flag reads return revealRequested and revealed; un-redaction needs an elevated (Maintainer or above) member through a session or connected client — an API key cannot.
• OTLP. Any enabled agent’s token also authenticates OpenTelemetry exports — see Sending OpenTelemetry data.
• Cortex has a get_collector_health tool (last check-in, interval, enabled, unhealthy agents).