Monitoring

Health Check Endpoints

Host ports on the VM (ssh dev / ssh prod); one instance per estate.

# Logrite services
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9800/health   # Backend: no open /health — 302 to /login means up
curl http://127.0.0.1:9001/health          # Analyzer (9001 on dev, 9000 on prod)
curl http://127.0.0.1:9100/health          # Semantic Search
curl http://127.0.0.1:9300/health          # AI Service
curl http://127.0.0.1:9500/health          # MCP Server
curl http://127.0.0.1:9999/health          # Webhook Server

# OpenSearch — https, self-signed, authenticated
curl -sk -u "$OS_USER:$OS_PASS" https://127.0.0.1:9200/_cluster/health
curl -s -u "$OS_USER:$OS_PASS" http://127.0.0.1:5601/api/status   # Dashboards (401 without credentials)

Services status

# on the dev VM — every service on a host: owner, state, health and the image it runs
curl -s http://127.0.0.1:8090/api/services?host=dev | python3 -m json.tool

The console’s Services page shows the same data per service:

• Ownership — compose / coolify / external / none (never blank)
• Container state and health
• The image, and the commit it was built from
• CPU and memory, with the denominator named

Container monitoring

# What is running, its status and its published ports
docker ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}'

# Live resource use
docker stats --no-stream

Caution

A container docker did not sample reads as “not measured”, never 0. Zero sorts quietest and reads as the thing to leave alone, which is the opposite of true. Every share in the console names its denominator — cores, the container’s own limit, or host total — because 180% CPU is alarming until you know the box has twelve cores.

OpenSearch Health

# Cluster health
curl -s http://localhost:9200/_cluster/health | jq .

# Individual index health
curl -s "http://localhost:9200/_cat/indices?v&health=red"

# Node stats
curl -s http://localhost:9200/_nodes/stats | jq '.nodes | to_entries[] | {name: .value.name, heap: .value.jvm.mem.heap_used_percent}'

Semantic Search Model Health

opensearch-mgr and the 30-minute model-health-check.js cron are gone with the old host. The Semantic Search service provisions and recovers its own model, and reports it on /health (checks.model) — see Semantic Search:

curl -s https://dev-search.logrite.ai/health
# {"status":"success","data":{"status":"healthy","checks":{"opensearch":true,"model":true},…}}

Status Report

Send a full status report to Discord:

# The console collects this continuously; there is no reporter script to run
curl -s http://127.0.0.1:8090/api/overview | python3 -m json.tool

The overview includes:

• Every service’s state, health and ownership, on both hosts
• OpenSearch cluster health
• Disk use, with a reclaim preview
• Recent deploys and their outcomes

Reading a host’s status: ok vs degraded

/api/overview returns one reading per host, assembled from three independent collections. They fail independently and mean different things, so the payload reports them separately:

Field	What it means	What it does NOT mean
ok: false + error	The host did not answer at all — cat /proc/meminfo failed. Genuinely unreachable.	—
degraded.containers	The host answered; docker ps/stats did not. Docker is wedged or stopped.	That the host is unreachable, or that it runs no containers
degraded.docker_df	The host and docker both answered; the disk total could not be measured.	That there is nothing to reclaim
stale: true	The collection overran its budget, so this is the previous reading. age_seconds says how old.	That collection failed

A missing figure is never rendered as zero

0 B reclaimable and “docker could not measure this” are opposite claims that look identical on a tile — and the false one is the reassuring one. The console prints “not measured” instead. Same rule for the container counts: a host whose docker ps failed shows ?, not 0 / 0, all health checks pass.

Why a healthy dev VM used to report itself unreachable

Worth knowing, because the symptom points at the network and the cause is not there. Three defects composed, each invisible on its own:

1. docker system df is the slowest call the collector makes — measured on dev at 492 images (97.6 GB) and 699 build-cache records (42.5 GB): 16.0s alone, 23.6s with three collections in flight. It was being given the default 25s timeout.
2. collected_at is stamped when a collection STARTS, and the 4s freshness cache was checked against it. A reading that takes 10–25s is therefore already expired when stored, so the cache never hit once and every poller collected — the load that caused the timeout was the dashboard’s own traffic, which is what made the failure intermittent.
3. All three collections shared one try/except, so a timeout on the disk total discarded 31 healthy containers and every metric beside them, and returned ok: false — which the page renders as “host unreachable”.

Net effect: dev reported unreachable while answering every other query in five seconds, and prod — which is collected over SSH — reported fine. Nothing was logged, because the error was returned as data and printed nowhere, so journalctl -u logrite-management held 0 occurrences over 24 hours.

Now: the disk total has its own 120s lifetime (collector.DF_TTL) and a timeout matched to what it costs, the snapshot cache is measured from completed_at, collections are single-flight per host, and every degraded section is logged as COLLECT <host> <section> unavailable: <reason>.

# Is the console's reading of a host complete, and how old is it?
curl -s http://127.0.0.1:8090/api/overview \
  | python3 -c 'import json,sys
d=json.load(sys.stdin)["hosts"]
for h,v in d.items():
    print(h, "ok=%s" % v["ok"], "age=%ss" % v.get("age_seconds"),
          "degraded=%s" % (list(v.get("degraded") or {}) or "-"),
          "stale" if v.get("stale") else "")'

# Has anything been degrading? (nothing logged = nothing degraded)
sudo journalctl -u logrite-management --since '1 hour ago' | grep COLLECT

If the disk total starts timing out again, prune before raising a limit

Its cost tracks the estate’s history, not its size. Dev was holding 48 GB of reclaimable images and build cache on an 85%-full disk; clearing that is what makes the call fast, and the console’s own Reclaim view previews it.

Disk: what is reclaimed automatically, and what never is

Measured on dev on 2026-10-01, with the disk at 84%:

• 522 local images of our own services had never been trimmed. builder.prune_images() deletes registry tags only, and the hourly disk guard matched the old control plane’s image names.
• 30.7 GB of build cache had built up. Both cleaners pruned by age (a week), and nothing on a host that rebuilds daily gets a week old. docker builder prune -f frees only dangling cache, 0.7 GB of it.
• The console’s janitor waited a full interval after every start, on a console re-installed several times a day, so it never ran.

Kind	Reclaim button	Automatic (janitor, daily, both hosts)
Build cache	docker builder prune -af: all of it, costs rebuild time only	-af --filter until=48h: anything unused for two days; the hourly dev guard uses the same rule
Dangling images	docker image prune -f (never -a)	yes
Old images (new)	one docker rmi each, no -f	yes
Stopped containers	per item only	never
Volumes	never: removing one deletes its data	never

An old image is a commit-stamped build of our own services (127.0.0.1:5000/*) that is none of the following:

• one of the newest 3 of its service
• used by any container, running or stopped (a stopped one is a --preadopt- rollback)
• named anywhere in pins.json
• a build base (*-base; the backend prebuild needs the local copy)
• a moving tag (latest, dev)

Sizes are each image’s unique layers. Docker’s own “reclaimable” headline is computed differently (21.9 vs 52.5 GB on dev), so the total is an estimate.

Volumes are listed read-only and named. Each row says what it belongs to:

• the compose project, when it was declared by compose
• otherwise the service it was last attached to, from a ledger the console records while containers exist (/var/lib/logrite-deploy/volume-owners.json)
• otherwise what its files say. Dev’s 11 orphans were 6 Splunk directories, one PostgreSQL 16 data directory, and 4 empty volumes.

A build-cache size cap does not work on this Docker

On docker 29.7.1 / BuildKit v0.32.0, docker builder prune --max-used-space and --keep-storage both free 0 bytes, with or without -a, even with the cap set 2 GB under the current size. They parse, report success, and do nothing, which is how the first version of this job freed nothing. An age window with -a does work: until=48h freed 8.0 GB of dev’s cache on 2026-10-01.

The janitor’s last run is kept in janitor-state.json, so a restart no longer resets its clock. An overdue run waits a 15-minute grace, never zero, so an install is still not a prune. The hourly dev guard also no longer runs docker container prune -f, which deleted the console’s --preadopt- rollback copies along with other stopped containers.

Log Monitoring

# Follow a service's logs
docker logs -f <container>

# The last minute only — a total cannot tell "recovered" from "still broken"
docker logs --since 60s <container>

# OpenSearch and Dashboards are containers like everything else
docker logs -f "$(docker ps --format '{{.Names}}' | grep -i opensearch | head -1)"

Did the request reach the service?

Every Node service writes two lines per request — one when the headers arrive, one when the response finishes:

POST /api/v1/log-analytics/MigratedWork id=56c6faff-b062-4612-86c5-fc5e6ab894ca
+ POST /api/v1/log-analytics/MigratedWork 200 13673ms id=56c6faff-b062-4612-86c5-fc5e6ab894ca

+ is a 2xx, ~ a 3xx, ! a 4xx/5xx. The id is the same value the caller was handed back in the X-Request-Id response header, so a partner quoting their id can be found directly:

docker logs <container> 2>&1 | grep 'id=<the id they quoted>'

Note

The arrival line is written the moment the request headers are parsed, before any body is read. That is what makes it usable as evidence: if a caller reports a failure and there is no arrival line for it, the request never reached this process — whatever the client says happened, happened upstream of the origin.

Caution

Before 2026-09-25 the finish line printed the router-relative path, so every log-analytics upload finished as + POST / 200 13673ms and GET /health as + GET / 200 1ms — you could not tell from the log which endpoint had served, and neither line carried the request id. req.path is a getter over req.url, which express rewrites while dispatching and restores only as the next() chain unwinds; a terminal handler that answers never calls next(), so the restore never runs. The path is now captured on arrival. If you are reading a log written by an older image, expect the bare /.

Triaging a client-side ECONNRESET

A Postman “Could not send request — Error: read ECONNRESET” is a TCP reset, not an HTTP status, and it is reported identically whether the origin refused, the edge dropped it, or the client’s own network broke. The log lines above are what tell those apart:

1. Grep for the arrival line in the window the caller names. An arrival line with no finish line means the origin took the request and the connection died mid-answer — look at how long that endpoint holds the socket.

2. No arrival line at all means the request never reached the origin. Nothing in this service can cause or prevent that; the reset happened between the client and Cloudflare. Ask the caller to retry, and confirm the payload is reproducible from another network before treating it as a service defect.

3. Check the container did not restart — a redeploy resets every open connection, and the symptom is identical:

   docker inspect <container> --format 'RestartCount={{.RestartCount}} OOMKilled={{.State.OOMKilled}} Started={{.State.StartedAt}}'

4. Check the tunnel for a matching error, which localises a failure to the origin leg rather than the client leg:

   sudo journalctl -u cloudflared --since '<time>' | grep -i 'error\|canceled'

Caution

Cloudflare answers its own timeout with a real 524 and a body — measured at 128.1s against the dev tunnel, not the documented 100s. So a 524 and an ECONNRESET are different failures with different causes: the first means the origin was too slow, the second means the connection never completed. Do not treat them as the same report.

On endpoints that hold the connection while they work — the public log-analysis upload transcribes every image before answering — prefer Prefer: respond-async and poll statusUrl, and send an Idempotency-Key so a retry after a reset returns the original answer instead of starting a second analysis.

Downtime Tracker Dashboard

Live dashboard: https://downtime.logrite.ai (a compose service deployed by the management console on both estates; source in MasterControl services/downtime/dev/logrite-downtime-tracker, repo Deployd-io/logrite-downtime-tracker).

Renders two ledgers:

• downtime.json — host-level reboots and heartbeat gaps (kept by tracker.py ticking from cron every 5 min).
• services/*.json — per-service downtime events (kept by services-tracker.py, same tick).

Settings: Your connection and Server network

Both downtime.logrite.ai (gear button → Settings) and management.logrite.ai (sidebar → Settings) show the same two sections. Both sites use the same code: netinfo.py, kept identical in ops/management/ and in the tracker repo.

Section	What it shows	Where the answer comes from
Your connection	The public IP of the device viewing the page, its city / region / country, approximate coordinates, time zone and network	IP: CF-Connecting-IP. Location: Cloudflare’s visitor-location headers (cf-ipcity, cf-iplatitude, …) when the zone sends them; otherwise one ipinfo.io lookup of that IP, cached for 6 h
↳ Exact location of this device	Address, coordinates and accuracy (± m)	The browser’s Geolocation API, only after the user clicks Use my device location and allows it. The address comes from OpenStreetMap Nominatim, called from the browser. It is never sent to or stored by either server
Server network	The server’s public egress IP and its location	The server itself asks ipinfo.io/json, and the answer is cached for 1 h. The console asks the prod VM over the collector’s ssh hop, so each row is that VM’s own answer. The tracker follows its environment switcher (api/network/server is on the proxy allow-list)

Routes: console GET /api/network (?force=1 from Refresh drops the cache); tracker GET /api/network/visitor and GET /api/network/server.

Caution

IP geolocation is city-level at best, and it is wrong behind a VPN, a mobile carrier or a corporate proxy. When Cloudflare’s country and the lookup’s country disagree, Cloudflare’s is kept and the page says the city is unreliable. The server IP is not an address anyone reaches the site on. Every hostname comes in through a Cloudflare tunnel, so it is only the egress of the VM’s network. Dev and prod currently share one egress, and the console says so.

Note

/api/network/visitor is deliberately not proxied. Read through another environment, it would describe this server as that environment’s visitor.

Reason-column chip vocabulary

Each row’s “Reason” cell renders a single chip. Branches are evaluated top-to-bottom — the first match wins.

Chip	Colour	When	Tooltip explains
host outage	red	host_event: true AND not a graceful reboot (≥ host_event_threshold distinct services share a down_at within ±host_event_window_seconds of the host event)	Host was cut — power loss, hypervisor reset, or host crash. Lists the N correlated services.
unexpected reboot	red	kind: "reboot" AND reboot_type: "ungraceful" AND no host_event	Kernel rebooted without systemd reaching a shutdown target — power loss / panic / hard-reset class.
reboot	indigo	kind: "reboot" AND reboot_type: "graceful"	systemd ran the full shutdown sequence cleanly — systemctl reboot, planned maintenance. Not a power-loss event.
reboot	indigo	kind: "reboot" AND reboot_type: "unknown" AND no host_event	Kernel rebooted; graceful/ungraceful undetermined (journal unavailable).
power outage	indigo	kind: "gap" OR legacy entry (no kind) AND no host_event	Host unreachable for the duration — cron stopped, VM paused, or pre-classifier event.

host_event overlay

Set on a host downtime entry when correlated with N+ simultaneous service-level down events. Containers don’t coordinate to die in the same second on their own; only a host-level cut can synchronize them. Recorded by both:

• tracker.py:cmd_tick() at detection time (live ticks).
• classifier.py reclassify_all() during back-fill of historical events.

Tuned by two keys in /home/victor/uptime-tracker/downtime-config.json:

{
  "host_event_threshold":       5,
  "host_event_window_seconds":  60
}

Schema on a host downtime entry (all new fields optional):

{
  "kind":         "reboot",
  "reboot_type":  "unknown",
  "host_event":   true,
  "host_event_evidence": {
    "service_count":     14,
    "sample_services":   ["docs", "downtime-tracker", "backend-main", "..."],
    "window_seconds":    60,
    "source":            "services-ledger-correlation"
  }
}

Graceful reboots also drag services down (systemd takes them down) and so naturally satisfy the correlation. The dashboard intentionally suppresses the host outage chip when reboot_type == "graceful" — that’s a chosen reboot, not a host cut.

Re-running the classifier

cd /home/victor/uptime-tracker
python3 classifier.py reclassify

Idempotent. Walks every per-service ledger and downtime.json, refreshes classification, evidence, and the host_event overlay. Run after tuning thresholds or after a major outage to back-fill the new labels.

Inode note — downtime.json is bind-mounted into the dashboard container by inode. The classifier writes it in-place (matching tracker.py:save()) to keep the inode stable; an os.replace would silently break the mount and leave the dashboard serving an orphaned copy. Per-service ledgers are NOT bind-mounted and use the atomic-replace pattern as before.