RCA Generator (retired)

Retired — there is nothing to run

The RCA generator was ./cli/logrite-manager.js rca on the pre-2026-08-03 host. It decided which services were down by reading pm2 jlist and read their logs from ~/.pm2/logs/. That host was reflashed: /root/logrite no longer exists, there is no PM2 anywhere, and every service now runs as a docker compose container. The script is still in MasterControl (cli/logrite-manager.js, lib/rca-generator.js), but run today it finds no PM2 processes and so reports nothing. No equivalent has been built for the compose platform.

What to use instead

Need	Use
Is anything down, and since when?	The uptime board, https://downtime.logrite.ai — one board for both estates, with each outage classified (host reboot, platform outage, heartbeat gap) — see Monitoring
A service’s recent logs	The console’s Logs tab at https://management.logrite.ai, or docker logs <container> on the host
What was deployed, by whom, and did it fail?	The console’s deployment history, and the deploy announcements in Discord — see Discord notifications
Who changed what	Audit logs
A written root-cause analysis	Write it by hand from the above; past write-ups live under docs/status/ in MasterControl

The rest of this page is kept as a record of what the retired tool did.

What it did (historical)

The RCA Generator used Groq’s LLM to analyse service outages and audit-log history and post a structured Root Cause Analysis report to Discord.

How it was triggered

cd /root/logrite

# Generate and send RCA report to Discord
./cli/logrite-manager.js rca

What it analysed

The RCA Generator gathered:

1. PM2 process state (pm2 jlist) — which processes were not online
2. Recent audit log entries — Last 50 operations (who did what)
3. OpenSearch cluster health — Index status, node stats
4. Recent logs — from ~/.pm2/logs/<service>-out.log and the PM2 daemon log

This context was sent to Groq with a structured RCA prompt.

Report format

# Root Cause Analysis Report

**Incident:** Service degradation detected (3 services errored)
**Time:** 2026-03-02 14:00 UTC
**Affected Services:** analyzer-main, semantic-search-main, ai-service-main

## Timeline
- 13:45 UTC — OpenSearch ML model unloaded (confirmed by model-health-check logs)
- 13:50 UTC — Semantic search health checks started failing
- 13:55 UTC — AI Service unable to complete tool calls (semantic_search failed)
- 14:00 UTC — Analyzer queries timing out

## Root Cause
The OpenSearch ML embedding model (`all-MiniLM-L6-v2`) was unloaded after an
OpenSearch internal GC pressure event. Services that depend on the model
(Semantic Search, AI Service) started returning errors. The Analyzer was
indirectly affected because AI Service tool calls it depends on were failing.

## Contributing Factors
1. No automatic model reload on unload event
2. Health check cron interval (30 min) too long to catch fast degradation

## Recommendations
1. Reduce model health check interval from 30 min to 5 min
2. Add model loaded status to Semantic Search /health endpoint
3. Configure `opensearch-mgr watch` mode for continuous monitoring
4. Add circuit breaker in AI Service for semantic_search tool failures

Programmatic usage

const { generateServiceOutageRCA } = require('./lib/rca-generator');
const { getRecentAuditEntries } = require('./lib/audit-log');

const auditEntries = await getRecentAuditEntries(50);

const report = await generateServiceOutageRCA(
  ['analyzer-main', 'semantic-search-main'],
  auditEntries,
  new Date().toISOString()
);

// report.summary, report.rootCause, report.timeline, report.recommendations

Configuration

It used the same GROQ_API_KEY as other services and read the master config/ directory.