Anomaly Detection

Overview

OpenSearch anomaly detection automatically identifies unusual patterns in log data without requiring manual threshold configuration. It learns baseline behavior and scores deviations as anomaly grades (0.0–1.0).

Configured Detectors

Detector	Index	Features	Category Fields
payment-latency-detector	payment_queries	avg_response_time, max_response_time	method
external-service-latency-detector	external_service_calls	avg_response_time, max_response_time	service, method
database-latency-detector	database_queries	avg_response_time, max_response_time	query_type, table_name

Anomaly Grade Severity

Grade	Severity	Meaning
≥ 0.8	CRITICAL	Severe deviation from baseline
≥ 0.6	HIGH	Significant anomaly
≥ 0.4	MEDIUM	Notable deviation
≥ 0.2	LOW	Minor deviation
< 0.2	NORMAL	Within expected range

CLI Commands

# Setup detectors for all indexes
opensearch-mgr anomaly-setup

# Setup for specific index
opensearch-mgr anomaly-setup --index payment_queries

# Force recreate (if already exists)
opensearch-mgr anomaly-setup --force

# Show status and recent anomalies
opensearch-mgr anomaly-status

# List all detectors
opensearch-mgr anomaly-list

# View anomaly results
opensearch-mgr anomaly-results payment-latency-detector
opensearch-mgr anomaly-results --all
opensearch-mgr anomaly-results --all -t 0.5  # Grade >= 0.5 only

# Start/stop detectors
opensearch-mgr anomaly-start --all
opensearch-mgr anomaly-stop payment-latency-detector

# Delete detectors
opensearch-mgr anomaly-delete --all --force

Demo Data Workflow

# 1. Generate 24h of data with built-in anomalies
node /root/opensearch/scripts/generate-anomaly-demo-logs.js --hours 24 --ingest

# 2. Setup detectors
opensearch-mgr anomaly-setup

# 3. Wait 2-3 minutes for detector initialization

# 4. Check status
opensearch-mgr anomaly-status

# 5. View detected anomalies
opensearch-mgr anomaly-results --all -t 0.3

Built-in Anomaly Schedule (Demo Data)

Time	Duration	Description
3:00 AM	15 min	Spike — DB connection pool exhausted
9:00 AM	30 min	Gradual rise — morning traffic surge
2:00 PM	10 min	Extreme spike — external API timeout
8:00 PM	45 min	Sustained high — background job contention

Troubleshooting

Detector shows INIT_FAILURE:

• Ensure target index has data (minimum ~30 data points)
• Verify timestamp field format: yyyy-MM-dd HH:mm:ss
• Check OpenSearch logs: opensearch-mgr logs opensearch

No anomalies detected after setup:

• Wait 2-3 minutes — detectors need a learning period
• Run: opensearch-mgr anomaly-results --all -t 0 to see all results including low-grade
• Try historical analysis on existing data

Detector won’t start:

• Verify plugin is installed: curl localhost:9200/_cat/plugins | grep anomaly