Sending OpenTelemetry data

Endpoints (2026-10-09)

OpenTelemetry data enters Logrite through the ingest-shim, the same service that takes collector-agent batches. It authenticates the sender with a collector-agent token, writes the token’s workspace onto every resource, and forwards the request to Data Prepper on the internal docker network.

Signal	Prod	Dev
Traces	https://ingest.logrite.ai/v1/traces	https://dev-ingest.logrite.ai/v1/traces
Logs	https://ingest.logrite.ai/v1/logs	https://dev-ingest.logrite.ai/v1/logs

The older hostnames traces.logrite.ai / logs.logrite.ai (and their dev- forms) are moved to the shim (port 9700) by ops/security/tunnel_otlp_front.py, so an exporter posting to /v1/traces or /v1/logs on those hosts reaches the same handler and needs the same token.

export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://ingest.logrite.ai/v1/traces
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://ingest.logrite.ai/v1/logs
# The space after "Bearer" must be URL-encoded in this variable.
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$LOGRITE_AGENT_TOKEN"
export OTEL_RESOURCE_ATTRIBUTES=service.name=my-service

LOGRITE_AGENT_TOKEN holds a <collector agent token>; never commit it.

The token

Use the token of any enabled collector agent of the workspace. An agent created with the default current_directory source works; its token is shown once, at creation or after a rotate. A separate agent per service lets each service be disabled or rotated on its own. The token decides the workspace; nothing the sender puts in the payload can change it.

Request rules

• Body: application/x-protobuf only (OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf). Content-Encoding: gzip is accepted. A JSON body answers 415 {"error":"use http/protobuf"}.
• Size: at most 8 MiB after decompression, else 413.
• Native gRPC is not served (Cloudflare does not proxy it); use OTLP/HTTP protobuf.

What the shim changes

In every ResourceSpans / ResourceLogs entry the shim removes any resource attribute named workspace_id or logrite.agent_id and appends two of its own:

Attribute	Value
workspace_id	the token’s workspace
logrite.agent_id	the token’s agent id

Every other byte of the request is forwarded unchanged (uncompressed).

Status codes

Answer	When
200 {}	Data Prepper accepted the export
401	missing, unknown or revoked token
403	the agent is disabled, or has no workspace
400	the protobuf could not be parsed
413 / 415	too large / not protobuf
429 or 503 with Retry-After	Data Prepper is busy or unavailable — retry
503 with Retry-After	Data Prepper refused the shim’s own credential (shim misconfigured; retryable)
502	any other upstream error

Worked example: CareVault

ops/demo/carevault/compose.yml (MasterControl) sends each app’s traces through the shim with its own agent token, which provision.py writes:

x-app-env: &app-env
  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: ${OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:-https://ingest.logrite.ai/v1/traces}
services:
  fhir-gateway:
    environment:
      <<: *app-env
      OTEL_EXPORTER_OTLP_HEADERS: Authorization=Bearer%20${AGENT_TOKEN_FHIR_GATEWAY:?set AGENT_TOKEN_FHIR_GATEWAY}

The same pattern repeats for ENROLLMENT, CARE_MANAGEMENT, PRIOR_AUTH and MEMBER_SERVICES. The shared Basic header and the otlp-bridge service are gone.

Data Prepper is internal only

Data Prepper’s OTLP sources (traces 21890, logs 21891) are reached only by the shim, at http://data-prepper:21890/opentelemetry.proto.collector.trace.v1.TraceService/Export and http://data-prepper:21891/opentelemetry.proto.collector.logs.v1.LogsService/Export. They are not published on the host and no tunnel route points at them (see Ports).

The sources still require HTTP Basic auth. That credential is now shared by the shim and Data Prepper only:

Service	Env keys
data-prepper	OTLP_INGEST_USERNAME, OTLP_INGEST_PASSWORD
ingest-shim	OTLP_UPSTREAM_USERNAME, OTLP_UPSTREAM_PASSWORD

Set both pairs in one console change (POST /api/env, see Environment variables). bin/render_config.py refuses to start Data Prepper when either variable is empty. The shim’s upstream URLs can be overridden with OTLP_TRACES_URL / OTLP_LOGS_URL.

Cutover is a rollout step

The shim routes, the tunnel script and the per-workspace log routing ship in code. Moving the tunnel hostnames, removing the 21890:21890 / 21891:21891 host ports and rotating the upstream credential are done per estate during rollout. Before that, the old hostnames still answer at the gRPC method path with the estate Basic credential.

What happens to the data

• Traces are routed by the workspace_id resource attribute (/attributes/resource.attributes.workspace_id inside the pipeline) into trace-analytics-raw-<workspace_id>. Spans without it go to trace-analytics-raw-logrite.
• Logs are routed the same way (2026-10-09): a log with workspace_id lands in log-analytics-<workspace_id>-otlp, inside the workspace’s own log-analytics-<ws>-* family, so that workspace (and only it) can read it. Logs without it still land in log-analytics-logrite.
• Because the shim writes workspace_id from the token and strips any value the sender set, the attribute is no longer self-asserted (DS-F16 closed).
• Mapping. The index template logrite-trace-analytics-raw (priority 200, pattern trace-analytics-raw-*) maps serviceName, traceId, spanId, parentSpanId, kind and traceGroup as keyword and every string under *attributes.* as keyword. Source and installer: data-prepper/templates/trace-analytics-raw.index-template.json and data-prepper/bin/apply-trace-template.sh --reindex in MasterControl.
• Logs go through a grok processor (query_exec_ms from find query executed in N ms) and a date processor that stamps log_ingest_time from the ingest host’s clock (the M-26-14 baseline-3 attestation).
• Ingest-time redaction and sampling do not apply to OTLP. Those workspace settings act on collector-agent batches and the ingest endpoint only — see Ingest-time redaction and sampling.

History

• 2026-10-07 — both Data Prepper sources required one Basic credential per estate; anonymous exports had been accepted before that.
• 2026-10-08 — the trace route pointer was corrected; before it every span went to trace-analytics-raw-logrite.
• 2026-10-09 — OTLP moved behind the ingest-shim with per-workspace collector-agent tokens; OTLP logs routed per workspace; the ops/demo/otlp-bridge bridge was removed.

Audit changes (2026-10-10)

Documented from ai-service main at c8e7ef98 (shim) and LogriteMasterControl main at 9ce9a1fa (Data Prepper).

• The token is checked before the body is read. /v1/traces and /v1/logs run the same per-IP rate limit and bearer check as POST /ingest before reading up to 8 MiB: 429 (retryAfterSeconds), 401 missing bearer token / invalid or revoked agent token, 403 agent disabled / agent has no workspace, 503 when the token cannot be resolved right now.
• Heartbeat, scan trigger, redaction. A successful OTLP upload marks the collector agent as seen (heartbeat); a logs upload also triggers the workspace’s agent log scan. When the workspace’s ingest policy enables redaction, OTLP log bodies, string log attributes and string span attributes are masked with the same redactor /ingest uses, in the same protobuf walk that stamps the resource attributes.
• Payload errors are final. An upstream 400, 413 or 415 passes through with retryable: false, so a collector stops retrying a payload that can never be accepted.
• Access log. Each data request (/v1/traces, /v1/logs, /ingest) writes one line: [shim:access] path=<p> status=<s> ms=<n> ws=<id> agent=<id> bytes=<n> upstreamMs=<n>.
• /ingest counts. The reply’s stored excludes permanently dropped lines, and the heartbeat carries the agent’s lastShip.
• Dead-letter files. Every Data Prepper OpenSearch sink names a dlq_file (/usr/share/data-prepper/dlq/<pipeline>-<route>.jsonl, e.g. trace-pipeline-tenant_tagged.jsonl), bind-mounted from /var/lib/logrite-data/data-prepper/dlq (owned 10001:10001). A document OpenSearch rejects is kept there instead of being dropped.