Operations¶

Where it runs¶

camden, as a systemd service, reading /opt/agent-sessions read-only and exporting OTLP to the local Alloy on localhost:4318. Alloy's Grafana Cloud routing is already owned by Fleet Management; this service holds no Grafana credential and adds no Alloy configuration.

The transcript trees, their retention timer and the SQLite search index are owned elsewhere. This service is a reader and is never in the path of archiving, pruning or restore.

Do not duplicate the existing aggregate collector¶

agent-session-metrics.timer on camden already publishes five-minute metadata and aggregate metrics about the archive itself: file counts, byte coverage, index freshness, parser errors. That job owns the state of the archive. This service owns the content of the transcripts: per-turn, per-item and per-tool detail. Do not re-emit its series.

Backfill¶

Native client telemetry starts when it is switched on. The transcripts are retroactive. A ranged backfill marks every metric point with codingagent.backfill=true so a window where both were live can be excluded rather than double counted. Logs and spans retain their record attributes and timestamps; codingagent.backfill is a metric-only dimension under the field-destination contract.

Run a backfill with both RFC3339 bounds:

codingagent2otel -config /opt/codingagent2otel/config.yaml \
  -backfill-since 2026-01-01T00:00:00Z \
  -backfill-until 2026-02-01T00:00:00Z

The lower bound is inclusive and the upper bound is exclusive. A turn is assigned by its completion time, falling back to its start time when no completion time exists. The command rejects a missing, empty or reversed bound. With neither flag it runs the normal live tail.

Backfill scans the same trees in the configured bounded batches and uses the normal atomic byte-offset checkpoint format. Each UTC-normalized window has a deterministic checkpoint.backfill-<sha256>.json beside the configured live checkpoint. It never reads or writes the live cursor. Repeating the exact window over unchanged input resumes at the end and exports no duplicate series; appended records are picked up on the next run. Do not run the same window concurrently because its two processes would share one checkpoint.

Local throughput measurement¶

On 2026-09-22, BenchmarkBoundedBackfill processed a 6,145-record, approximately 0.94 MB synthetic Codex fixture through bounded source reads, decoding, range selection, metric aggregation and atomic checkpoint writes on an Apple M4 Pro (darwin/arm64). Three three-second samples were 3.55, 3.75 and 3.95 MB/s; the median was 3.75 MB/s, or about 24,500 records/s (251.1 ms per fixture).

go test ./cmd/codingagent2otel -run '^$' \
  -bench '^BenchmarkBoundedBackfill$' -benchtime=3s -count=3

The benchmark uses 256 KiB and 512-record source caps and a disabled network exporter, so use the byte rate to estimate local scan and decode time. Add collector and network headroom separately when planning a full-history run; the slower of local processing and OTLP export determines elapsed time.

Health¶

health.listen exposes liveness and the self-observability snapshot. A green health check is not outcome proof: check codingagent.selfobs.ingest_lag and codingagent.selfobs.unhandled_record_types for whether the service is actually keeping up and actually understanding what it reads.

When a client updates¶

A Codex or Claude Code release can change record shapes. codingagent.selfobs.unhandled_record_types and codingagent.drift_findings are the alarms. An absent record type is a finding, not a default: a decoder that quietly produces nothing looks identical to a quiet day.