Troubleshooting¶

This page covers the most common operational problems. For a structured end-to-end verification walkthrough, see RUNBOOK.md. For generator health metrics, see self-observability.md.

Executable symptom matrix¶

The ten source symptoms on this page are marked with troubleshooting-symptom comments and have one row in the maintained machine-readable matrix. The clean-environment check executes every local, non-destructive precondition, remedy, and post-remedy assertion without Docker, Compose, a live stack query, or an external-product write. Run it through the repository's troubleshooting-check recipe when that root-owned wiring is present.

The matrix deliberately separates local startup evidence from live delivery evidence. A local assertion can prove configuration parsing, load-time validation, an offline inventory, or a safe control-plane bind; it cannot prove that Grafana accepted a later batch. Rows marked live-delivery-required name the destination and credential prerequisite, and the clean check reports the external part as BLOCKED_EXTERNAL rather than silently treating it as passed.

Start with optional-lane dispositions¶

curl -s -u control http://127.0.0.1:8088/control/status | jq '.optional_lanes'

All nine lanes report one of enabled, partial, disabled, or unsupported. partial includes only closed non-secret missing field names; disabled is intentional/no request; unsupported is a hard stop. Fleet metrics and API registration are separate rows. For Synthetic Monitoring, partial after an apply normally means the required docker compose restart synthkit was not performed or the snapshot/registration binding is stale.

No data appearing in Grafana¶

DRY_RUN is still true¶

Cause: DRY_RUN defaults to true. A live push is always an explicit opt-in.

Fix: Set DRY_RUN=false in .env. Restart the container or process.

Verify: curl -s -u control http://127.0.0.1:8088/control/status | jq '.dry_run' must return false.

Credentials are wrong or missing¶

Cause: GC_TOKEN, or one of the endpoint/user pairs, is empty or incorrect.

Fix:

1. Confirm the minimum set is filled in .env: GC_TOKEN, GC_PROM_RW, GC_PROM_USER, GC_OTLP_ENDPOINT, GC_OTLP_USER, GC_LOKI, GC_LOKI_USER.
2. Check that GC_PROM_USER is the numeric Mimir instance ID, not an email address.
3. Confirm the CAP token has metrics:write, logs:write, and traces:write scopes.

Verify: Look at sink failures in the status strip:

curl -s -u control http://127.0.0.1:8088/control/status | jq '.sinks[] | {sink,failures,last_error_code}'

A non-zero failures count with last_error_code="authentication" confirms an authentication problem without exposing remote response text.

Inline comment in .env corrupted a value¶

Cause: Docker Compose's env_file does NOT strip inline comments. TOKEN=abc # my token sets the variable to abc # my token.

Fix: Move comments to their own line above the value. Restart.

Metrics arrive but traces or logs are missing¶

Cause: The three sinks are independent. A credentials problem on one does not affect the others.

Fix: Check each sink separately in GET /control/status. Fill the missing endpoint/user pair and restart.

Series cap / kill switch¶

Symptom: Some constructs have data, others do not — especially lower-priority or substrate constructs.

Fix: Increase or unset SERIES_CAP in .env. If the cap is intentional, reduce the blueprint's declared constructs. For a high-DPM blueprint, also check its fixed-minute series_budget: it must cover the projected series count multiplied by DPM per series. Startup and /control/schema expose that projection; high-dpm-churn is 115 projected series × 6 DPM = 690 data points per minute.

Loki rejected high-cardinality stream labels¶

Cause: Loki rejects streams where a label carries high cardinality (e.g. request IDs, trace IDs). synthkit's Loki sink asserts this contract at startup; the error appears in the process log.

Fix: High-cardinality fields must be JSON payload fields, not stream labels. If you are authoring a custom app blueprint with a telemetry DSL, check your labels: declarations — a ref to a high-card key is only legal in log body or span attributes, never as a label. See the internal/highcard constraint in architecture.md.

Control plane unreachable¶

Cause: JSON_HTTP_ADDR defaults to 127.0.0.1:8088 (loopback only) for direct binary runs. In Docker Compose the binary binds 0.0.0.0:8088 inside the container, but the host-side interface is SYNTHKIT_BIND (defaults to 127.0.0.1).

Fix (reach from another host): Prefer an SSH tunnel. For deliberate non-loopback exposure, set SYNTHKIT_BIND, a non-empty CONTROL_TOKEN, and CONTROL_EXPOSURE_ACK=trusted-network or tls-proxy; startup fails without all required values.

ssh -L 8088:localhost:8088 <host>
# then access http://localhost:8088/control/ui locally

Fix (reach from Grafana Cloud): Configure a PDC Tailscale connection so Grafana Cloud can reach the Tailscale IP directly without public exposure.

Control-plane state not persisting across restarts¶

Cause: The /data bind mount is a single-file mount, or the directory is not owned by uid 65532.

The control plane saves state atomically (write to a temp file → rename). A single-file bind mount breaks the rename step. A directory not owned by uid 65532 (distroless nonroot) produces a permission denied error on every save attempt — visible in persist.last_error:

curl -s -u control http://127.0.0.1:8088/control/status | jq '.persist'

Fix — wrong uid:

if [ -L control-state-data ] || { [ -e control-state-data ] && [ ! -d control-state-data ]; }; then echo 'refusing unsafe state path' >&2; exit 1; fi
mkdir -p control-state-data
chmod 700 control-state-data
docker run --rm --volume "$PWD/control-state-data:/data" --entrypoint /bin/sh \
  node:24-alpine@sha256:d32cdf619f63fe0471182d08996dd516c6275bb5fd31ae06e55a570bd9e1ad43 \
  -ceu 'chown 65532:65532 /data; chmod 0700 /data'
docker compose restart

Fix — single-file mount: Remove the single-file bind mount from docker-compose.yml and replace it with a directory bind as shown in deployment.md. A state file absent at startup is normal; it is created lazily on the first mutation.

Off-tailnet / offline push failures¶

Cause: The Forgejo autocommit hook (or similar) cannot reach the Forgejo server outside the Tailscale tailnet. This is expected and harmless for synthkit itself — the push-status hook exits 0 silently when offline.

For synthkit sinks: The in-memory queue buffers only items waiting for their first delivery attempt. Each sink applies its bounded retry policy. When those retries are exhausted, the failed sub-batch is discarded and counted as loss; synthkit has no WAL and does not retain that sub-batch until connectivity returns. Later items continue normally when the affected sender shard completes a successful flush.

Inspect the authoritative queue state separately from push-attempt history:

curl -s -u control http://127.0.0.1:8088/control/status |
  jq '.queues[] | {sink,depth,blocked_enqueues,dropped_items,last_loss_ms,last_recovery_ms,current_loss,affected_shards}'

dropped_items is cumulative for the process lifetime. current_loss=false with a non-zero dropped count means delivery recovered after historical loss; it does not mean the lost items were replayed. depth=0 likewise means the queue drained, not that no loss occurred.

Using -once -dump as an offline diagnostic¶

Before debugging live connectivity, always confirm blueprints load and series look correct offline:

DRY_RUN=true BLUEPRINT_NAMES=otlp-native ./synthkit -once -dump 2>&1 | less

Expected output per blueprint: - loaded blueprint "<name>" line - synthkit up: N blueprints summary - [dry-run promrw|loki|otlp] summaries with example series/streams/spans

Cross-check a few metric names against signals/ — synthkit never invents names, so anything unexpected is a bug or a misconfigured blueprint.

This command requires no network connectivity and exits cleanly after one tick.

If startup logs WARNING: no blueprints selected, the process is intentionally in setup mode and emits nothing. Set BLUEPRINT_NAMES to exact names and restart; use * only for the full catalog.

Debugging further¶

Matrix rows¶

Each row contains the five executable parts required for the symptom: precondition, action, expected diagnostic, remedy, and post-remedy assertion. Proof boundary states whether the final assertion is local startup evidence or needs live delivery. The acceptance check executes the local parts in a clean temporary environment and never supplies real credentials.

External prerequisite dispositions¶

The clean check cannot prove an external product from a missing credential or an absent tenant. It therefore emits one explicit disposition for each optional lane rather than silently dropping the scenario. Supply the named prerequisite and run the live follow-up from RUNBOOK.md when the product is available.