Self-Observability¶

synthkit ships its own operational telemetry alongside (and completely separate from) the synthetic data it produces. This is the generator's self-observability: RED metrics on the synthetic-push pipeline, Go runtime metrics, per-tick traces, the operational log stream, and continuous process profiles via Pyroscope — all sent to a different Grafana Cloud stack through their own credential triplets, never GC_TOKEN.

Self-observability is off by default and is independent of synthetic DRY_RUN. The profiling path has no master flag of its own; it follows SELFOBS_ENABLED but has its own credential triplet.

What it ships¶

OTLP signals (metrics + traces + logs)¶

All three signal types flow over a single OTLP/HTTP endpoint to the self-obs stack:

Signal	What it covers
Metrics	`synthkit.push` (count/items/bytes/duration per sink/blueprint/outcome), `synthkit.tick` (invocations + duration), `synthkit.cycle.duration`, `synthkit.dropped_ticks`, delivery-queue depth/flush/loss metrics (`synthkit.queue.`), Fleet Management operations (`synthkit.fleet.`), and observable gauges: `synthkit.ledger.size`, `synthkit.volume.multiplier`, `synthkit.blueprint.count`, `synthkit.cardinality.series`
Traces	Per-tick spans wrapping each construct invocation; `cycle` spans (backdated to the generation window); `push <sink>` child spans for each live push; `flush <sink>` spans for the decoupled delivery queue; `fleet <op>` spans for FM registration and heartbeat round-trips
Logs	Structured OTLP LogRecords for push failures (event=`push_error`), tick errors (event=`tick_error`), FM failures (event=`fleet_error`), config-change events (event=`config_change`), and the operational heartbeat (every 5 minutes)

Go runtime metrics (goroutines, memory, GC) are collected via the OTel contrib runtime instrumentation against the self-obs MeterProvider.

Delivery-queue metrics use only the bounded sink label:

Metric	Meaning
`synthkit.queue.depth`	Current enqueued-but-not-yet-flushed items
`synthkit.queue.enqueue_blocked`	Times a producer blocked on a full shard buffer
`synthkit.queue.flush` / `.duration` / `.batch`	Completed flushes, latency, and attempted item count
`synthkit.queue.dropped.items`	Cumulative items discarded after sink retry exhaustion
`synthkit.queue.current_loss`	`1` while any affected sender shard has not completed a later successful flush; otherwise `0`
`synthkit.queue.last_loss`	Unix seconds of the most recent discarded sub-batch

Loss metrics do not imply replay. A transition of current_loss back to 0 records recovery for new delivery; dropped.items remains cumulative until the process restarts.

Continuous profiles (Pyroscope)¶

When GC_PYROSCOPE_* credentials are set, the generator's own process is profiled continuously and sent to a Grafana Cloud Profiles (Pyroscope) instance via its own GC_PYROSCOPE_* credential triplet — independent of the OTLP self-obs triplet, though both are typically the same staff stack. Profile types: CPU, heap allocation/inuse (objects + bytes), goroutines, and optionally mutex and block profiles. Application name is synthkit.

Isolation design¶

internal/selfobs is the sole package in synthkit that imports the OpenTelemetry SDK. The SDK is banned on the synthetic-data path — the synthetic OTLP sink hand-encodes proto and never touches the OTel API.

The seam is stdlib-only in both directions:

Sinks report push outcomes through internal/pushhook (a plain function type). selfobs.PushObserver() returns an observer that internal/selfobs registers; the sinks themselves never import selfobs.
The runner reports per-tick outcomes through runner.TickFunc (also a plain function type). The runner stores a method value from selfobs; it never links the SDK.

selfobs builds its own TracerProvider, MeterProvider, and LoggerProvider, and never installs them as OTel globals (otel.SetTracerProvider is never called). The synthetic OTLP sink, which bypasses the OTel global API entirely, is therefore completely unaffected.

See Architecture §6.1 for the full isolation rationale.

Configuration¶

Self-observability activates only with SELFOBS_ENABLED=true and the complete GC_SELF_OTLP_ENDPOINT, GC_SELF_OTLP_USER, and GC_SELF_OTLP_PASSWORD triplet. Its in-process status can prove configuration/startup, not remote landing: verify fresh synthkit_* telemetry in the staff context after SELFOBS_METRIC_INTERVAL plus the delivery deadline.

Process profiling is a separate lane. It additionally requires the complete GC_PYROSCOPE_URL, GC_PYROSCOPE_USER, and GC_PYROSCOPE_PASSWORD triplet. Verify the synthkit process profile in the staff profiles tenant. These are not the customer synthetic profiles emitted by a workload declaration through GC_PROFILES_* and GC_TOKEN.

Enable self-observability by setting SELFOBS_ENABLED=true and providing the OTLP credential triplet. All three vars are required; if any is missing, synthkit logs a warning and falls back to a no-op handle.

OTLP self-observability¶

Env var	Required	Default	Description
`SELFOBS_ENABLED`	—	`false`	Master on/off switch
`GC_SELF_OTLP_ENDPOINT`	When enabled	—	OTLP gateway base URL for the self-obs stack (e.g. `https://otlp-gateway-<region>.grafana.net/otlp`)
`GC_SELF_OTLP_USER`	When enabled	—	Self-obs stack ID (HTTP Basic username)
`GC_SELF_OTLP_PASSWORD`	When enabled	—	CAP token with `metrics:write`, `logs:write`, `traces:write` — never `GC_TOKEN`
`SELFOBS_TAGS`	—	—	Extra resource attributes as `key=value,key=value` CSV
`GC_SELF_GRAFANA_URL`	—	—	Staff Grafana base URL (e.g. `https://your-stack.grafana.net`) — enables deep-links to the self-obs dashboard in the operator UI; leave empty to hide them
`SELFOBS_METRIC_INTERVAL`	—	`15s`	Metric flush cadence (Go duration string); traces and logs use their own batchers and are unaffected

Continuous profiling (Pyroscope)¶

Profiling has no master flag of its own. It is independent of synthetic DRY_RUN, but activation requires SELFOBS_ENABLED=true and the complete three-variable GC_PYROSCOPE_* triplet. An incomplete gate is a no-op reported by the optional-lane disposition.

Env var	Required	Default	Description
`GC_PYROSCOPE_URL`	When profiling	—	Profiles ingest URL (e.g. `https://profiles-prod-XXX.grafana.net`)
`GC_PYROSCOPE_USER`	When profiling	—	Profiles instance ID (Basic-auth username)
`GC_PYROSCOPE_PASSWORD`	When profiling	—	CAP token with `profiles:write` scope
`PYROSCOPE_TAGS`	—	—	Extra profile tags as `key=value,key=value` CSV
`PYROSCOPE_MUTEX_FRACTION`	—	`5`	`runtime.SetMutexProfileFraction`; `0` = mutex profiling off
`PYROSCOPE_BLOCK_RATE`	—	`5`	`runtime.SetBlockProfileRate` in nanoseconds; `0` = block profiling off

A value of 5 for both is high-fidelity and fine for a lab or demo generator. Set both to 0 to reduce overhead on resource-constrained hosts.

Example `.env` snippet¶

# Self-observability — ships to a SEPARATE stack (never GC_TOKEN)
SELFOBS_ENABLED=true
GC_SELF_OTLP_ENDPOINT=https://otlp-gateway-prod-eu-west-0.grafana.net/otlp
GC_SELF_OTLP_USER=123456
GC_SELF_OTLP_PASSWORD=glc_...
SELFOBS_TAGS=env=prod,team=platform
GC_SELF_GRAFANA_URL=https://myorg.grafana.net
SELFOBS_METRIC_INTERVAL=15s

# Continuous profiling (also ships to a separate stack)
GC_PYROSCOPE_URL=https://profiles-prod-006.grafana.net
GC_PYROSCOPE_USER=78910
GC_PYROSCOPE_PASSWORD=glc_...
PYROSCOPE_TAGS=env=prod
PYROSCOPE_MUTEX_FRACTION=5
PYROSCOPE_BLOCK_RATE=5

Relationship to DRY_RUN¶

SELFOBS_ENABLED and the complete staff credential triplets control these process signals. Synthetic DRY_RUN is stamped as run-mode context but does not suppress them. This lets operators observe a dry-run synthkit process without sending customer synthetic data.