Credentials¶

synthkit reads credentials from a .env file (gitignored — never commit secrets). Each signal type uses its own credential triplet. The self-observability path uses an entirely separate Grafana Cloud stack — never GC_TOKEN.

The two stacks¶

Keep these separate. Using GC_TOKEN for self-observability would intermingle the generator's own signals with the synthetic data.

Credential reference¶

Synthetic data sinks¶

A single Cloud Access Policy (CAP) token with metrics:write, logs:write, traces:write, and profiles:write scopes covers the Remote-Write, OTLP, Loki and Pyroscope synthetic-data endpoints. Faro and Sigil use separate credentials.

GC_TOKEN is the password for all four sinks. The user ID for each sink differs (Mimir ID vs stack ID vs Loki ID — they are different numbers).

AI generation ingest / Sigil (optional)¶

The ai_agent workload can export generations, workflow steps and evaluations with GC_SIGIL_ENDPOINT, GC_SIGIL_TENANT_ID and GC_SIGIL_TOKEN. Use the Sigil tenant ID and its ingest credential, not GC_PROM_USER or GC_TOKEN. All three values are needed to configure this lane; ordinary workload traces and metrics remain separate.

RUM / Faro (optional)¶

Needed only for blueprints with rum: true on a workload, or app workload nodes with a RUM lane.

Synthetic Monitoring (optional)¶

Used by the version-matched Docker provisioner (sm-provision Compose profile; a source checkout may run the binary directly, and separately configured in-cluster Jobs use the dedicated named SM state/Lease without shared disk), not as a synthetic sink. The SM token is a separate bearer token — not GC_TOKEN. The tool must match the current snapshot version. Provisioning and first v2-format activation require explicit apply/restart; later unchanged-spec/target/schema release bumps retain activation. Adoption/migration approval versions remain strict. See Synthetic Monitoring.

See Synthetic Monitoring for the two-phase startup.

Fleet Management (optional)¶

See Fleet Management.

Self-observability — OTLP (optional)¶

Sends synthkit's own RED metrics, traces, and operational logs to a separate stack.

Self-profiling — Pyroscope (optional)¶

Sends the synthkit process's continuous profiles to a separate stack. Follows SELFOBS_ENABLED and is independent of synthetic DRY_RUN.

Getting credentials from Grafana Cloud¶

1. Open your Grafana Cloud stack → Security → Access policies.
2. Create a policy with the scopes you need (at minimum: metrics:write, logs:write, traces:write).
3. Generate a token.
4. Find the endpoint URLs under Details for each data source (Mimir, Loki, Tempo, Profiles).

If you use gcx, gcx config view shows the active configuration with secret values redacted; use gcx config list-contexts and gcx config current-context to select the intended stack deliberately.

Finding the three required numeric identifiers¶

The three user fields are different, but each must be a positive decimal identifier: digits only, greater than zero. Do not put an endpoint URL, a Grafana organisation ID, a slug, or a token in any of them.

1. In the synthetic-data stack, open Connections → Prometheus and copy its displayed instance ID into GC_PROM_USER.
2. Open Connections → OpenTelemetry and copy its displayed stack ID into GC_OTLP_USER.
3. Open Connections → Loki and copy its displayed instance ID into GC_LOKI_USER.

The adjacent connection panels also provide GC_PROM_RW, GC_OTLP_ENDPOINT, and GC_LOKI. Use the endpoint and numeric identifier from the same stack. Presence checks may confirm that a value is non-empty; never print .env or a token while diagnosing a failed identifier check.

Browser readback and optional API credentials¶

For the first run, use the selected stack's Grafana Explore in your ordinary browser, as shown in the Docker Quick Start. Your Grafana login supplies query access; neither gcx nor a second CLI credential is required for this route. Ask the stack administrator for access if you cannot open its data sources.

An optional direct API client needs the corresponding read scopes, such as metrics:read, logs:read and traces:read. A CAP token created with only the minimum ingest write scopes cannot query data and returns invalid scope requested. Do not assume every token is write-only: inspect the granted scopes. Use the credential appropriate to that operation.

The query username is the per-signal instance ID, not the stack ID, and the two are different numbers for the same stack. Mimir wants the Prometheus instance ID and Loki wants the Loki instance ID; the stack ID belongs only to the OTLP gateway. Passing a stack ID to a query endpoint fails with

401 authentication error: invalid authentication credentials

which is the same body a revoked token produces, so a perfectly good credential reads as dead and the real fault is invisible. Resolve the identifiers at run time rather than from memory:

curl -sH "Authorization: Bearer $READ_TOKEN" \
  "https://grafana.com/api/orgs/<org-id>/instances" \
  | jq -r '.items[] | "\(.slug) prom=\(.hmInstancePromId) loki=\(.hlInstanceId)"'

The three 401 bodies are worth telling apart: invalid authentication credentials is a wrong tenant/token pair, invalid scope requested means the token is live but its policy lacks the scope, and invalid token means revoked or malformed.

Filling in .env¶

Create .env with the no-overwrite procedure in the Docker Quick Start, then enter the values in a local editor or a secure terminal prompt. Preserve existing settings; never copy the example over a configured file.

Do not pass the real .env to raw docker compose config; rendered output can expose interpolated credentials. just compose-check uses .env.example as fake input for deployment rendering.

The minimum set for a live synthetic push: GC_TOKEN + GC_PROM_RW/GC_PROM_USER + GC_OTLP_ENDPOINT/GC_OTLP_USER + GC_LOKI/GC_LOKI_USER. Leave optional blocks empty to disable RUM, SM, FM, and self-obs.

Before switching to live mode, run the explicit preflight:

go run ./cmd/synthkit -preflight

For a Compose deployment, run the same command through the image without starting the long-running service:

docker compose run --rm --no-deps synthkit -preflight

Bounded orchestrators may pass -one-shot-deadline <RFC3339-expiry> with exactly one of -validate or -preflight. The command refuses an elapsed deadline before loading configuration, and preflight HTTP probes share the remaining absolute budget. Validation rechecks the deadline before publishing its result; a deadline observed before final success produces a nonzero exit. Always check the command exit status rather than a partial output line. Omitting the flag preserves the ordinary one-shot behavior. This entry and request-context boundary does not retract requests already received by a remote service or roll back receiver-side processing.

The preflight first validates configuration locally, then sends an empty authenticated request to each mandatory ingest lane. It does not emit telemetry. Each network request has a five-second timeout and the command exits non-zero unless all three lanes are ready. Output is deliberately redacted; it names only the lane, state, and a fixed reason code:

prometheus: statically valid
loki: statically valid
otlp: statically valid
prometheus: ready
loki: unauthorized (http-403)
otlp: unreachable (tls)

unauthorized distinguishes HTTP 401 from 403. unreachable distinguishes DNS, TLS, timeout/connection, unexpected HTTP status, and endpoint-path failures. Tokens, endpoint URLs, user IDs, and response bodies are never printed.

Mandatory live values have these static requirements:

Normal dry-run commands remain credential-free and offline. They do not execute preflight probes; network/auth checks occur only when -preflight is supplied. Live startup (DRY_RUN=false) always performs the static checks before constructing sinks, so malformed or transposed endpoints fail synchronously rather than during background delivery.

See Configuration for the full environment variable reference including behaviour knobs (TICK_DEFAULT, SEND_SHARDS, queue tunables, etc.).