Control Plane¶

synthkit embeds an operator control plane served on port 8088 (configured via JSON_HTTP_ADDR). It has two surfaces:

• Operator UI at /control/ui — a browser dashboard for live runtime management.
• HTTP API — a JSON API used by the UI, the Grafana Infinity datasource, and curl/automation.

With CONTROL_TOKEN set, every control route except sanitized GET /control/readiness and minimal GET /control/startup requires HTTP Basic auth. The sibling Infinity JSON host keeps only /healthz public. Loopback may remain token-free; non-loopback exposure has an additional fail-closed acknowledgement gate.

State vocabulary¶

These terms describe different points in the blueprint lifecycle:

• Loaded means the blueprint passed startup loading and is part of the running process.
• Enabled means a loaded blueprint is not in the control state's disabled list. It can be disabled or enabled live; this is not a YAML enabled: field.
• Emitting means the running blueprint has produced telemetry observed by the sink-status report. It is a runtime outcome, not a promise made by loading. The opt-in public probe field blueprint_emitting (see Per-blueprint flags on the public probe) is a different, state-derived signal: enabled and scheduled to tick, not observed delivery.
• Staged means a custom upload or fetched git blueprint has been written to the staging area for a future restart.
• Pending means staged custom/git content differs from the startup manifest, so a restart will add, remove, or update loaded content.
• Active describes a runtime incident scenario currently applied to its qualified blueprint/scenario ID.

Use GET /control/status for sink readiness and per-blueprint emission, GET /control/inventory for the live emission/cardinality inventory and safe query identity, GET /control/health for per-construct tick and process health, GET /control/diagnostics for startup/load problems, and GET /control/blueprints/pending for staged-versus-running changes. GET /control/blueprints/staged lists staged custom/git blueprints.

Operator UI¶

Open http://127.0.0.1:8088/control/ui (or the host's address if exposed).

Managed console features¶

CONTROL_MANAGED_FEATURES optionally contains a JSON object such as {"blueprint_sources":"Managed by operator","custom_uploads":"Use the approved blueprint process","reset":"Contact the operator to reset"}. The only admitted keys are blueprint_sources, custom_uploads, and reset; each value must be a nonempty reason. Unknown keys and malformed objects fail startup. An empty variable or {} preserves the existing index response bytes.

The console disables remote-source add/fetch/delete controls, custom-upload validate/save/delete controls, and reset respectively, and displays the configured reason as text. Runtime metadata beside the base-path tags is HTML-escaped before injection. Built assets are shared unchanged across configurations. This is presentation only, not authorization: direct API reads and mutations remain available under the existing authentication/exposure policy. Do not use these locks as a security boundary. Compose passes the variable through its selected env file.

Reverse-proxy prefix¶

Set CONTROL_BASE_PATH=/x/y for the external console URL /x/y/control/ui/. The same UI build works at any configured prefix: the server inserts a trusted <base href="/x/y/control/ui/"> and API prefix /x/y/control/ into the page. Navigation, assets, deep links and the trailing-slash redirect use that prefix. Empty (the default) keeps the existing /control/ui/ and /control/ URLs.

The prefix is a canonical absolute path, without a trailing slash, encoding, empty segments, ./.. segments, query or fragment. Only ASCII letters, digits, ., _, ~ and - within segments are accepted. Invalid configuration fails loading. X-Forwarded-Prefix and other request headers are never trusted; the operator must configure the actual public prefix. This setting does not replace the token/exposure requirements above.

A proxy can strip /x/y before forwarding /control/ to synthkit, or preserve the prefix when the host router mounts the control handler at /x/y/control/. The handler strips a preserved prefix before dispatch, while injecting external URLs for either proxy mode. Both ordinary and HA modes configure the base path before serving and mount the handler at CONTROL_BASE_PATH + "/control/" as well as /control/ for stripping proxies. With an empty prefix, /control/ is mounted once. Do not apply http.StripPrefix a second time. The prefix is not a new API wrapper: Infinity GET payloads and existing root-host routes are unchanged.

The UI provides:

• Overview — per-blueprint emission status, sink readiness strip, dry-run indicator.
• Scenarios — activate and deactivate named incident scenarios defined in blueprints.
• Scaling — set live workload pod counts within blueprint-declared bounds.
• Failures — ad-hoc failure injection (any mode, any target, any intensity).
• Load — master volume multiplier (scales the shape-driven synthetic volume across blueprints; the ai_agent workload is paused by 0 but not scaled by other values, see Pausing volume with 0).
• Blueprints — enable/disable blueprints; manage custom and git-sourced blueprints.
• Constructs / Kinds — toggle individual construct instances or all constructs of a kind.
• Diagnostics — load-time problems (skipped blueprints, dropped config entries).
• Config — redacted runtime config view.

The schema driving the UI (available modes, targets, scenarios, scalable workloads) is derived from the currently loaded blueprints at startup — it is never hardcoded.

With no BLUEPRINT_NAMES selection, the UI remains available in a distinct Setup required state and states that no synthetic telemetry is emitted. The public readiness probe remains healthy so the container does not restart-loop, while live_ready stays false until a selected blueprint has completed its intended delivery lanes.

Authentication¶

When CONTROL_TOKEN is set:

• All control reads except sanitized /control/readiness and minimal /control/startup, all Infinity data routes except /healthz, and every mutation require HTTP Basic auth: username control, password = CONTROL_TOKEN.
• A browser hitting a guarded route for the first time triggers Chrome/Firefox's native credential dialog.
• The Grafana Infinity datasource stores the password in secureJsonData and authenticates its server-side reads and infinity actions. Browser-direct (fetch) dashboard action buttons do not inherit datasource credentials; they use the browser's separate Basic challenge and therefore require a trusted HTTPS endpoint.

For the mutation examples below, run this once in Bash or Zsh. When CONTROL_TOKEN is set it uses a mode-0600 temporary netrc file, keeping the token out of process arguments. When the token is unset, requests remain unauthenticated:

control_auth=()
control_netrc=
if [ -n "${CONTROL_TOKEN:-}" ]; then
  control_netrc="$(mktemp)"
  chmod 600 "$control_netrc"
  printf 'machine 127.0.0.1 login control password %s\n' "$CONTROL_TOKEN" > "$control_netrc"
  control_auth=(--netrc-file "$control_netrc")
  trap 'test -z "$control_netrc" || rm -f "$control_netrc"' EXIT
fi

# Example: uses the temporary netrc when CONTROL_TOKEN is set
curl -fsS "${control_auth[@]}" -X POST http://127.0.0.1:8088/control/load \
  -H "Content-Type: application/json" \
  -d '{"volume_multiplier": 2.0}'

Customer control dashboard¶

synthkit-control-dash generates the customer dashboard with -action-mode fetch|infinity and -ds-uid. fetch is the default and preserves browser-direct POST actions. infinity sends each POST server-side through the named Infinity datasource and requires both -ds-uid and -write-base-url. Grafana's vizActionsAuth feature toggle must be enabled for Infinity actions; see Grafana's feature note.

In Infinity mode the control-plane credential is stored in the Infinity datasource, and -write-base-url must be an absolute HTTP or HTTPS URL without credentials, a query or a fragment that the datasource reaches. It must not end in /control; generated actions append /control/.... Without -write-base-url in fetch mode, buttons POST to relative /control/... paths on Grafana's own origin and work only when a reverse proxy on that origin routes /control/ to the synthkit control plane. Otherwise pass a browser-reachable -write-base-url. In infinity mode, plain HTTP is acceptable only for the hop from Grafana through Private Data Source Connect (PDC), or from inside the cluster, to a ClusterIP control-plane Service. Never expose the control plane over plain HTTP across the public internet; use HTTPS elsewhere. Anyone who can query the Infinity datasource can issue authenticated control POSTs and change load or scenarios, even without viewing this dashboard or using its action buttons. Restrict datasource query access and dashboard visibility to operators. Grafana's Viewer restriction on dashboard actions does not protect the datasource from direct queries. The upstream contributor reported a Grafana 13.3 click-test through PDC; synthkit's offline generation checks do not verify live action execution.

For an instructor console, pass -layout control-plane -action-mode infinity with -ds-uid, -prom-uid and -loki-uid. The generated dashboard has status tiles, one Start/Stop card per scenario, emitted-family impact charts for eligible target services, collapsed fleet controls and a three-hour default range. -folder annotates its destination folder. Impact charts are omitted when the target node has no matching emitted metric or log family.

Endpoint reference¶

Read-only (GET; authenticated when CONTROL_TOKEN is set, except startup/readiness)¶

The root path / serves the Infinity JSON host. Every Infinity data route requires the same Basic auth when CONTROL_TOKEN is set; only /healthz remains public.

Mutations (POST / DELETE — guarded by CONTROL_TOKEN when set)¶

Array-bearing mutation routes are replacements, not additive patches: submitting an array omits every existing item not included in that request. For routine one-at-a-time scenario and blueprint changes, use /control/scenarios/{activate,deactivate} and /control/blueprints/{disable,enable} instead. The item routes are idempotent and avoid replacing another operator's change.

Startup versus delivery readiness¶

/app/synthkit -startupcheck makes a credential-free GET to /control/startup and succeeds only on HTTP 200. The JSON body contains only {"started":true} or {"started":false}. An unwired/incomplete boot predicate returns 503; an unstarted process or fatal initialization failure cannot serve a successful probe. The endpoint is available at the root control mount and the configured CONTROL_BASE_PATH mount, like readiness. POST is not supported.

In ordinary mode, startup means the existing fatal boot checks and runner/control construction finished and HTTP is serving. In lease mode, the existing config/build/credential preflight boundary must also have completed before the view is published. It does not require election, a writable state volume, every selected source loading successfully, or a first event/ACK. Existing fail-closed configuration and source-admission rules are not bypassed or redefined. A degraded process that the existing startup rules allow to run can pass this check.

Both CLI probes use JSON_HTTP_ADDR directly (default 127.0.0.1:8088, unspecified binds mapped to loopback), without reading -env or starting telemetry. Both retain a five-second context and four-second HTTP-client timeout. -startupcheck cannot be combined with another operating mode or -one-shot-deadline.

Startup success is not ongoing liveness, delivery readiness, backend readback, state writability, HA leadership/fencing authority, or source/custody admission. A selected DRY_RUN=true process can finish boot while -healthcheck still exits nonzero and /control/readiness reports ready=false, live_ready=false. A sparse live request lane has no guaranteed first-event deadline: missing delivery must still hold readiness red. Do not raise traffic rates, synthesize an event, disable a probe, or relax readiness to finish boot.

Older images do not recognize -startupcheck. Chart defaults retain the existing -healthcheck for startup and readiness for compatibility with their pinned image. Only opt into a startup-only command after verifying the exact compatible image; do not change readiness to this command or claim that a successful startup probe makes a selected dry-run deployment live-ready.

Delivery lanes in readiness¶

Readiness requires every expected lane of the delivery manifest to be live-ready. Each lane in readiness.lanes of authenticated GET /control/status carries the manifest's freshness class as request_driven, and the class decides what live-ready means:

A request-driven lane pushes only when requests were minted, so at a real low-traffic rate a quiet period after a success is not evidence of a fault and keeps readiness green. Its first push is still required: a request-driven lane that has never pushed holds readiness red, as above. A lane fed by both a scheduled and a request-driven contributor is scheduled.

Per-blueprint flags on the public probe¶

GET /control/readiness is public so Docker can call it without credentials, and by default it names no blueprint: it carries aggregate blueprints.loaded, skipped and active counts only. When CONTROL_TOKEN is configured, an operator that does not hold it therefore cannot see per-blueprint scheduling state. READINESS_BLUEPRINT_FLAGS=true (default false) adds it to the public probe itself: whether each loaded blueprint is enabled and scheduled to tick, not whether telemetry was delivered.

• Off (default). The response is byte-identical to the probe without this setting; a golden test pins the exact bytes. The setting is read once at startup.
• On. One field is appended after the existing ones: "blueprint_emitting":{"alpha":true,"beta":false}. The key is the blueprint's plain name and the value is a bare boolean, so nothing else can ride along per blueprint: no reason, error, path, lane, source or secret. The object is {} when nothing is loaded and keys are sorted. The HTTP status, ready and reason_codes are unchanged by the flags. GET /control/status is unchanged.
• What true means. The blueprint is loaded (skipped blueprints are not loaded, so they are absent, not false) and is not in the control state's disabled list; at least one of its schema constructs is enabled, meaning its kind is not in the disabled kinds and its blueprint/kind:name key is not in the disabled constructs; and the process is in a state that runs ticks: running, serving HTTP, not in setup mode, leading when HA is enabled (a standby reports all false), and with no HA bootstrap or transition outstanding. It is a statement about control state, not an observation of delivery. Sink health stays in ready, live_ready and reason_codes, and a blueprint in DRY_RUN reads true while live_ready is false. For observed push history use authenticated GET /control/status.
• Naming decision: plain name. Options were the plain name, an opaque or hashed id, or not exposing blueprints. A blueprint name is topology and can carry operator meaning, so listing it weakens the sanitized-probe contract on purpose, and only when the operator opts in. An unsalted hash of a short, guessable name hides nothing, while a keyed id needs a new secret and gives the consumer a mapping to maintain, which defeats the purpose of reading the probe without credentials. The operator chose the names and is the one enabling the setting, so the plain name is the honest option: enabling it publishes the names of loaded blueprints to anyone who can reach the port. Expose the port accordingly, for example only on a trusted network or behind a proxy that limits /control/readiness. A test pins that keys are plain loaded names and that names in the disabled list but not loaded never appear.
• Browser reach. The control plane's CORS wrapper echoes any request Origin on every route, including /control/readiness. With the setting on, a web page opened in a browser that can reach the port can therefore read the blueprint names cross-origin, not only a process on that network. A trusted-network exposure must account for that, for example by keeping the port off networks that run untrusted browsers or by limiting /control/readiness at a proxy. CORS behaviour is unchanged by this setting.
• Limits of the construct test. It reads the control schema's construct list, which has no workloads: workloads are not enabled or disabled per instance and keep ticking while the blueprint is enabled. A blueprint that declares workloads and whose constructs are all disabled therefore reads false although its workloads still run. A loaded blueprint with no schema constructs at all keeps its blueprint-level state, since the construct test cannot judge it. With no schema source configured the construct test is skipped for every blueprint and only the blueprint-level conditions apply. Use authenticated GET /control/inventory for what a blueprint actually produces.

Common operations¶

Verify a selected blueprint without container shell access¶

Use the control surfaces in this order: /control/readiness, authenticated /control/status, /control/diagnostics, /control/inventory, and /control/health. The status payload names a blocked delivery lane; diagnostics names a rejected blueprint or load problem; health names a failing construct tick. The X-ray view renders the same inventory, including each construct's scope and safe query identity, so a blueprint-scoped family can be queried with its declared blueprint selector while a substrate family is queried by declared substrate identity.

After readiness, wait two declared emission intervals plus the delivery deadline before treating a missing declared signal class as evidence. Use the inventory's exact metric names and query identity instead of inventing a hand-maintained mapping. Logs and traces are checked by their listed source or service identity in the same bounded window. A 200 from a mutation only proves that control state changed; the before/after emitted-data observation remains a separate acceptance check.

Activate a scenario¶

curl -fsS "${control_auth[@]}" -X POST http://127.0.0.1:8088/control/scenarios/activate \
  -H "Content-Type: application/json" \
  -d '{"scenario": "mine/db-pressure"}'

Scenarios are identified as blueprint-name/scenario-name. Item activation/deactivation is safe when another client may be changing a different scenario at the same time. To intentionally replace the whole list (including deactivating all scenarios), use POST /control/scenarios with {"active_scenarios": []}.

Scale a workload live¶

curl -fsS "${control_auth[@]}" -X POST http://127.0.0.1:8088/control/scaling \
  -H "Content-Type: application/json" \
  -d '{"mine/mine-api": 8}'

Node count cascades: the k8s cluster and EC2 construct both re-derive their node counts via fixture.DeriveNodes so the substrate stays consistent. Scale-down retires old pod and node series automatically (state.DropWhere).

Inject a failure ad-hoc¶

curl -fsS "${control_auth[@]}" -X POST http://127.0.0.1:8088/control/failures \
  -H "Content-Type: application/json" \
  -d '{"connection_saturation": {"enabled": true, "intensity": 0.7, "scope": "mine-db"}}'

Check sink readiness¶

curl -fsS "${control_auth[@]}" http://127.0.0.1:8088/control/status | jq '.sinks[] | {sink, last_success_ms, failures, last_acknowledgement}'

dry_run: true in the response means no live push is happening regardless of sink health — re-check DRY_RUN in your .env.

last_acknowledgement distinguishes protocol acceptance, partial rejection and unverified or deferred outcomes. HTTP transport success alone is not a durable checkpoint receipt or backend readback. See checkpoint foundations for units and current limits.

Boost volume temporarily¶

curl -fsS "${control_auth[@]}" -X POST http://127.0.0.1:8088/control/load \
  -H "Content-Type: application/json" \
  -d '{"volume_multiplier": 3.0}'

Pausing volume with 0¶

volume_multiplier accepts 0 to 100. An explicit 0 is applied as a multiplier of zero: it is never read as unset (full volume) and never rejected with a 400:

• Scaling. The multiplier multiplies the shape factor (diurnal, weekly and environment weight) that volume-shaped series draw on, so at 0 that factor is exactly 0, below the deepest overnight trough, and series computed as a multiple of it read zero. The web_service and app workloads treat exactly 0 as a pause instead of settling at their off_peak_rps floor or the one-call keep-alive of a prod trough, and the ai_agent workload mints no conversations at 0 (see below):
  • no requests are minted, so no correlated traces, logs or RUM events are produced, and the derived span, RED and service-graph call counts, including the Beyla lane's, add nothing;
  • no Pyroscope profile samples are pushed;
  • an app node's spec metrics (inline or from a profile) change as follows. Counters and histograms observe nothing and stay flat at their last value. A gauge whose value is a shape value reads 0. A gauge with any other value (const, normal or a range) describes the process itself and keeps emitting, which today means go_goroutines, node_process_resident_memory_bytes, node_eventloop_lag_seconds and nodejs_eventloop_lag_seconds from the catalog profiles plus any such gauge a blueprint declares. Constant target_info identity series also keep emitting.

Any non-zero value, such as 0.01, scales exactly as before. The ai_agent workload is driven by its own sessions_per_min and no non-zero value scales it; at 0 it mints no conversations, so no sessions, generations, workflow steps, scores or spans are written and its gen_ai metrics hold their last cumulative value on both the Prometheus and native OTLP lanes. The k8s_profiling construct pushes no cluster profiles at 0. Constructs whose series are not scaled by volume, such as inventory gauges, keep emitting. It is a volume pause, not a process or blueprint stop. - Unset is different from 0. A control snapshot that has no volume_multiplier key means 1.0, as does a fresh store. A persisted 0 reloads as 0 after a restart; a negative persisted value is treated as corrupt and falls back to 1.0. POST /control/reset returns to 1.0. - Upgrade note. Before this behaviour, a persisted 0 (in the file store or an HA state backend) was read back as the unset default and ran at full volume. After upgrading, such a persisted 0 pauses volume on the next start. Check volume_multiplier in GET /control/state if an old snapshot may hold a 0 you did not mean as a pause, and post {"volume_multiplier": 1} to restore it. - Readiness. GET /control/readiness is unchanged at 0: volume is not an input to ready, live_ready or reason_codes. blueprint_emitting is also unchanged: it reports scheduling state (see Per-blueprint flags on the public probe), so a loaded, enabled blueprint stays true at 0 because it still ticks. Read the live multiplier from GET /control/state (volume_multiplier) or the synthkit_volume_multiplier self-observability gauge, and restore volume with {"volume_multiplier": 1}.

Disabling kinds at startup¶

POST /control/kinds needs a running process, so the first tick has already gone out by the time it is called. To keep a kind dark from the first tick, set STARTUP_DISABLED_KINDS to a comma-separated list of bare construct kinds (see configuration.md):

STARTUP_DISABLED_KINDS=synthetic_monitoring,fleet_management

The setting is generic: any construct kind the catalog registers is valid, whether or not a selected blueprint declares it, so setup mode and a partial boot set accept the same value. A kind the catalog does not register (a typo) fails startup with an error naming it, using the same kind check as POST /control/kinds.

Precedence. The effective disabled_kinds at boot is the list persisted in the snapshot at CONFIG_SNAPSHOT_PATH (or the Kubernetes control document) unioned with the startup list. The union is written back through the ordinary state mutation path and applied before the first tick, so GET /control/state reports the effective list from the first request. With HA enabled, only the leader writes it, during activation.

• POST /control/kinds still replaces the whole list and persists the replacement. A replacement that omits a startup kind re-enables it for the rest of the run, except synthetic_monitoring: its lane was removed from the process at startup, so omitting it does not restore the lane (see below).
• On the next restart the startup kinds are added again on top of whatever was persisted, so removing a kind from the live list never outlives a restart while the variable still names it.
• Removing a kind from STARTUP_DISABLED_KINDS does not re-enable it while the snapshot still lists it: replace the list with POST /control/kinds first.
• POST /control/reset clears the list along with the rest of the state for the rest of the run; the next restart adds the startup kinds again.

Disabling a kind stops its construct ticks, so no series of that kind is pushed. Two kinds also have a side effect outside the tick:

• fleet_management: while the kind is disabled, at startup or through POST /control/kinds, the Fleet Management controller is suspended. It makes no request of any kind: no registration, no heartbeat and no unregistration. Collectors an earlier run or earlier heartbeats registered stay registered (they simply stop heartbeating), and enabling the kind again resumes heartbeats. Disabling the kind never unregisters anything (ordinary roster shrinkage while the kind is enabled still does; teardown is the explicit -fleet-unregister path, and -fleet-unregister-owned removes everything the ownership ledger records). See fleet-management.md.
• synthetic_monitoring: the lane is removed before the provisioning handoff, as an unregistered lane is, so the provisioner's snapshot is removed (an empty lane set invalidates any prior one) and sm-provision refuses to run with "private snapshot is unavailable or invalid"; nothing is registered with the Synthetic Monitoring API. The kind stays valid in POST /control/kinds and listed in /control/schema kinds, so re-posting the list GET /control/state returned works. Because the lane no longer exists in the process, a replacement that omits synthetic_monitoring does not bring it back: it stays absent until the variable is removed and the process restarts.

State persistence¶

Control state (volume multiplier, active scenarios, scaling, disabled blueprints/constructs/kinds, and the live-applied blueprint set with its owner labels) persists across restarts in the snapshot file at CONFIG_SNAPSHOT_PATH. The file is written lazily on the first mutation — it is normal for it not to exist on a fresh deploy.

State is not written on a clean shutdown (it is already in the file from the last mutation). It is written only when a mutation succeeds.

See also¶

• incidents.md — declaring and triggering incident scenarios in blueprints
• custom-blueprints.md — uploading and managing custom blueprints
• configuration.md — CONTROL_TOKEN, CONTROL_EXPOSURE_ACK, binds, and state paths