Agent Observability provisioning (manual)¶

agento11y-provision creates the Agent Observability evaluation resources a blueprint declares under agent_observability: (evaluators, hooks, rules, collections, rule actions and test suites), records them in a private local custody directory, and later disables or tears down what it created. It is a manual, operator-driven command. This page is the one guide to it.

What it is not:

• Not runtime admission. Loading a blueprint with an agent_observability: block, or holding a successful receipt from this command, does not enable any synthkit pipeline. The emitter ignores the block. Runtime admission is separate, later work.
• Not shipped in a release, image or Compose service. The command is source-built only. The published container contains no Agent Observability provisioner, and nothing here supports a container, Kubernetes or HA handoff.
• Create-only. There is no update, adopt, retry, reset or migration path. A declared resource that already differs remotely refuses; changing a created resource means a new key and ID.
• Same user, same directory. Every cooperating invocation must run as the same effective user against the same physical custody directory with the same Binding. Copying, chmod/chown of custody, or reinitialising a directory is never a recovery step.

Build and run¶

From the repository root:

just build-packages ./cmd/agento11y-provision </dev/null
./agento11y-provision --version
./agento11y-provision --help

build-packages runs go build for the named package and leaves ./agento11y-provision in the working directory. just build only compile-checks the module; it does not produce this binary. --version prints the source-build revision, not a release identity.

Declaring resources in a blueprint¶

The block is top level, beside environments and workloads. Each closure names one real agent declared by a workload instance in the same blueprint (agent.instance is the post-fanout workload instance name, agent.agent_name the agent's declared name, which must have a real producer):

agent_observability:
  closures:
  - key: planner
    agent: {instance: pipeline, agent_name: Planner}
    evaluators:
    - key: e
      spec: {evaluator_id: fictional-check, version: "1", kind: regex,
             config: '{"pattern":"fiction","reject":false}',
             output_keys: [{key: passes, type: bool, pass_value: false}]}
    collections:
    - key: c
      spec: {name: Fictional collection}
    rules:
    - key: r
      spec: {rule_id: fictional-rule, enabled: true, selector: user_visible_turn,
             match: '{"agent_name":"Planner"}', sample_rate: 0, evaluator_ids: [e]}
    hooks:
    - key: h
      spec: {rule_id: fictional-hook, enabled: true, phase: preflight, priority: 0,
             match: {agent_name: Planner}, evaluator_ids: [e], action_on_fail: deny,
             blocked_tools: [fictional_tool], redact: [{id: one, regex: fictional_secret}]}
    actions:
    - key: a
      parent_rule: r
      spec: {condition: {kind: all_evaluators_pass},
             action_config: {kind: add_to_collection, collection_ids: [c]}}

Suites nest a draft with its cases, initial bindings and publication review: suites[].{key, spec, draft: {key, spec, cases[], bindings: {key, bindings[]}, publication: {key, spec}}}. BLUEPRINT-SCHEMA.md lists every accepted key.

Rules for the block:

• Every entry has a key unique across the whole blueprint ([A-Za-z0-9_.-]+). References between entries (evaluator_ids, collection_ids, parent_rule, binding evaluator_id) use these local keys and must stay inside their closure; the engine substitutes the acquired IDs.
• spec fields use the exact management API spellings. Unknown keys, case variants, duplicate keys, YAML anchors, aliases, merge keys, hexadecimal or lossy numbers and a trailing YAML document all refuse. Scalars must resolve to plain string, integer, float, boolean or null: an explicit !!str, !!int, !!float, !!bool or !!null tag is accepted but must still match the field's type, and any other tag or type (timestamps, binary, custom tags) refuses. null refuses on a typed field but is kept as a value inside open case maps (input, expected, metadata). false, 0, empty lists and numeric spellings such as 1.00 are kept exactly as written.
• A blueprint that declares the block is held to these scalar, anchor and single-document rules across the whole document, not only inside the block, so no other part of the file can carry hidden authoring input. Keep such blueprints free of anchors and of timestamp-like or other specially tagged values.
• Three aliases are translated once before the plan is built. Evaluator config and hook/rule match accept a native object or a JSON-object string. A hook's blocked_tools: [...] means tool_filter: {blocked_names: [...]} and conflicts with an explicit tool_filter, even an equal or empty one. A hook's redact: [{id, regex}] means redact: {patterns: [...]}.
• The block holds no destination, credential, custody, mode or admission field. Blueprint source is visible through existing operator tooling, so it must never contain secrets.
• Shipped blueprints do not declare this block. Fictional examples belong under command test data or e2e/fixtures/.

Private inputs¶

The Binding, token, preview and approval files must be regular files owned by the invoking user, mode 0600 or stricter, with one link, and every parent directory must be owned by that user or root and not group- or world-writable (a root-owned sticky directory such as /tmp is allowed). A symlink anywhere in the path, including a symlinked parent such as macOS /tmp or /var, refuses: pass the resolved path. The token is never accepted in arguments, the environment or a dotenv file, and the command reads no ambient credential. The token file or standard input must hold the token alone: a trailing newline or any other whitespace or control character refuses, so write it with printf '%s' rather than echo.

The Binding is a JSON object with exactly deployment_id, state_id, contract_revision, source_revision and destination: {grafana_origin, deployment_path, grafana_org, plugin_route, effective_tenant, runtime_endpoint, runtime_tenant}. contract_revision must be agento11yprovision-state-3264k-v2 and source_revision must be sigil-bc963fb09-engine-inventory404-v1; no flag or field selects another. plugin_route is fixed to /api/plugins/grafana-agento11y-app/resources/eval. Requests go to grafana_origin plus deployment_path plus that route over verified TLS, with no proxy and no insecure option. Rotating the token keeps the Binding unchanged; any changed origin, path, org, tenant, runtime endpoint, deployment, state or revision is a Binding mismatch and refuses.

Verbs¶

Any flag a verb does not list refuses, as does a repeated scalar flag. --mode belongs to preview only. --state-dir must already exist (prepare it as a 0700 directory owned by the invoking user); the command creates only its lock and custody files inside it. Pass --expect-enrolled on every invocation after the first successful apply: without it, only a genuinely empty directory may be initialised.

--out must name a file that does not exist, is not directly inside the state directory and is distinct from every input. It is created 0600 and never overwritten. A preview can contain the private Binding, so keep it out of shared or committed locations.

Lifecycle¶

1. Stop every emitter that could produce telemetry for the affected agents and keep them stopped until the receipt is accepted. The command does not check this; the approval asserts it.
2. plan offline and review the digest and counts.
3. preview --out preview.json with the token: an authenticated read of the current inventory.
4. Author the approval from that preview: an ApprovalDocument binding the preview's digest, input, inventory, base, mode and namespaces, with a unique approval_id, the operator, a construction_id (for recover it must equal --construction-id), issued_ms and expires_ms at most 15 minutes apart, and the operator assertions and evidence (authorised operator, effective destination, emitters stopped and kept stopped, exclusive namespace, outstanding work may finish, non-atomic initial publication, permanent records may remain, no synthetic re-entry).
5. apply --preview-file preview.json --approval-file approval.json. The command re-acquires the same state, re-observes the inventory and refuses if anything differs from the approved preview; a stale preview is never refreshed. Each mutation is journalled durably before it is sent, one request per step, and the receipt is built from actual read-back.

Cleanup follows the same preview, approval and mutate sequence with --mode deactivate or --mode teardown and --construction-id naming the construction to clean up. Deselecting or deleting the YAML never removes custody history or produces an empty success.

If a run ends with a pending operation (a lost response, a killed process, exit 3 or 4), an ordinary re-run that reaches the engine refuses with exit 4 (a refusal earlier than that, such as an unreadable approval file, exits 2). Run preview --mode recover --construction-id <id> for the construction that owns the pending operation (the cleanup construction for a lost disable or delete, not the earlier apply), approve that preview with a recovery section carrying independent attribution of the original attempt, then recover. Recovery keeps the original construction's plan, ID and mode; only its preview and approval are in recover mode. Nothing is resent and no current absence clears a pending operation on its own.

Settling the pending operation is its own step. A recover that resolves it records the resolution durably and exits 4, because custody changed; it is not a completion. Run a second preview --mode recover for the same construction, approve it afresh (without a recovery section), and recover again to finish the construction's remaining steps.

Capacity refusal¶

An offline plan and its resource count do not establish that custody can admit an apply. Before any management request, preview forecasts the worst-case custody needed to finish an apply (including a no-op re-apply), deactivation or suffix recovery and then start one canonical full teardown. If that forecast exceeds capacity, preview refuses with exit 2, sends no management request and creates no preview output. Bootstrap and each construction Start repeat the admission check before publishing the new authority; an earlier preview is not a capacity reservation.

The forecast includes maximum encoded backend/approval values, retained history and generation space. There is no universal resource-count ceiling: the actual declaration and existing custody history must fit. A no-op re-apply or deactivation may itself be refused to preserve teardown room; refusal leaves the custody head unchanged. Do not reset, prune or move custody, raise limits, or retry unchanged input to bypass it. Review a refused fresh declaration before provisioning; for an existing construction, follow the cleanup and recovery lifecycle above. A pending operation still requires attributed recovery, not replay.

This is a bounded reservation for one canonical full teardown with one recovery per unsettled step, not permission for arbitrary split cleanup plans or unlimited no-progress suffix Starts. Teardown can still honestly retain unsupported resources and dependencies and report incomplete. The canonical lifecycle contract owns these limits; no runtime admission is enabled.

Output and exit codes¶

Standard output carries only digests, counts, the result and "runtime_admission": false. Standard error carries only a reason code. Paths, destinations, tenants, agent names, tokens and response bodies are never printed.

For apply, recover, deactivate and teardown the command reads the custody head before the engine call and again after any failure, under the same lock. It exits 2 only when the head (generation and checksum, or pristine absence) is unchanged and nothing is pending; a failed re-read also exits 4.

The whole invocation, including the lock wait, is bounded to 10 minutes and each request to 15 seconds. No flag raises these limits. A hard deadline exits 4 without releasing evidence; it is not proof that nothing was committed.

Developing the command¶

The TestPublic* proofs in cmd/agento11y-provision run the built binary against a local TLS server whose certificate they pass through SSL_CERT_FILE/SSL_CERT_DIR. Go honours those variables on Linux only; on macOS the system trust store is used and the proofs fail the TLS handshake. Run them on Linux (the remote gate runner or CI).

Limits of what this proves¶

Operator assertions, the local lock and configuration equality are not backend fencing, queue drain, atomic activation, remote non-commit or anti-rollback. Retained, permanent or unresolved resources are expected, honest outcomes. A bootstrap cannot tell a deleted custody directory from a genuinely new one; protect the directory accordingly.