---
description: This is a demo meant for a sandbox AWS account and a demo Grafana Cloud
  stack. It makes some choices on purpose that you would not make in production.
---

# Security

This is a demo meant for a sandbox AWS account and a demo Grafana Cloud stack. It makes some
choices on purpose that you would not make in production. This page lists them so you can decide
before you apply.

## Content capture

`content_capture = true` (the default) puts prompt text, assistant responses and tool content in
telemetry:

- Claude Code: the gateway pushes `OTEL_LOG_USER_PROMPTS`, `OTEL_LOG_ASSISTANT_RESPONSES` and
  `OTEL_LOG_TOOL_CONTENT`, and full content mode for the Agent Observability plugin. Prompts,
  responses and tool output land in Loki and Agent Observability.
- `OTEL_LOG_TOOL_DETAILS=1` is pushed regardless of `content_capture`, because without it MCP
  server and tool names export as `custom`. It also logs tool inputs, including Bash commands.
- The in-app agents record prompts, responses and tool content in their generations and logs.
- The LLM-judge evaluators send sampled content to the judge model, which Grafana Cloud calls with
  the Bedrock credentials you configured for the Agent Observability judge provider.

The demo only ever handles fictional data (invented clubs, bookmakers and offers, public open
source repositories, and PII probes built from test card numbers and reserved example domains).
With real users, capture content only for groups that need it, and treat the stack's logs and
Agent Observability data as sensitive.

`content_capture = false` switches the content settings off for Claude Code and the agents. The
prompt-level panels and the content-based evaluators then have nothing to work with.

## Credentials in Terraform state

Terraform generates or receives every credential the demo uses, so they are all in state:

- Grafana Cloud access policy tokens (ingest: metrics, logs, traces and Agent Observability write;
  Firehose: metrics and logs write), the PDC network token, and an Admin service account token
  for the experiments;
- the Cognito app client secret and every developer's password;
- the gateway session signing secret and admin read and write keys, the Postgres passwords;
- the private CA and gateway TLS private keys.

Use an encrypted remote backend with tight access (for example S3 with encryption and a lock, or
OpenTofu state encryption). The examples ship without a backend block: local state on your laptop
is fine for a short demo, and nothing else.

The same values also live in:

- Kubernetes Secrets in the demo namespace (`<prefix>-grafana-otlp`, `<prefix>-agento11y`,
  `<prefix>-faro`, `<prefix>-experiments`). The experiments Secret holds a stack Admin service
  account token, so anyone who can read Secrets in that namespace has Admin on the stack. Restrict
  Secret access in the namespace accordingly. The experiments account is Admin because Agent
  Observability's evaluation and suite writes are Admin-only today and there is no narrower Grafana
  role to name. A custom role scoped to those permissions replaces it once their exact names are
  confirmed and documented.
- Secrets Manager (`<prefix>/agent-host` and the Firehose credentials), readable only by the
  agent host role and the Firehose roles. They are created with a zero-day recovery window so
  destroy removes them at once.

The agent host's user data holds no credentials; it reads them from Secrets Manager at boot.

## The agent host and the gateway

- The instance has no key pair and no inbound security group rules. Access is through SSM Session
  Manager, so anyone with `ssm:StartSession` on it is effectively root on the host and can read
  every credential on it. The instance role itself is narrower than the usual SSM role: it grants
  Session Manager and Run Command only (not `AmazonSSMManagedInstanceCore`'s
  `ssm:GetParameter*` on every parameter in the account), and its ECR pull permissions are scoped
  to this module's own image repositories rather than every repository in the account.
- The gateway is published on `127.0.0.1:8443` on the host only. Developer containers reach it on
  the compose network. The gateway additionally only accepts clients from private address ranges.
- TLS uses a private CA generated by Terraform. The login bot checks the gateway certificate
  against it before signing in. The CA is trusted only inside the developer containers; on your
  laptop, `just login-tunnel` pins the gateway's leaf key in a throwaway browser profile instead
  (see [coding-agents.md](../coding-agents.md)). Never add the CA to a system or browser trust store:
  it has no name constraints and its private key is in Terraform state.
- The gateway pushes a separate Agent Observability client token (`sigil:write`,
  `metrics:write`, `traces:write`) to every signed-in developer as a managed environment variable,
  so any session can read it. It is scoped to the Claude Code plugin's own generations, metrics
  and traces, and shares nothing with the ingest token. The ingest token (metrics, logs, traces
  and Agent Observability write) never enters a developer container: on the host only the
  gateway's `telemetry.forward_to` export and the host's own Alloy use it, and in the cluster it
  sits in the `<prefix>-grafana-otlp` and `<prefix>-agento11y` Secrets for Alloy and the in-app
  agents.
- The network is the boundary around the developer containers. Developers run on their own bridge
  network, and a host firewall (`agent-host/host/agent-host-firewall`) keeps that network off
  everything except DNS, the gateway on 443 and the public internet. The firewall runs before
  Docker starts and before every compose run. It drops instance metadata (`169.254.0.0/16`), every
  RFC 1918 and CGNAT range and the host itself, and drops all IPv6 from the developer and PDC
  bridges; the instance's IMDS IPv6 endpoint is also disabled. Only the gateway's own fixed address
  may reach IMDS, for its Bedrock credentials. The same holds on the trusted core bridge, where the
  gateway also has a fixed address and every other service is dropped on the way to
  `169.254.169.254`. A developer container cannot reach IMDS, the VPC or Postgres. Postgres and the
  PDC agent sit on a third bridge that the developer network has no route to.
- Developer containers can reach the gateway admin API. It is served on the same port 443 the
  developers use for inference, so the network does not separate them; the admin endpoints are
  protected only by the admin read and write keys, which never enter a developer container. A
  developer's session JWT should be refused there: confirm in your lab that a request to
  `/v1/organizations/spend_limits/effective` with a session token instead of `x-api-key` gets 401
  or 403 before relying on it.
- Each developer session auto-approves `python3`, `node` and
  `awk` (plus read-only git, file tools, `jq` and the MCP servers) with no `curl`, `rm`, `npm` or
  `npx` - the traffic loop's own scripted prompts need an interpreter to write and run small
  scripts. A prompt that talked a session into running an arbitrary script still runs it inside
  that isolated network, with nowhere to send anything private and no path to a credential beyond
  what the container already has. Edits and writes are auto-approved only inside the session's own
  temporary working directory; `Read`, `Glob` and `Grep` are approved anywhere, and reads and
  edits under `~/.claude` are denied; because
  `python3` and `node` can still write anywhere the container user can, the container re-seeds
  its user-scope Claude config (`settings.json`, the MCP server list, user `CLAUDE.md`, agents,
  commands and skills) on every start and before every session, so anything a session persists
  there is dropped before the next one runs. Treat the containers as semi-trusted scripted demo
  traffic. Nobody has proven them safe against adversarial input, and the network isolation is what
  limits a misbehaving script.

## Guards and evaluation scope

Every Agent Observability object the module creates is prefixed `touchline_` and scoped so it only
acts on the demo's traffic:

- In-app agent rules match the demo's agent names (`touchline-orchestrator` and the four
  specialists).
- The Claude Code online evaluation rule matches agent name `claude-code` and the tag
  `service.namespace=touchline`, which the gateway sets on the demo developers' plugin, so other
  Claude Code users of the same stack are not scored.
- The Claude Code guards (the hook rules below) match agent name `claude-code/touchline` and its
  subagents (`claude-code/touchline/*`), which the gateway sets for the demo developers through
  `AGENTO11Y_AGENT_NAME`. They cannot use the tag: the plugin's guard calls carry no tags
  (agento11y plugin v0.48.0). Other Claude Code users of the same stack are not matched.

What the guards do:

| Rule | Phase | Action |
|---|---|---|
| `touchline_claude_code_pii_gate` | preflight | deny: card number, US SSN or UK National Insurance number in a prompt |
| `touchline_claude_code_secrets` | preflight | warn: API keys, tokens, private keys |
| `touchline_claude_code_redact_api_keys`, `_redact_common_pii` | preflight | redact rules, warn mode |
| `touchline_claude_code_redact_tool_secrets` | postflight | redacts secrets in tool content |
| `touchline_claude_code_content_safety` | preflight | warn: toxic content (LLM judge) |
| `touchline_agents_injected_tool_result` | preflight | warn: instructions injected through tool results |

Guards are a demonstration of the mechanism, not a data-loss control. The plugin's guard calls
time out after 5 seconds, and a timeout fails open. Warn rules record and never block. Check
redaction behaviour on your own stack before claiming it covers prompts.

## Bedrock invocation logging

`bedrock_invocation_logging_enabled` is off by default, for good reasons:

- The setting is account- and region-wide. It captures prompt and completion text for every
  `bedrock-runtime` call in the region, from every workload in the account, not only this demo.
- It replaces any existing invocation logging configuration.
- On destroy the module removes the configuration; it does not restore whatever was there
  before.
- The log group is KMS-encrypted with 7-day retention, large payloads go to an S3 bucket that
  expires objects after 7 days, and the logs are forwarded to Grafana Cloud Logs.

Enable it only in a dedicated sandbox account.

## Other exposure

- The EKS Auto Mode example enables the public API endpoint, open to `0.0.0.0/0` unless you
  set `endpoint_public_access_cidrs`. Narrow it, or run Terraform from inside the VPC.
- The site has no Ingress by default; you reach it with `kubectl port-forward`. With
  `site_ingress` set, it is as public as your ingress class makes it: read the warning in
  [apps/site/README.md](https://github.com/rknightion/grafana-aio11y-demo/blob/main/apps/site/README.md) first, because `POST /api/picks` calls the orchestrator
  (and so Bedrock) with no authentication or rate limiting, so anyone who reaches it spends your
  Bedrock budget.
- The Faro collector URL is public by design (it is in every page). Without an Ingress, the
  Frontend Observability app allows any origin.
- The default contact point delivers nowhere, so the demo alerts page nobody. If you set
  `alert_contact_point`, the alerts go to a real destination.
- Container images are public on GHCR and contain no credentials and no Claude Code binary.
  `images.tag` (a plain tag) defaults to this module's own release; `images.digests` lets you pin
  one or more images immutably by SHA-256 instead, and the agent host never silently re-pulls an
  image once it has one matching the configured tag or digest. Pin by digest (`images.digests`)
  for anything longer-lived than a demo.
- Every published image is signed keylessly with cosign by the shared `container-publish`
  reusable workflow that `.github/workflows/images.yml` calls, so the signing identity is that
  reusable workflow, not this repository's own workflow file. Verify an image (and its digest)
  before you pin it, with cosign 3.x (the signatures use the newer bundle format, which cosign 2
  reports as "no signatures found"):

  ```bash
  cosign verify \
    --certificate-identity-regexp '^https://github\.com/rknightion/\.github/\.github/workflows/container-publish\.yml@' \
    --certificate-oidc-issuer https://token.actions.githubusercontent.com \
    --certificate-github-workflow-repository rknightion/grafana-aio11y-demo \
    ghcr.io/rknightion/grafana-aio11y-demo-gateway:<tag>
  ```

  `--certificate-github-workflow-repository` ties the signature to builds started from this
  repository, since the reusable workflow's identity alone is shared by every repository that
  calls it.