Security¶

This is a demo meant for a sandbox AWS account and a demo Grafana Cloud stack. It makes some choices on purpose that you would not make in production. This page lists them so you can decide before you apply.

Content capture¶

content_capture = true (the default) puts prompt text, assistant responses and tool content in telemetry:

• Claude Code: the gateway pushes OTEL_LOG_USER_PROMPTS, OTEL_LOG_ASSISTANT_RESPONSES and OTEL_LOG_TOOL_CONTENT, and full content mode for the Agent Observability plugin. Prompts, responses and tool output land in Loki and Agent Observability.
• OTEL_LOG_TOOL_DETAILS=1 is pushed regardless of content_capture, because without it MCP server and tool names export as custom. It also logs tool inputs, including Bash commands.
• The in-app agents record prompts, responses and tool content in their generations and logs.
• The LLM-judge evaluators send sampled content to the judge model, which Grafana Cloud calls with the Bedrock credentials you configured for the Agent Observability judge provider.

The demo only ever handles fictional data (invented clubs, bookmakers and offers, public open source repositories, and PII probes built from test card numbers and reserved example domains). With real users, capture content only for groups that need it, and treat the stack's logs and Agent Observability data as sensitive.

content_capture = false switches the content settings off for Claude Code and the agents. The prompt-level panels and the content-based evaluators then have nothing to work with.

Credentials in Terraform state¶

Terraform generates or receives every credential the demo uses, so they are all in state:

• Grafana Cloud access policy tokens (ingest: metrics, logs, traces and Agent Observability write; Firehose: metrics and logs write), the PDC network token, and an Admin service account token for the experiments;
• the Cognito app client secret and every developer's password;
• the gateway session signing secret and admin read and write keys, the Postgres passwords;
• the private CA and gateway TLS private keys.

Use an encrypted remote backend with tight access (for example S3 with encryption and a lock, or OpenTofu state encryption). The examples ship without a backend block: local state on your laptop is fine for a short demo, and nothing else.

The same values also live in:

• Kubernetes Secrets in the demo namespace (<prefix>-grafana-otlp, <prefix>-agento11y, <prefix>-faro, <prefix>-experiments). The experiments Secret holds a stack Admin service account token, so anyone who can read Secrets in that namespace has Admin on the stack. Restrict Secret access in the namespace accordingly. The experiments account is Admin because Agent Observability's evaluation and suite writes are Admin-only today and there is no narrower Grafana role to name. A custom role scoped to those permissions replaces it once their exact names are confirmed and documented.
• Secrets Manager (<prefix>/agent-host and the Firehose credentials), readable only by the agent host role and the Firehose roles. They are created with a zero-day recovery window so destroy removes them at once.

The agent host's user data holds no credentials; it reads them from Secrets Manager at boot.

The agent host and the gateway¶

• The instance has no key pair and no inbound security group rules. Access is through SSM Session Manager, so anyone with ssm:StartSession on it is effectively root on the host and can read every credential on it. The instance role itself is narrower than the usual SSM role: it grants Session Manager and Run Command only (not AmazonSSMManagedInstanceCore's ssm:GetParameter* on every parameter in the account), and its ECR pull permissions are scoped to this module's own image repositories rather than every repository in the account.
• The gateway is published on 127.0.0.1:8443 on the host only. Developer containers reach it on the compose network. The gateway additionally only accepts clients from private address ranges.
• TLS uses a private CA generated by Terraform. The login bot checks the gateway certificate against it before signing in. The CA is trusted only inside the developer containers; on your laptop, just login-tunnel pins the gateway's leaf key in a throwaway browser profile instead (see coding-agents.md). Never add the CA to a system or browser trust store: it has no name constraints and its private key is in Terraform state.
• The gateway pushes a separate Agent Observability client token (sigil:write, metrics:write, traces:write) to every signed-in developer as a managed environment variable, so any session can read it. It is scoped to the Claude Code plugin's own generations, metrics and traces, and shares nothing with the ingest token. The ingest token (metrics, logs, traces and Agent Observability write) never enters a developer container: on the host only the gateway's telemetry.forward_to export and the host's own Alloy use it, and in the cluster it sits in the <prefix>-grafana-otlp and <prefix>-agento11y Secrets for Alloy and the in-app agents.
• The network is the boundary around the developer containers. Developers run on their own bridge network, and a host firewall (agent-host/host/agent-host-firewall) keeps that network off everything except DNS, the gateway on 443 and the public internet. The firewall runs before Docker starts and before every compose run. It drops instance metadata (169.254.0.0/16), every RFC 1918 and CGNAT range and the host itself, and drops all IPv6 from the developer and PDC bridges; the instance's IMDS IPv6 endpoint is also disabled. Only the gateway's own fixed address may reach IMDS, for its Bedrock credentials. The same holds on the trusted core bridge, where the gateway also has a fixed address and every other service is dropped on the way to 169.254.169.254. A developer container cannot reach IMDS, the VPC or Postgres. Postgres and the PDC agent sit on a third bridge that the developer network has no route to.
• Developer containers can reach the gateway admin API. It is served on the same port 443 the developers use for inference, so the network does not separate them; the admin endpoints are protected only by the admin read and write keys, which never enter a developer container. A developer's session JWT should be refused there: confirm in your lab that a request to /v1/organizations/spend_limits/effective with a session token instead of x-api-key gets 401 or 403 before relying on it.
• Each developer session auto-approves python3, node and awk (plus read-only git, file tools, jq and the MCP servers) with no curl, rm, npm or npx - the traffic loop's own scripted prompts need an interpreter to write and run small scripts. A prompt that talked a session into running an arbitrary script still runs it inside that isolated network, with nowhere to send anything private and no path to a credential beyond what the container already has. Edits and writes are auto-approved only inside the session's own temporary working directory; Read, Glob and Grep are approved anywhere, and reads and edits under ~/.claude are denied; because python3 and node can still write anywhere the container user can, the container re-seeds its user-scope Claude config (settings.json, the MCP server list, user CLAUDE.md, agents, commands and skills) on every start and before every session, so anything a session persists there is dropped before the next one runs. Treat the containers as semi-trusted scripted demo traffic. Nobody has proven them safe against adversarial input, and the network isolation is what limits a misbehaving script.

Guards and evaluation scope¶

Every Agent Observability object the module creates is prefixed touchline_ and scoped so it only acts on the demo's traffic:

• In-app agent rules match the demo's agent names (touchline-orchestrator and the four specialists).
• The Claude Code online evaluation rule matches agent name claude-code and the tag service.namespace=touchline, which the gateway sets on the demo developers' plugin, so other Claude Code users of the same stack are not scored.
• The Claude Code guards (the hook rules below) match agent name claude-code/touchline and its subagents (claude-code/touchline/*), which the gateway sets for the demo developers through AGENTO11Y_AGENT_NAME. They cannot use the tag: the plugin's guard calls carry no tags (agento11y plugin v0.48.0). Other Claude Code users of the same stack are not matched.

What the guards do:

Guards are a demonstration of the mechanism, not a data-loss control. The plugin's guard calls time out after 5 seconds, and a timeout fails open. Warn rules record and never block. Check redaction behaviour on your own stack before claiming it covers prompts.

Bedrock invocation logging¶

bedrock_invocation_logging_enabled is off by default, for good reasons:

• The setting is account- and region-wide. It captures prompt and completion text for every bedrock-runtime call in the region, from every workload in the account, not only this demo.
• It replaces any existing invocation logging configuration.
• On destroy the module removes the configuration; it does not restore whatever was there before.
• The log group is KMS-encrypted with 7-day retention, large payloads go to an S3 bucket that expires objects after 7 days, and the logs are forwarded to Grafana Cloud Logs.

Enable it only in a dedicated sandbox account.

Other exposure¶

• The EKS Auto Mode example enables the public API endpoint, open to 0.0.0.0/0 unless you set endpoint_public_access_cidrs. Narrow it, or run Terraform from inside the VPC.
• The site has no Ingress by default; you reach it with kubectl port-forward. With site_ingress set, it is as public as your ingress class makes it: read the warning in apps/site/README.md first, because POST /api/picks calls the orchestrator (and so Bedrock) with no authentication or rate limiting, so anyone who reaches it spends your Bedrock budget.
• The Faro collector URL is public by design (it is in every page). Without an Ingress, the Frontend Observability app allows any origin.
• The default contact point delivers nowhere, so the demo alerts page nobody. If you set alert_contact_point, the alerts go to a real destination.
• Container images are public on GHCR and contain no credentials and no Claude Code binary. images.tag (a plain tag) defaults to this module's own release; images.digests lets you pin one or more images immutably by SHA-256 instead, and the agent host never silently re-pulls an image once it has one matching the configured tag or digest. Pin by digest (images.digests) for anything longer-lived than a demo.
• Every published image is signed keylessly with cosign by the shared container-publish reusable workflow that .github/workflows/images.yml calls, so the signing identity is that reusable workflow, not this repository's own workflow file. Verify an image (and its digest) before you pin it, with cosign 3.x (the signatures use the newer bundle format, which cosign 2 reports as "no signatures found"):

cosign verify \
  --certificate-identity-regexp '^https://github\.com/rknightion/\.github/\.github/workflows/container-publish\.yml@' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-github-workflow-repository rknightion/grafana-aio11y-demo \
  ghcr.io/rknightion/grafana-aio11y-demo-gateway:<tag>

--certificate-github-workflow-repository ties the signature to builds started from this repository, since the reusable workflow's identity alone is shared by every repository that calls it.