Prerequisites¶

Everything here is a one-time setup outside Terraform. Work through it before the first apply; most apply failures trace back to one of these.

Grafana Cloud¶

A stack¶

Any Grafana Cloud stack you can administer. Note its slug (the <slug> in https://<slug>.grafana.net) and URL; they become grafana_cloud_stack_slug and grafana_stack_url in examples/complete.

A dedicated stack is simplest. On a shared stack the module scopes its objects to the demo (see security.md), but it still creates stack-wide objects such as the Knowledge Graph rule file and, if you let it, the Application Observability switch.

Deploying this module twice with the same var.name at the same Grafana Cloud stack collides. Every object name, access policy, folder uid and Agent Observability id derives from name (touchline by default), so a second deployment with the same name fights the first over the same Grafana objects and access policies instead of creating a second, independent set. Give each deployment on a shared stack its own name.

Cloud access policy token (the grafana.cloud provider alias)¶

Create an access policy in the Grafana Cloud portal (Administration > Users and access > Cloud access policies) with realm your organization, and a token for it. Scopes:

Pass it as grafana_cloud_access_policy_token.

Frontend Observability token¶

The Frontend Observability app is managed through its own API, so the grafana.cloud alias also needs frontend_o11y_api_access_token: a Cloud access policy token with frontend-observability:read, frontend-observability:write and frontend-observability:delete on the stack. It can be a second token or the same policy with these scopes added. Pass it as grafana_frontend_o11y_api_access_token. Not needed when frontend_observability_enabled = false.

Admin service account token (the grafana.stack alias)¶

In the stack, Administration > Users and access > Service accounts: create a service account with the Admin role and a token. Admin is required because Agent Observability evaluator, guard and rule writes are Admin-only by default, and the module also creates a service account (for the experiments), a datasource, dashboards and alert rules. Pass it as grafana_stack_service_account_token.

The grafana.stack provider also needs stack_id (the numeric stack id): the Knowledge Graph resources refuse to plan without it. examples/complete reads it from data.grafana_cloud_stack through the grafana.cloud alias.

Agent Observability¶

• Enable the Agent Observability app on the stack (Observability > Agent).
• Point its LLM-judge provider at Amazon Bedrock. The module's llm_judge evaluators ask for provider bedrock and the model in bedrock_models[judge_model] (Haiku by default). The judge runs in Grafana Cloud, not in your cluster, so configure the provider in the app's settings with AWS credentials that can invoke that inference profile in your region. The module does not create those credentials. Without this, deterministic evaluators (regex, heuristic) still work and the guards still fire, but LLM-judge scores never appear.

Knowledge Graph¶

Two independent switches:

• Onboarding: either initialize the Knowledge Graph yourself (Observability > Knowledge Graph, follow the onboarding) before the first apply, or set manage_knowledge_graph = true and let the module do it. The module then creates its own Cloud access policy token and stack Admin service account token to run the onboarding flow, and removes both when the variable goes back to false. Onboarding is a stack-wide singleton: destroying it (or applying with manage_knowledge_graph = false after it was true) disables Knowledge Graph for the whole stack, not just this demo. Leave it false on a shared stack that already has Knowledge Graph on.
• This demo's objects: with knowledge_graph_enabled = true (the default), the module adds a service-graph rule file and a trace configuration scoped to this demo's namespace, once the Knowledge Graph is initialized by either route above.

The team span-metrics dimension used by the agents dashboard stays a manual App Observability step either way; it has no Knowledge Graph or Terraform equivalent (see Application Observability below).

Application Observability¶

The agents dashboard and the service map need Application Observability switched on. Either turn it on yourself (Observability > Application) or set manage_app_observability = true. Read the warning on that variable first: it is a stack-wide singleton, and destroy switches the product off for the whole stack.

One manual step: the agents dashboard splits cost by team using the team span attribute. Add team as a span-metrics dimension in Application Observability settings. There is no Terraform resource for this; without it the per-team panels group everything under one empty team.

Alerting features¶

The rules rely on two alerting features: simplified routing (rules name their contact point directly) and Grafana-managed recording rules (the recording rules write touchline_* series back to the stack's Prometheus datasource). Both are normally on for Grafana Cloud stacks. If a rule group apply fails complaining about notification_settings or record, ask Grafana support to enable alertingSimplifiedRouting and grafanaManagedRecordingRules.

AWS¶

Bedrock model access¶

• In the Bedrock console for your region, request access to the Anthropic models you use (Claude Haiku 4.5 and Claude Sonnet 4.6 with the defaults). Anthropic models also need the one-time use-case details form for the account; until it is approved every call fails with an access error.
• bedrock_models names system cross-region inference profiles (eu., us., apac., global. and so on). The prefix must match the provider region; a plan-time check rejects a mismatch. For us-east-1, for example, use us.anthropic.claude-haiku-4-5-20251001-v1:0.
• SCPs: a cross-region profile serves requests from several regions (an eu. profile called from eu-west-1 is often served elsewhere in the EU). An organization SCP that denies Bedrock outside your home region breaks these calls in ways that look like intermittent access errors. Allow bedrock:InvokeModel* in every destination region of the profiles you pick, or use a sandbox account without that restriction.
• Service quotas: the defaults are light, but Claude tokens-per-minute quotas in a new account can be low. Throttles show up on the Bedrock dashboard and the throttling alert.

An EKS cluster¶

• Bring your own, or create one with examples/eks-auto-mode.
• It needs the EKS Pod Identity agent add-on (built in on Auto Mode). The module creates a Pod Identity association for the agents' service account; without the agent the pods get no AWS credentials.
• The identity running Terraform needs enough Kubernetes RBAC to create a namespace, Secrets, service accounts, namespaced Roles and RoleBindings (Alloy's Kubernetes metadata watch and the experiments job's Lease access are both namespace-scoped, not cluster-wide) and the chart's workloads. Cluster-admin through an EKS access entry is the simple answer, but nothing here needs cluster-scoped RBAC.
• No StorageClass, ingress controller or load balancer controller is required. The site stays ClusterIP unless you set site_ingress.

A subnet for the agent host¶

A subnet in the same account and region with outbound internet (a NAT gateway or equivalent): the host pulls container images, downloads Claude Code, reaches SSM, Secrets Manager, S3 (the render bundle it fetches on first boot, before anything else starts), Bedrock and Grafana Cloud. No inbound access is needed. examples/eks-auto-mode outputs a private subnet that fits. Leave agent_host_subnet_id unset (and agent_host_enabled = false) to run only the in-cluster half.

Permissions for whoever runs apply¶

IAM (roles, policies, instance profile), EKS (Pod Identity associations), Bedrock (inference profiles, and the logging configuration if you enable it), Cognito, EC2 (instance, security group), Secrets Manager, Firehose, CloudWatch (metric stream, log groups), S3 and KMS. AdministratorAccess in a sandbox account is the practical choice for a demo.

Tools¶