Secrets¶

No Grafana credential belongs in Git, a request object, Composition input, status, or function log.

Organization credential¶

Create a Grafana Cloud access policy token with only the organization-level capabilities needed to manage stacks and their Cloud resources. Store it in the external secret manager as JSON:

{
  "cloud_access_policy_token": "REPLACE_SECURELY"
}

The example expects this document at /platform/grafana-cloud/organization/credentials. Do not put the real token directly in a shell command, terminal history, CI variable dump, or Kubernetes manifest — use your secret-management workflow or a permission-restricted temporary file.

deploy/aws/secret-store-and-credentials.yaml wires this in three pieces:

A SecretStore (grafana-vending-secrets) pointing at AWS Secrets Manager.
An ExternalSecret (grafana-cloud-org-credentials) that reads the token from /platform/grafana-cloud/organization/credentials and templates it into the JSON shape the provider expects, refreshed hourly.
A namespaced ProviderConfig (grafana-cloud-org) that references the generated Kubernetes Secret and is used for all organization-level Cloud operations.

Namespace RBAC should prevent request authors from reading the generated Secret directly.

Rotating administrator token¶

For every stack, the Composition creates:

A StackServiceAccount with the Admin role.
A StackServiceAccountRotatingToken, created after Grafana reports the service-account ID — this is why a brand-new request needs at least two reconciliation passes before its credentials exist.
A Kubernetes connection Secret written by the rotating-token resource.
A PushSecret that exports a structured administrator document.
An ExternalSecret that reads the exported token and URL back into the stack namespace.
A stack-local ProviderConfig used for Grafana resources inside that stack.

The token lifetime is 30 days with a seven-day early rotation window. The PushSecret refresh interval is one hour, so a newly rotated token is copied to the external store well inside the overlap window.

The exported document has this shape:

{
  "stack_name": "Example stack",
  "stack_slug": "example",
  "stack_url": "https://example.grafana.net",
  "stack_region": "prod-us-central-0",
  "usage": "development",
  "change_reference": "CHANGE-EXAMPLE",
  "configuration_item_reference": "CONFIG-EXAMPLE",
  "stack_service_account_token": "GENERATED",
  "telemetry_access_policy_secret_path": "/platform/grafana-cloud/stacks/REGION/USAGE/SLUG/telemetry-publisher"
}

The generated token comes from the connection Secret at reconciliation time; it is never embedded in rendered YAML.

Rotating telemetry token¶

When spec.telemetryAccess.enabled is true (the default), the Composition creates a stack-realm AccessPolicy with only:

stacks:read
metrics:write
logs:write
traces:write

An AccessPolicyRotatingToken uses the same 30-day lifetime and seven-day early rotation window. A separate PushSecret publishes the token and policy metadata under the stack's telemetry-publisher path. Workloads should use this token for telemetry and never receive the administrator token.

Static StackServiceAccountToken, AccessPolicyToken, and ServiceAccountToken resources remain available in the upstream provider but are deliberately not used here — their rotating counterparts avoid creating a permanent credential lifecycle outside the control plane.

SSO and incident profile secrets¶

deploy/aws/optional-profile-secrets.yaml provides the ExternalSecret shape for OAuth client secrets and the incident relay authorization value, each read from its own remote path under /platform/grafana-cloud/profiles/<profile-name>. These are intentionally excluded from deploy/aws/kustomization.yaml — apply this file only after the corresponding remote secrets and profile definitions exist, since an ExternalSecret referencing a missing remote value fails to sync. See SSO for how a request selects a profile.

AWS permissions¶

deploy/aws/iam-policy.json is the minimum policy shape for the example paths:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadVendingMachineInputsAndOutputs",
      "Effect": "Allow",
      "Action": ["secretsmanager:DescribeSecret", "secretsmanager:GetSecretValue"],
      "Resource": ["arn:aws:secretsmanager:*:*:secret:/platform/grafana-cloud/*"]
    },
    {
      "Sid": "CreateAndRotateVendingMachineOutputs",
      "Effect": "Allow",
      "Action": ["secretsmanager:CreateSecret", "secretsmanager:PutSecretValue", "secretsmanager:TagResource"],
      "Resource": ["arn:aws:secretsmanager:*:*:secret:/platform/grafana-cloud/stacks/*"]
    }
  ]
}

Attach it to the external-secrets ServiceAccount through the workload-identity mechanism for your cluster:

EKS Pod Identity — create a Pod Identity association for namespace external-secrets and ServiceAccount external-secrets.
IRSA — annotate that ServiceAccount with its role and use the standard EKS OIDC trust relationship.
Other Kubernetes platforms — use the cloud identity integration recommended for that platform.

Do not put long-lived AWS keys in the SecretStore. The controller should obtain short-lived credentials from workload identity. Restrict CreateSecret, PutSecretValue, and TagResource to the output prefix; restrict read access to the input and output paths actually required.

The example uses AWS Secrets Manager, not Systems Manager Parameter Store. The composition function itself only emits SecretStore references, so another ESO provider can be substituted if it supports ExternalSecret and PushSecret with the required structured-value behaviour.

PushSecret uses deletionPolicy: None. Removing a request does not delete its external credential documents — see Architecture → baseline and optional resources and the decommission runbook in the project README for the intentional non-destructive lifecycle this produces.

Extra composition RBAC¶

platform/rbac/composition-rbac.yaml grants the Crossplane composition RBAC manager an aggregated ClusterRole (crossplane-compose-grafana-vending-secrets) with full verbs on pushsecrets and externalsecrets (and their /status subresources). Crossplane needs this because the composition function emits PushSecret and ExternalSecret objects as part of a stack's composed resources, which is outside Crossplane's default RBAC surface.

Next steps¶

SSO — how a request selects a platform-owned identity profile.
Security — supply-chain verification and the non-destructive lifecycle.
Installation — where the SecretStore and organization ProviderConfig are applied during bootstrap.