Secrets¶

No Grafana credential belongs in Git, a request object, Composition input, status, or function log.

Organization credentials¶

For every registered organization, create a Grafana Cloud access policy token with only the organization-level capabilities needed to manage stacks and their Cloud resources. Store each in the external secret manager as JSON. Do not reuse one organization credential as a fallback for another.

{
  "cloud_access_policy_token": "REPLACE_SECURELY"
}

Use a separate credential path and Kubernetes Secret for each organization. Do not put the real token directly in a shell command, terminal history, CI variable dump, or Kubernetes manifest — use your secret-management workflow or a permission-restricted temporary file.

deploy/aws/secret-store-and-credentials.yaml wires this in three pieces:

A SecretStore (grafana-vending-secrets) pointing at AWS Secrets Manager.
An ExternalSecret per organization and request namespace that reads that organization's token and templates it into the JSON shape the provider expects, refreshed hourly.
A namespaced ProviderConfig per organization and request namespace that references the local generated Kubernetes Secret and is used only for that organization's Cloud operations. Copies retain the registry's ProviderConfig name because v2 managed resources resolve it locally.

Repeat the example's SecretStore too when it is namespaced. Namespace RBAC should prevent request authors from reading the generated Secret directly.

Rotating administrator token¶

For every stack, the Composition creates:

A StackServiceAccount with the Admin role.
A StackServiceAccountRotatingToken, created after Grafana reports the service-account ID — this is why a brand-new request needs at least two reconciliation passes before its credentials exist.
A Kubernetes connection Secret written by the rotating-token resource.
A PushSecret that exports a structured administrator document.
An ExternalSecret that reads the exported token and URL back into the stack namespace.
A stack-local ProviderConfig used for Grafana resources inside that stack.

The platform maximum lifetime caps the standard 30-day lifetime. The seven-day early rotation window is shortened when needed. Missing or invalid policy fails closed; provider-observed expiry appears in status.tokenExpiries. The PushSecret refresh interval is one hour, so a newly rotated token is copied to the external store well inside the overlap window.

The exported document has this shape:

{
  "stack_name": "Example stack",
  "stack_slug": "example",
  "stack_url": "https://example.grafana.net",
  "stack_region": "prod-us-central-0",
  "organization": "example-primary",
  "usage": "development",
  "change_reference": "CHANGE-EXAMPLE",
  "configuration_item_reference": "CONFIG-EXAMPLE",
  "stack_service_account_token": "GENERATED",
  "telemetry_access_policy_secret_path": "{outputSecretPrefix}/{organization}/{usage}/{slug}/telemetry-publisher"
}

The generated token comes from the connection Secret at reconciliation time; it is never embedded in rendered YAML.

Both administrator and telemetry documents use the identity path {outputSecretPrefix}/{organization}/{usage}/{slug}. spec.organization and spec.usage are immutable and must resolve in the platform-owned organization registry, so a request cannot silently move future documents to a new organization or usage path while orphaning documents at the old path.

Rotating telemetry token¶

When spec.telemetryAccess.enabled is true (the default), the Composition creates a stack-realm AccessPolicy with only:

stacks:read
metrics:write
logs:write
traces:write

An AccessPolicyRotatingToken uses the same 30-day lifetime and seven-day early rotation window. A separate PushSecret publishes the token and policy metadata under {outputSecretPrefix}/{organization}/{usage}/{slug}/telemetry-publisher. The immutable, platform-owned organization and usage segments keep this external identity stable. Workloads should use this token for telemetry and never receive the administrator token.

Static StackServiceAccountToken, AccessPolicyToken, and ServiceAccountToken resources remain available in the upstream provider but are deliberately not used here — their rotating counterparts avoid creating a permanent credential lifecycle outside the control plane.

SSO and incident profile secrets¶

deploy/aws/optional-profile-secrets.yaml provides the ExternalSecret shape for OAuth client secrets and the incident relay authorization value, each read from its own remote path under /platform/grafana-cloud/profiles/<profile-name>. These are intentionally excluded from deploy/aws/kustomization.yaml — apply this file only after the corresponding remote secrets and profile definitions exist, since an ExternalSecret referencing a missing remote value fails to sync. See SSO for how a request selects a profile.

AWS permissions¶

deploy/aws/iam-policy.json is the minimum policy shape for the example paths:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadVendingMachineInputsAndOutputs",
      "Effect": "Allow",
      "Action": ["secretsmanager:DescribeSecret", "secretsmanager:GetSecretValue"],
      "Resource": ["arn:aws:secretsmanager:*:*:secret:/platform/grafana-cloud/*"]
    },
    {
      "Sid": "CreateAndRotateVendingMachineOutputs",
      "Effect": "Allow",
      "Action": ["secretsmanager:CreateSecret", "secretsmanager:PutSecretValue", "secretsmanager:TagResource"],
      "Resource": ["arn:aws:secretsmanager:*:*:secret:/platform/grafana-cloud/stacks/*"]
    },
    {
      "Sid": "DeleteManagedVendingMachineOutputs",
      "Effect": "Allow",
      "Action": ["secretsmanager:DeleteSecret"],
      "Resource": ["arn:aws:secretsmanager:*:*:secret:/platform/grafana-cloud/stacks/*"],
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/grafana-cloud-vending-machine": "managed"
        }
      }
    }
  ]
}

Attach it to the external-secrets ServiceAccount through the workload-identity mechanism for your cluster:

EKS Pod Identity — create a Pod Identity association for namespace external-secrets and ServiceAccount external-secrets.
IRSA — annotate that ServiceAccount with its role and use the standard EKS OIDC trust relationship.
Other Kubernetes platforms — use the cloud identity integration recommended for that platform.

Do not put long-lived AWS keys in the SecretStore. The controller should obtain short-lived credentials from workload identity. Restrict CreateSecret, PutSecretValue, and TagResource to the output prefix; allow DeleteSecret only for output documents carrying the function's stable grafana-cloud-vending-machine: managed tag; restrict read access to the input and output paths actually required.

The example uses AWS Secrets Manager, not Systems Manager Parameter Store. The composition function itself only emits SecretStore references, so another ESO provider can be substituted if it supports ExternalSecret and PushSecret with the required structured-value behaviour.

PushSecret uses retain behaviour by default. Removing an unarmed request therefore does not delete its external credential documents. With an authorized spec.lifecycle.externalResources: Delete, the first reviewed stage only arms deletion; after status.deletionReady=true confirms the rotating tokens are deletion-managed and ESO has finalized and currently synced each enabled credential PushSecret. Stage 2 removes the dependent access claims and waits for their Kubernetes objects and finalizers to be gone while the Stack still exists. Stage 3 removes the request, after which ESO removes the administrator and telemetry documents. AWS Secrets Manager defaults to a 30-day recovery window for that deletion, and the supplied IAM policy includes tag-conditioned DeleteSecret on the output prefix. A platform operator using another backend must verify its PushSecret Delete support before authorizing this lifecycle. See Architecture → decommission and access-claim ordering and the decommission runbook.

Extra composition RBAC¶

platform/rbac/composition-rbac.yaml grants the Crossplane composition RBAC manager an aggregated ClusterRole (crossplane-compose-grafana-vending-secrets) with full verbs on pushsecrets and externalsecrets (and their /status subresources). Crossplane needs this because the composition function emits PushSecret and ExternalSecret objects as part of a stack's composed resources, which is outside Crossplane's default RBAC surface.

Next steps¶

SSO — how a request selects a platform-owned identity profile.
Security — supply-chain verification and the Retain-by-default lifecycle.
Installation — where the SecretStore and per-organization ProviderConfig are applied during bootstrap.

Product bootstrap and token-use network policy are described in Governance.

New Cloud product clients¶

PDC networks/tokens and Frontend Observability apps use the organization Cloud ProviderConfig, because a stack service-account token cannot authenticate those Cloud clients. Keep the organization CAP permissions limited to the selected products; the pinned PDC resource documentation requires accesspolicies:read, accesspolicies:write and accesspolicies:delete, and Frontend Observability requires frontend-observability:read, frontend-observability:write, frontend-observability:delete and stacks:read. No live authorization has been tested by this reference.

Cloud-provider integrations and metrics endpoint scrape jobs require integration-management:read, integration-management:write and stacks:read. The deployment ExternalSecret exposes its organization CAP under cloud_provider_access_token and connections_api_access_token as well as cloud_access_policy_token. The integration composite creates a dedicated namespaced ProviderConfig using the observed organization credential reference and the stack-details Secret. It supplies cloudProviderUrl explicitly from the identity-bound managed Stack observation; that endpoint is absent from the pinned provider's connection Secret. The Connections client uses its provider default endpoint. Its CloudIntegration children use the ordinary stack ProviderConfig. Credential values remain in Secrets; the function passes references only.

OnCall uses the stack service-account token through the pinned provider's documented auth fallback. The stack-details Secret supplies the OnCall URL. API admission and fixture observations do not prove the remote permission set or notification delivery.