Troubleshooting¶
Request never becomes Ready¶
Cause. GrafanaCloudStackRequest becomes Ready only when every currently desired composed resource reports Ready. Rotating-token resources (StackServiceAccountRotatingToken, AccessPolicyRotatingToken) are intentionally rendered one reconciliation after Grafana reports the parent service-account or access-policy ID — a brand-new request is expected to take at least two reconciliation passes.
Diagnosis.
kubectl describe grafanacloudstackrequest -n grafana-vending <slug>
kubectl get managed -n grafana-vending
Look for any composed resource stuck Synced=False — its condition message names the upstream API error.
Two controllers fighting over the same resource¶
Cause. Argo CD's default label-based resource tracking claims Crossplane-generated children as its own, because it inherits ownership by label rather than annotation. This produces a loop where Argo CD and Crossplane each try to reconcile the same object toward different desired state.
Fix. Set application.resourceTrackingMethod: annotation in the Argo CD argocd-cm ConfigMap, as shipped in deploy/argocd/argocd-values.yaml. See Installation → Argo CD installation.
Request stuck Progressing in the Argo CD UI¶
Cause. The default Argo CD health check has no opinion on Crossplane Composition, CompositeResourceDefinition, ProviderConfig, or the platform.example.org/* custom resources, so it may report a state that does not match actual readiness.
Fix. Apply the resource.customizations health Lua scripts from deploy/argocd/argocd-values.yaml. They teach Argo CD to read Crossplane's own Synced/Ready conditions and treat a request as Progressing until its composed resources are ready, Healthy once they are, and Degraded on a Synced=False condition.
Rendered request has the wrong API group¶
Cause. Every XRD, Composition, and example in this repository ships with the placeholder API group platform.example.org. If a fork has not replaced it consistently, some manifests apply against the placeholder group while others reference the real one, and the composite resource never matches its Composition.
Fix. Replace platform.example.org in every XRD spec.group, every Composition spec.compositeTypeRef.apiVersion, the stack Composition input apiVersion, the examples, and your own request manifests. See Configuration for the complete list of places it appears, and Getting started for the warning at the top of that flow.
PushSecret fails to create an AWS Secrets Manager target¶
Cause. ESO 2.7.0 and 2.8.0 carry an open regression (external-secrets/external-secrets#6593) that sends an empty replica-region request when creating a new AWS Secrets Manager PushSecret target, which AWS rejects.
Fix. Stay on ESO Helm chart 2.6.0 until a fixed release exists, and prove creation of a brand-new remote secret against the fixed release before removing the pin. Do not add a replica region merely to work around the symptom — see the pinned-versions note in Getting started.
SSO profile Secret never syncs¶
Cause. deploy/aws/optional-profile-secrets.yaml is deliberately excluded from deploy/aws/kustomization.yaml. Applying it before the corresponding remote secret and profile exist leaves the ExternalSecret unable to resolve its remoteRef, and it never syncs.
Fix. Confirm the remote secret exists at the profile's path (/platform/grafana-cloud/profiles/<profile-name>) before applying optional-profile-secrets.yaml. See Secrets → SSO and incident profile secrets.
An administrator's SSO change keeps getting reverted¶
Cause. spec.sso.mode: enforced makes Crossplane the sole owner of OAuth/SAML settings — any UI edit is treated as drift and repaired on the next reconciliation.
Fix. This is expected behaviour for enforced mode. If stack administrators should own later SSO edits, move the request to createOnly — see Architecture → reconciliation and out-of-band changes for the complete mode table, and SSO for the handoff semantics.
A deleted request left the stack and tokens behind¶
Cause. This is intentional, not a bug. Pruning a GrafanaCloudStackRequest deletes the Kubernetes composite and composed objects, but the Stack has delete protection enabled, no composed resource carries the Delete management policy, rotating tokens set deleteOnDestroy: false, and PushSecret uses deletionPolicy: None. External Grafana and secret-manager objects are orphaned, not destroyed.
Fix. There is no fix — this is the designed non-destructive lifecycle. To actually destroy a stack, follow the decommission runbook in the project README, which is a separately reviewed operation, not something Git deletion alone triggers. See Security → non-destructive lifecycle.
Running the validation gate¶
Run the complete local gate before opening a change:
It performs public-release scanning, Go formatting and module consistency checks, race-enabled unit tests with coverage, go vet, YAML syntax parsing, and Kustomize rendering for the platform, AWS examples, and comprehensive catalog base. See Security → public-release scanning for what the scan itself covers.
Next steps¶
- Architecture — the full reconciliation-mode table.
- Secrets — the credential flow this page's secret-related problems reference.
- FAQ — shorter, less diagnostic questions.