Skip to content

Architecture

Components

A single manager binary (cmd/main.go) runs everything: one controller-runtime manager hosting a reconciler per enabled CRD, a validating admission webhook server, a metrics endpoint, and health probes. There is no separate agent or sidecar — the manager Pod is the whole operator.

Kubernetes API server
   │  watch / list / status update
   ▼
manager Pod
 ├─ Pipeline reconciler                ─┐
 ├─ PipelineDiscovery reconciler        │  each has its own admission
 ├─ Collector reconciler                │  webhook, registered only if
 ├─ CollectorDiscovery reconciler       │  the matching controller flag
 ├─ RemoteAttributePolicy reconciler    │  is enabled
 ├─ ExternalAttributeSync reconciler    │
 └─ TenantPolicy reconciler (status)   ─┘
        │ rate-limited HTTP client (pkg/fleetclient)
        ▼
 Grafana Cloud Fleet Management API (pipeline.v1.PipelineService)

Controllers (internal/controller/)

Each CRD except TenantPolicy has a dedicated file: pipeline_controller.go, pipeline_discovery_controller.go, collector_controller.go, collector_discovery_controller.go, policy_controller.go (RemoteAttributePolicy), external_sync_controller.go, and tenant_policy_controller.go. All of them are watch-driven — SyncPeriod is deliberately left unset so there is no full-resync storm on an interval; a reconcile fires only on a CRD change or a status-driven RequeueAfter.

Reconcile pattern, common to the Fleet-backed controllers (Pipeline, Collector, RemoteAttributePolicy, ExternalAttributeSync, the two discovery controllers):

  1. Fetch the CR from the informer cache (a single Get, no List in the hot path for per-resource controllers).
  2. Skip if status.observedGeneration already matches metadata.generation — the spec has not changed since the last successful sync.
  3. Add a finalizer before the first Fleet API call, so a delete always has a chance to clean up the remote side.
  4. Call the Fleet Management API (UpsertPipeline, BulkUpdateCollectors, ListCollectors, the external source's Fetch, and so on) through the shared rate-limited client.
  5. Write status (ID, sync timestamps, counts) via Status().Update() — never the plain Update(), which would race with spec changes — and set Ready / Synced conditions via meta.SetStatusCondition.
  6. Emit a Kubernetes event summarising the outcome. See events.md for the full reason catalogue and conditions.md for condition types/reasons.
  7. On delete: call the matching Fleet delete API, treat 404 as success, then remove the finalizer.

Pipeline and Collector are kept at MaxConcurrentReconciles=1 because they share the same Fleet API rate budget — running them concurrently only queues more requests at the limiter without raising throughput. RemoteAttributePolicy, ExternalAttributeSync, and the discovery controllers are safe to parallelise (pure cache reads, or one Fetch/List call per source) and default to higher concurrency; see flags.md for the exact defaults and the --controller-*-max-concurrent flags that override them.

TenantPolicy's own reconciler does not call Fleet Management at all — it only re-validates the spec (matcher syntax, namespace selector) and mirrors the result into status.

Fleet API client (pkg/fleetclient/)

A single shared client wraps every outbound call in:

  • A token-bucket rate limiter (golang.org/x/time/rate), default 3 requests/second with a burst of 50 — see flags.md for --fleet-api-rps / --fleet-api-burst.
  • Prometheus instrumentation (fleet_api_requests_total, fleet_api_errors_total, fleet_api_request_duration_seconds, fleet_api_rate_limiter_wait_duration_seconds) — see metrics.md.
  • Optional OpenTelemetry trace spans, wired through when OTEL_EXPORTER_OTLP_ENDPOINT is set in the manager's environment. Tracing is a no-op exporter by default; setting that environment variable switches to a real batched OTLP-over-gRPC exporter configured from the standard OTEL_EXPORTER_OTLP_* variables, with a bounded 5s flush on shutdown.

Authentication is HTTP Basic auth against the Stack ID and API token supplied via the fleetManagement.* Helm values or an existingSecret.

External sources (pkg/sources/)

ExternalAttributeSync delegates to a pluggable Source interface with two implementations selected by spec.source.kind: pkg/sources/http (bearer or basic auth, JSON response with an optional dotted recordsPath) and pkg/sources/sql (a single read-only SELECT against Postgres or MySQL, DML/DDL keyword-denylisted). pkg/netguard backs the HTTP source's SSRF defences — an admission-time denylist plus a dial-time re-check of the resolved IP — described in Security Model and Hardening.

Admission webhooks (api/v1alpha1/*_webhook.go)

Every CRD has a validating webhook (no mutating webhooks), registered only for the CRDs whose controller is enabled — the discovery and attribute controllers ship their webhook alongside their reconciler. Webhooks enforce what CRD schema validation cannot express as OpenAPI structural rules: matcher syntax, configType agreeing with contents, reserved collector.-prefixed attribute keys, namespace-name syntax on spec.targetNamespace, and, when the relevant manager flags are set, TenantPolicy matcher requirements and the cross-namespace discovery SubjectAccessReview check. All webhooks are registered with failurePolicy: Fail, so a webhook outage blocks new creates/updates rather than silently admitting unvalidated specs — see Webhook TLS Setup for how the serving certificate is provisioned.

Multi-tenancy layer (internal/tenant/)

Backs the TenantPolicy CRD's admission-time checks: reading the admission request's UserInfo, matching it against TenantPolicy.spec.subjects, and unioning requiredMatchers across every policy that matches. See Tenant Policy for the full mechanism, coverage, and known v1 gaps.

Where to go next

  • flags.md — every manager flag and its Helm value.
  • metrics.md — the Prometheus metrics this reconcile loop and client emit.
  • security.md — the privilege model this architecture implies.
  • versioning.md — how this API group evolves from v1alpha1.