Alerts¶
Start with Alert Profiles. The committed Grafana-managed set is the recommended profile: a high-signal default that leaves environment-specific rules paused. baseline is the smaller first-on-call set; strict enables nearly everything and needs a deliberate local render.
An alert rule evaluates a condition and opens an incident when it stays true for its configured window. A recording rule continuously writes a derived time series for dashboards or other alerts; it does not notify anyone by itself. In the Grafana-managed format, paused means the rule is kept as configured but is not evaluated. A profile is the named choice of which Grafana rules start enabled; it does not alter their expressions or notification labels.
Choose the delivery artifact that matches the rule engine:
deploy/alerts/grafana-managed/is the recommended Grafana-managed set. It spans Prometheus and Loki and preserves each rule'snoDataState,execErrState, andpausedstate.deploy/alerts/prometheus/tailscale2otel.rules.yamlis the supported committed artifact for Prometheus, Mimir, and Cortex rulers. It contains the Prometheus-backed alert and recording rules with normalized metric names andrunbook_urlannotations. Prometheus cannot represent Grafana's per-rule pause state and cannot run the omitted Loki rules.
Push the Grafana-managed directory with gcx:
Generated, not hand-written¶
The manifests are generated by gen/build_rules.py. Edit the generator, not the JSON, and regenerate with:
Every *.json in the output directory is deleted before writing, so a renamed or removed rule cannot linger as a stale file that keeps getting pushed. The Prometheus-compatible file is generated alongside it with just gen promrules. Both outputs are deterministic and checked for drift in CI.
Every rule follows the canonical Grafana 3-node pipeline (A query → B reduce → C threshold), expressed as the spec.expressions map keyed by refId with C marked "source": true. Datasource UIDs default to the portable Grafana Cloud defaults (grafanacloud-prom / grafanacloud-logs); swap them for a self-hosted stack.
Rules are organised into five families:
tailscale2otel-health- exporter self-health (scrape staleness, cardinality cap, API auth failures, checkpoint errors, enrichment cache age, and more)tailscale2otel-security- tailnet security and governance (tailnet-lock errors, key expiry, posture coverage, unverified contacts). The ACL risk-scoring gauges and the curatedtailscale.config.audit.changescounter are natural additions here - e.g.tailscale_acl_unrestricted_rules_ratio > 0(any-to-any non-deny rules),tailscale_acl_ssh_wildcard_ratio > 0(wildcard SSH rules), orincrease(tailscale_config_audit_changes_total{tailscale_audit_change="auth_provider"}[1h]) > 0.tailscale2otel-integrations- MDM/EDR posture sync, log-stream delivery health, and a paused accepted-event staleness rule for continuously active source/signal pairstailscale2otel-network- DERP relay usage, region latency, flow data presence and flow-log truncation, per-target node-metrics scrape health, and the webhook-only node IP-forwarding faulttailscale2otel-recording- precomputed recording rules (DERP byte fraction, posture ratios, total active series)
Format traps - three of these fail silently
The manifest format is not the apiVersion: 1 provisioning format, and the differences do not announce themselves. gen/validate_manifests.py enforces all of them offline.
| trap | the rule |
|---|---|
| state casing | BOTH noDataState and execErrState spell the OK state "Ok". The API accepts only ["Error", "Ok", "Alerting", "KeepLast"]. Corrected 2026-07-27 - this table previously claimed an asymmetry ("OK" for execErrState), and 19 rules passed every local check and were then rejected at push time. |
| durations | Go-style strings - "30m0s", "1h0m0s", "0s". "5m" is rejected, and relativeTimeRange bounds are durations here, not integer seconds. |
| panel links | the paired __dashboardUid__ / __panelId__ annotations, __panelId__ a string. Top-level dashboardUid/panelId are provisioning-only and additionalProperties: false rejects them. |
| recording rules | no annotations, for, condition, noDataState or execErrState. The spec is exactly {title, trigger, metric, expressions, targetDatasourceUID, labels, paused}. |
Limit-agnostic cardinality alerting
Prefer count(tailscale2otel_series_overflowing_ratio == 1) > 0 for a cardinality-overflow alert - it needs no hardcoded threshold and stays correct when cardinality.metric_limit is changed. tailscale2otel_scrape_budget_ratio (last scrape duration ÷ interval; nearing 1 = risk of interval overrun) is another tailscale2otel-health signal worth enabling.
Default-disabled by design
Only a high-signal starter set ships with spec.paused: false. The rest are spec.paused: true - enable them in the Grafana UI once your tailnet has the relevant data. Pausing is orthogonal to the evaluation policy below: a paused rule still carries its declared no-data and error semantics, it just is not evaluating yet.
Evaluation policy - absence and error are per-rule, not global
Every rule declares one of four policies, which fix its noDataState and execErrState together. Previously all of them were fail-open on error, so a broken datasource or a malformed query read as healthy across the entire pack.
| policy | noData | execErr | for |
|---|---|---|---|
coverage_critical | Alerting | Alerting | the rule exists to notice something stopped - absence IS the alert |
core | NoData | Error | always emitted by a running exporter, so absence is abnormal |
optional | Ok | Error | absence means "not configured", but an error is still a fault |
advisory | Ok | Ok | neither absence nor a transient error is actionable here |
ExporterDown is the only coverage_critical rule. Per-collector scrape rules are core rather than coverage_critical on purpose: total absence means the exporter is gone, which ExporterDown already pages on, and promoting them would fan one outage into a page per collector.
Nothing here can watch Grafana's own ruler. If the ruler stops evaluating, no rule fires, including the coverage_critical one. See Runbooks - who watches the watcher for the three operator-owned answers.
Every alert links to a runbook and a panel
All 111 alert rules carry a runbook_url pointing at a section of Runbooks, and 107 of them carry the paired __dashboardUid__/__panelId__ annotations naming a canonical panel. Panel ids are resolved by title against the generated dashboard at build time, and generation hard-fails on a title that matches zero or more than one panel - so a renumbered or renamed panel breaks the build instead of silently producing a dead link.
Alert on accepted data, not only a running receiver
AcceptedIngestDataStale compares time() with tailscale2otel_ingest_last_event_timestamp_seconds. It catches a receiver or poller that remains healthy while delivering old events. The rule ships paused because audit and webhook streams can be legitimately quiet. Enable it only for source/signal pairs expected to be continuous, add label filters, and tune the one-hour threshold to that workload. The paused tailscale2otel:ingest_event_freshness_seconds recording rule exposes the same calculation for custom alerts.
Deploying¶
# push the whole directory (the folder manifest sorts first, so it lands first)
gcx resources push -p deploy/alerts/grafana-managed
Anything that speaks the Grafana app-platform API works too: kubectl against the Grafana API server, or the Terraform grafana_apps_rules_alertrule_v0alpha1 / grafana_apps_rules_recordingrule_v0alpha1 resources, which take the same spec.
gcx resources validate -p deploy/alerts/grafana-managed is a client-side check only - alertrules.rules.alerting.grafana.app does not support server-side dry-run, so it confirms the manifests parse and that the kind is served, nothing more. The spec check is python3 deploy/alerts/gen/validate_manifests.py, which runs offline and in CI.
Metric naming in rule expressions¶
All expressions query the normalized Prometheus names produced by the OTLP-to-Prometheus translation, not raw OTEL names. Dots become underscores, monotonic counters gain _total, units gain their suffixes (s → _seconds, By → _bytes, d → _days), and a gauge with unit "1" becomes *_ratio. See Metrics for the full translation table.
Non-Grafana backends
If you send metrics to a non-Grafana OTEL backend with different normalization rules, you will need to adjust the metric names in the rule expressions accordingly.
Wiring notifications¶
Every rule sets a severity label (critical / warning / info), plus service and domain. Wire those into your Grafana notification policy to fan alerts to the right contact points.
Prometheus-compatible deployment and local execution¶
Load the committed Prometheus-compatible file into your Prometheus, Mimir, or Cortex ruler using that ruler's normal rule-file configuration. It is generated from the same catalogue as the Grafana-managed set, includes recording rules and runbook_url annotations, and deliberately omits Loki-backed rules. Because Prometheus has no per-rule paused field, every included rule evaluates once loaded.
promtool test rules is the only local check that runs rules rather than only parsing them:
just gen promrules
promtool check rules deploy/alerts/prometheus/tailscale2otel.rules.yaml
promtool test rules deploy/alerts/tests/*.yaml
The committed file renders coverage_critical's "absence is the alert" semantics as an explicit or absent(...) arm, which Grafana expresses as noDataState: Alerting.
The offline contract tests cover the rest: