Grafana annotations¶

graph2otel can publish a curated, closed set of Microsoft domain events into Grafana as annotations, so a dashboard explains "what changed at 14:00" without any external automation. It is opt-in and off by default.

This is graph2otel's one non-OTLP egress path¶

graph2otel is OTLP-push-only: no Prometheus endpoint, no generic output framework, no Event Hub, no Log Analytics pipeline. That invariant was narrowly amended for annotations and nothing else. The annotation writer speaks exactly one call — POST /api/annotations — to exactly one destination, and metrics and logs do not gain a second path. The narrowness is carried by the service-account token's scope; see The required Grafana permission.

Nothing here polls Microsoft Graph. Annotations are derived from records the existing collectors already emit, observed at the telemetry emitter boundary. There is no Graph client in internal/annotations, and enabling the feature adds no new endpoint and no new Graph scope.

Enabling it¶

Setting grafana_annotations.url is the whole opt-in. Unset (the default) registers no writer, opens no client, and logs nothing.

grafana_annotations:
  url: "https://grafana.example.com"

The token is a credential, so it comes from the environment or a mounted file, never committed YAML:

export G2O_GRAFANA_ANNOTATIONS__TOKEN='glsa-...'
# or, for a Kubernetes/Docker secret mount:
#   grafana_annotations.token_file: /run/secrets/grafana-annotation-token

Every key and its default is in Environment Variables under G2O_GRAFANA_ANNOTATIONS__*.

The required Grafana permission¶

graph2otel needs exactly one action/scope pair:

No dashboard read, no alert-rule access, no folder management — graph2otel requires, requests and uses no other Grafana permission.

The documented minimum is a custom role granting only annotations:create. Create it and assign it to the service account:

# Create the role
curl -s -X POST "$GRAFANA_URL/api/access-control/roles" \
  -H "Authorization: Bearer $GRAFANA_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{
    "version": 1,
    "name": "custom:graph2otel:annotations:creator",
    "displayName": "graph2otel annotation creator",
    "description": "Write-only organization annotations for graph2otel",
    "global": false,
    "permissions": [
      {"action": "annotations:create", "scope": "annotations:type:organization"}
    ]
  }'

# Assign it to the service account
curl -s -X POST "$GRAFANA_URL/api/access-control/users/$SERVICE_ACCOUNT_ID/roles" \
  -H "Authorization: Bearer $GRAFANA_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"roleUid": "<uid returned by the create call>"}'

The easy option is the built-in fixed role, fixed:annotations:writer. It works, but it is broader than graph2otel needs: it grants three actions on annotations:type:organization — annotations:create, annotations:write, annotations:delete — and graph2otel only ever issues POST /api/annotations, so write and delete are granted and never used. Reach for it when standing up a custom role isn't worth the extra step; reach for the custom role above when the token needs to be as narrow as the feature.

Live-measured (2026-07-27, #400) against https://grafana.m7kni.com. A service account holding only annotations:create on annotations:type:organization wrote an annotation successfully (POST /api/annotations → HTTP 200, {"id":716,"message":"Annotation added"}). Grafana's own effective-permissions report for that token was exactly {"annotations:create":["annotations:type:organization"],"folders:read":["folders:uid:sharedwithme"]} — the folders:read on sharedwithme is Grafana's own unavoidable baseline, not something graph2otel asks for or uses. Narrowness was also proven live with the same token: GET /api/annotations → 403 "Permissions needed: annotations:read"; DELETE /api/annotations/716 → 403 "Permissions needed: annotations:delete"; POST /api/folders → 403 "Permissions needed: folders:create"; GET /api/search → 200 with an empty list, because Grafana filters search results by permission and the token can enumerate nothing. fixed:annotations:writer's three-action grant, and fixed:annotations.all:writer's further grant of all three actions on folders:*, were also measured directly. The live deployment runs on the custom role, custom:graph2otel:annotations:creator, created with "global": false.

One line above is not covered by that measurement: the annotations:type:dashboard scope for a dashboard_uid deployment is still docs-only. Nothing here was tested with dashboard_uid set.

It fails fast, on purpose¶

If url is set and the token cannot actually write an annotation, the process refuses to start and names the missing permission. The startup check is not a synthetic probe — it is the real process-start annotation (see Lifecycle marker), so proving the permission leaves no junk behind and needs no annotations:delete.

Live-measured (2026-07-27, #400): a token with no annotation permission gets POST /api/annotations → 403 with body {"message":"You'll need additional permissions to perform this action. Permissions needed: annotations:create","title":"Access denied"}, and the real graph2otel binary refuses to start on it — non-zero exit, level=ERROR msg="refusing to start", echoing Grafana's body plus the actionable permission line.

Discovering a dead token at the first real event would mean the annotations an operator is relying on for incident context are simply absent at the moment they look, with nothing in the process having said so.

The tag contract¶

Every annotation carries a small, closed, low-cardinality tag set. graph2otel is the selector a dashboard annotation query should filter on; the rest narrow it.

Tags are deliberately free of entity ids and hashes: Grafana indexes annotation tags, so a per-entity tag would grow the tag store without ever being queried.

The curated event set¶

Four categories. Each is independently gated by grafana_annotations.categories.<name>.enabled, and each can be switched between per-event and per-interval publishing with .rollup.

config_posture — "what changed at 14:00"¶

Rolled up by default: it is the highest-volume of the four on an active tenant, and its rate scales with administrative activity.

security_incident — timeline context, not a page¶

Individually annotated: naturally low volume, and a rolled-up count would lose the one thing an operator needs — which incident.

low and informational are deliberately excluded: a healthy tenant produces a steady trickle of them, and a picket fence of low-severity markers is what stops operators reading annotations at all.

service_health — the failure that is not graph2otel's fault¶

Individually annotated. This is the single most useful annotation for an operator triaging a red dashboard, because it explains collector degradation graph2otel did not cause.

license — why a collector says limited¶

Rolled up by default: a tenant-wide license change moves many SKUs at once.

This category is derived from the entra.license.consumed and entra.license.enabled gauge snapshots, not from a log record — "the SKU set changed" and "a SKU is exhausted" are comparisons between two snapshots, so they are properties of neither snapshot alone.

Exhaustion fires on the transition only. A SKU sitting at its ceiling for a month is one event, not one per poll.

Lifecycle marker¶

category:lifecycle, rule graph2otel.startup: one annotation per configured tenant per process start, carrying the build version and a one-way, secret-free configuration fingerprint — the same config.fingerprint the graph2otel.startup log record carries. It has no category gate, matching that log record.

"Configuration changed" is a comparison, not a field: two consecutive markers whose fingerprints differ is a configuration change, and two whose versions differ is an upgrade. The marker itself never claims something changed.

Volume control¶

Enough annotations and every dashboard becomes an unreadable picket fence, at which point operators stop reading them and the feature is worse than absent. Four mechanisms bound it:

• Curation. The rule set is closed, and each rule has a predicate — a directory audit that is not a conditional-access change is not annotated at all.
• Rollup. A rolled-up category publishes one region annotation per rollup_interval per tenant, carrying a count and a bounded summary of at most five distinct titles plus a +N more.
• Dedupe. Every annotation has a stable key derived from (tenant, rule, source identity) and nothing else — no clock, no counter. A re-delivery from an overlapping poll window, or after a restart, derives the same key and is suppressed. The key set is persisted in the checkpoint directory and evicted after dedupe_retention (48h default).
• Rate limit. max_per_minute (60 default) is a hard ceiling on what reaches Grafana. Overage is dropped and counted; it is never delayed, because a marker that arrives long after the moment it explains is worse than absent.

State-shaped sources re-emit their whole current set every tick, so their first observed set primes silently: publishing it would annotate every consent grant and every app credential that already existed, at once. The residual cost is that a first-ever occurrence landing in the very first tick after a restart is not annotated.

Failure isolation¶

Nothing about annotation publishing can block or fail collection.

• Enqueueing never blocks. A full queue drops and counts.
• A Grafana outage, an expired token or a 429 is counted, logged at WARN, and otherwise invisible to the poller.
• The only fatal failure is at startup, deliberately.

Three self-observability metrics report it, in the same shape as OTLP delivery health:

reason is a closed set: duplicate, queue_full, local_rate_limited, unauthorized, rate_limited, rejected, server_error, transport.

reason="duplicate" is the steady state, not a fault. A state-shaped source re-emits its whole set every tick and every already-published identity lands there. Alert on graph2otel_annotation_degraded and on reason="unauthorized", never on the duplicate count.

graph2otel_annotation_degraded carries no tenant_id, and that is truthful rather than an omission: one Grafana endpoint and one token serve the whole process, so a per-tenant series would claim a tenant-specific fault where none exists. It is the same scope as graph2otel_otlp_delivery_degraded.

Content safety¶

No secret and no raw effective configuration ever reaches annotation text.

Annotation text is built from a per-rule allow-list of attribute keys, never from the whole attribute map, so a field added to a source record later cannot silently ride out to Grafana. The lifecycle marker carries the version and the one-way fingerprint, never the configuration.

The cardinality rule is unaffected: annotations are neither metrics nor logs, and the tag set carries no per-entity value.

Persistence¶

The dedupe key set lives in checkpoint_dir alongside the collectors' watermarks, under grafana-annotations per tenant. It needs the same persistent volume: without one, every restart republishes everything inside the source collectors' overlap windows. The Helm chart defaults to an emptyDir, so production installs must set persistence.enabled=true.