Troubleshooting¶

This page is a symptom-first index, not a second copy of the detail. Two pages already cover most of the ground:

• Graph API gotchas — the live-verified quirk ledger for specific Microsoft Graph and O365 endpoint behavior (URL encoding, page-size ceilings, throttling shapes, per-endpoint traps).
• Alert runbooks — one section per shipped alert rule, each covering the fired / no-data / errored / false-positive states for that exact signal.

If nothing here matches, check the two pages above before opening an issue on GitHub.

Setup and permissions¶

graph2otel check passes but a collector still 403s¶

graph2otel check only confirms a Graph permission is granted and admin-consented — it cannot detect every prerequisite. Two known gaps:

• A directory-role gate. Identity Protection surfaces (entra.risk, entra.risk_detections) have been observed in Microsoft's docs to additionally expect a directory role (e.g. Security Reader) on the service principal, beyond the API permission grant. See Permissions.
• A revoked consent grant on a collector that used to work. Read cause= on the WARN collector completed with degraded outcome log line, or split graph2otel_scrape_outcomes_total by result. permission_denied on a previously-working endpoint means re-consent the app role named by the collector's RequiredPermissions(). See runbooks: g2o-collector-degraded-sustained.

A collector logs 401 even though the scope is already in the token¶

A 401 with the scope present, on Purview eDiscovery specifically, means the service principal is not registered with the Security & Compliance data plane — a Graph scope alone cannot fix it, no matter how it's re-granted. See Non-Graph data-plane registration.

Admin consent shows granted in the Entra admin center but calls still 403¶

Adding an application permission is not the same as consenting to it. A tenant administrator must separately click Grant admin consent — until then every call using that permission returns 403 regardless of what the admin center's permissions list shows. See Permissions.

mdca.discovery_parse has no Graph scope to grant¶

That's expected — it's the one collector that authenticates with a static portal token against the legacy MDCA API, not DefaultAzureCredential or a Graph token. graph2otel check reports it as a manual prerequisite it cannot verify. See Permissions.

Startup and configuration¶

The process exits immediately with a config validation error¶

Configuration mistakes stop the process before it constructs credentials, checkpoint stores, or collectors. The error names the exact path — including sequence indexes and collector map keys for YAML, or the exact G2O_* variable name for environment overrides. A near-miss collector name gets a did you mean "..." suggestion. See Configuration.

Startup fails because checkpoint_dir isn't writable¶

This is fatal by design — silently restarting from the initial lookback instead of the real watermark would create an unbounded replay. In Kubernetes this is almost always the Helm chart's default emptyDir; set persistence.enabled=true, since an emptyDir also silently loses every watermark on pod replacement. See Architecture and runbooks: g2o-checkpoint-persist-errors.

The process won't start because the admin address can't bind¶

A configured admin.addr that cannot bind is a fatal startup error — graph2otel never runs silently without its configured health surface. Check for a port conflict or an address the container/pod isn't allowed to bind. See Configuration.

Missing or empty data¶

A collector shows healthy_empty on the availability gauge¶

healthy_empty means the source answered successfully with zero rows — it proves the collector's source responded, not that graph2otel observed data. On a small tenant this is routine: the default m365.activity content types (Audit.Exchange + Audit.SharePoint) commonly carry zero content and the collector looks 100% healthy doing it. See Signals for the full state/reason contract.

An experimental or beta collector emits nothing¶

Beta-only collectors never register on the implicit "unset means enabled" default — they need an explicit enabled: true at some config layer before they run at all. Setting enabled: false, or leaving the collector unmentioned, both mean "not explicitly enabled." See Configuration.

graph2otel.collector.availability shows disabled or covered¶

Both are intentional absence, not a failure — disabled means the collector isn't switched on for that tenant, covered means a logical signal is being served by a different registered candidate. Look for degraded, failed, or startup_failed instead when hunting for a real problem. See Signals.

Blob-ingest collectors show no data¶

Check three things in order: the diagnostic setting actually points at the storage account you configured; Storage Blob Data Reader (a data-plane role — Owner alone is not sufficient and fails silently, listing blobs but 403ing only on content reads) is granted to graph2otel's app registration; and the container has actually received its first write — Azure creates insights-logs-<category> containers on first write, not at setting-creation time, so a missing container can just mean "nothing written yet." See Blob ingest.

A collector's Graph call returns 200 but the data looks wrong or truncated¶

Several Graph endpoints answer with rows that look complete but aren't — a bare Endpoint Analytics list can silently serve one device out of many, $count=true can drop the rows it's counting, and other per-endpoint quirks are cataloged with their exact wire evidence. Before concluding a tenant has no data, vary the request shape (bare vs $select vs $top vs single-entity) — see Graph API gotchas.

Throttling and delivery¶

Collectors are slow to update or fall behind their interval¶

Check g2o-throttle-saturation and g2o-collector-staleness first — none of the throttled Graph workloads reliably send Retry-After, so throttling degrades freshness silently rather than erroring loudly. The alert's workload label names the ceiling being hit; Identity Protection's 1 req/s limit is shared tenant-wide across every application, so another app in the tenant can throttle graph2otel. See runbooks: g2o-throttle-saturation and runbooks: g2o-collector-staleness.

Metrics or logs aren't reaching the backend¶

Grep the process's stderr for the otel sdk error line, which quotes the backend's rejection reason verbatim — a size limit, a timestamp-horizon rejection, or a credential problem read very differently and need different fixes. See runbooks: g2o-otlp-delivery-failing.

Records are being dropped as too old for the backend¶

Grafana Cloud's OTLP gateway rejects entries older than 7 days per-entry; graph2otel clamps its own poll to 165h to stay inside that window. A blob-derived stream can push records past the backend's window through ordinary aging — this is expected there, not a misconfiguration, and the alert for it ships paused for exactly that reason. See Configuration and runbooks: g2o-record-over-horizon.

A log record was truncated instead of dropped¶

This is content loss, not record loss — the record landed, but its largest attribute values were shortened to fit the backend's structured-metadata size limit. Query for attrs_truncated = "true" to find which records and which fields. See runbooks: g2o-record-attrs-truncated.

Querying logs and getting nothing back¶

A LogQL query on an attribute returns zero rows¶

graph2otel's log attributes (event_name, app_id, user_principal_name, …) are Loki structured metadata, not stream labels. A stream selector on one of them — {event_name="entra.signin"} — matches nothing and fails silently. Filter with a | label-filter after the {service_name="graph2otel"} stream selector instead. A line filter (|=) also cannot see structured metadata, so an empty line-filter result is not evidence a value is absent. See Signals.