Alert runbooks¶
One section per shipped rule. Every rule under alerts/rules/ — health alerts and paused detections alike — carries a runbook_url annotation pointing at its section here, so the link in a notification lands on the guidance for that exact rule. Paused rules have runbooks too: a paused rule is the one you are about to enable, and this page is where you find out whether enabling it is safe.
Each section answers the four states a responder actually meets — it fired, it went to no data, it errored, and it was wrong.
How the links work¶
Every rule also carries __dashboardUid__ + __panelId__, so Grafana renders a View panel link straight to the panel that shows the same signal, plus a dashboard_path annotation holding the same target as a deep link relative to your Grafana host (/d/graph2otel?dtab=<Tab>&viewPanel=<id>). Both are generated from dashboards/graph2otel.json, so they cannot name a panel or tab that does not exist. Import that dashboard before the rules if you want the links live.
runbook_url and the panel annotations are generated by grafana/build_rules.py. Renaming a section heading here breaks make grafana-check rather than shipping a dead link.
Rule-wide behaviour¶
Evaluator errors are never silent: every rule ships execErrState: Error, so a datasource outage, a bad query or an expression failure surfaces as an Error state rather than as a healthy rule. An erroring rule is not evidence of a healthy tenant, and it is not evidence of an unhealthy one either — treat it as "this rule told you nothing this interval" and check the datasource first.
No data is configured per rule, because absence means different things for different signals. Most rules use OK: an empty result is the healthy steady state for anything counting failures. g2o-collector-staleness is the one exception and uses Alerting.
Thresholds are starting points. Where a number came from one tenant, the section says so. Tune it against your own measurement rather than treating it as a recommendation.
graph2otel health alerts¶
g2o-entra-cred-expiry-critical¶
An Entra application or service-principal credential (client secret or certificate) lands in the lt_7d or expired bucket. When one of these expires, sign-in or an integration breaks with no warning.
No data: OK. No soon-expiring credentials is the healthy state. It also means the entra.credential_expiry collector may be off — check Self-obs → Collector availability before concluding your inventory is clean.
Evaluator error: Error. The metric is a bucketed count, so an error here is a datasource or expression problem, never a tenant problem.
False positives: a credential already scheduled for rotation is expiring on purpose, and the count cannot tell that apart from a forgotten one. Tenants with deliberately short-lived credentials sit in the bucket permanently.
Remediation: find the owning app from Entra → Applications, service principals and credentials, then the entity itself in the entra.credential_expiry log twin (owner_type, credential_type, the owning object). Rotate the credential, or move it into an automated rotation. If the alert is chronic because rotation is automated, suppress by owner_type rather than raising the threshold — you still want the unautomated ones.
g2o-entra-cred-expiry-warning¶
The earlier tier of the rule above, on the lt_30d bucket. Paused by default.
No data: OK, same reasoning as the critical tier.
Evaluator error: Error.
False positives: more of them than the critical tier by construction. Any credential renewed on a 30-day-or-shorter cycle keeps this bucket non-zero forever, which is why it ships paused.
Remediation: before enabling, decide your rotation lead time. If it is under 30 days, this rule will be permanently firing and should stay paused; the lt_7d critical tier is then your only credential alert. Otherwise enable it and treat it as a work queue, not a page.
g2o-intune-apple-token-expiry-critical¶
An Apple MDM token (APNS, VPP or DEP) has fewer than 14 days left. Paused by default. Unlike the bucketed credential metrics this is a raw days-remaining gauge over a small admin-configured token set, so the threshold is an exact day count you can tune freely.
No data: OK. A tenant with no Apple MDM emits no series at all, which is the normal case for a Windows-only fleet.
Evaluator error: Error.
False positives: rare — the gauge is a real countdown. The usual surprise is the opposite: the token set is per token_name, so a decommissioned token that Intune still lists keeps counting down and firing.
Remediation: APNS certificates renew in the Apple Push Certificates Portal and must be renewed with the same Apple ID that created them, or every enrolled iOS/macOS device has to re-enrol. VPP and DEP tokens renew in Apple Business/School Manager. Confirm the new expiry on Intune → Tokens, certificates and connectors.
g2o-intune-cert-expiry-critical¶
Intune-managed certificates in a profile are in the 0d_7d or expired bucket. Paused by default; needs the beta intune.certificates collector.
No data: OK. The collector is opt-in, so no series is the default state.
Evaluator error: Error.
False positives: short-lived certificates by design (a SCEP profile issuing weekly certificates) sit in the urgent bucket permanently. The bucket ladders differ between the two expiry collectors and this catches people out: Intune certificates use 0d_7d / 7d_30d / 30d_90d / over_90d / unknown, Entra credentials use lt_7d / lt_30d / lt_90d / gt_90d / expired.
Remediation: identify the profile from cert_profile_name on Intune → Tokens, certificates and connectors, then check the issuing authority — an expiring leaf set usually means the SCEP/PKCS connector or the CA itself stopped issuing, not that thousands of devices independently expired. g2o-collector-staleness on the connector collector is the corroborating signal.
g2o-intune-compliance-ratio-low¶
The compliant fraction of the Intune fleet is under 90% for 30 minutes. The and >= 5 fleet-size guard stops a two-device pilot tenant paging on one device.
No data: OK. No Intune compliance policies, or the intune.compliance collector disabled, means no series.
Evaluator error: Error. This rule is a ratio of two sums, so a partial datasource failure can error rather than produce a wrong number, which is the intended behaviour.
False positives: a compliance re-evaluation window right after a policy rollout drops devices out of compliant transiently. Devices that have not checked in are counted as not compliant, so a fleet that is largely offline (a holiday, a shift pattern) looks non-compliant without anything being wrong.
Remediation: open Intune → Compliance and baselines and split by state and policy. If the drop is one policy, it is a policy change; if it is spread evenly, it is a check-in problem — cross-check intune_devices_sync_staleness_seconds on Managed devices and inventory. The per-device detail is in the intune.device log twin. Widen for if your tenant pushes policy frequently; raise the >= 5 guard to match your smallest real tenant.
g2o-intune-compliance-noncompliant-spike¶
The non-compliant share rose more than 10 percentage points within an hour. Companion to the ratio rule, for a large compliant fleet where the absolute ratio takes a long time to cross 90%. Paused by default: the 1h/10pp thresholds have not been validated against real fleet churn.
No data: OK.
Evaluator error: Error.
False positives: likely, and the reason it is paused. Any bulk enrolment, policy assignment change or maintenance window produces a step change that looks identical to a regression.
Remediation: before enabling, run the expression over a month of your own fleet history and pick a swing size your normal churn does not reach. Once firing, treat it exactly like the ratio rule above — the difference is only in how early it fires.
g2o-collector-staleness¶
A collector has gone more than 3x its own effective poll interval without a successful scrape. Interval-aware: the expression divides graph2otel_scrape_staleness_seconds by graph2otel_collector_expected_interval_seconds (the scheduler's resolved interval, not the raw config value), so a 5-minute and a 24-hour collector each get a correct threshold. The 3x multiplier tolerates one missed poll plus backoff jitter; 2x was rejected because several workloads have mandatory client-side rate limiters that make an occasional missed poll routine.
No data: Alerting — the one rule that alerts on absence. The whole query returning zero rows means every collector's self-observability signal went dark at once: the process died, or the only collector for a tenant was removed. It does not mean one collector disappearing. Grafana evaluates this rule per (tenant_id, collector) pair the query returns, so a deliberately removed or disabled collector simply stops having a series and its instance resolves silently. That is the correct outcome, not a gap.
Evaluator error: Error. A vector-match failure here would mean the two metrics stopped carrying the same (tenant_id, collector) label pair, which is a graph2otel bug worth seeing rather than swallowing.
False positives: a long-running Graph call near an interval boundary can push one cycle over 3x. The for: 10m window plus the 3x margin absorb one slow cycle, not zero.
Remediation: open Self-obs → Collector health and read the scrape error rate by error type for that collector. A throttled workload shows up on Graph throttling and outbound HTTP as well — if g2o-throttle-saturation is also firing, fix that first, since staleness is the symptom. Check the exporter's own logs for the collector name.
What this rule no longer covers: a collector that hits a permanent 403 and declines the run — it records permission_denied, returns no error, and stamps last-success, so staleness stays flat and this rule stays silent. That is deliberate: an endpoint the tenant is not licensed for can never be actioned, and before #408 it paged critical forever. g2o-collector-degraded-sustained is the rule that covers that collector now, at warning. A 403 the collector could not handle still returns an error and still climbs staleness here, so a hard authorization failure has not gone quiet.
g2o-collector-degraded-sustained¶
A collector has not had one successful scrape in six hours. It is running and reporting on schedule — this is not staleness — but every run comes back degraded or failed. graph2otel_scrape_success_ratio is level-triggered (re-exported on every OTLP interval, not only when a scrape finishes), so max_over_time over 6h needs no interval arithmetic: a 24-hour collector whose last run succeeded holds 1 across the whole window and never fires.
Warning rather than critical on purpose. The two causes are a revoked Graph consent grant, which has already been broken for six hours by the time this fires, and an endpoint the tenant is not licensed for, which cannot be actioned at all. Neither is a 3am page.
No data: OK. A collector that is disabled or removed has no series, and its silent disappearance is the correct outcome.
Evaluator error: Error. The expression is a single range aggregation over one gauge, so an error is a datasource or expression problem, not a tenant one.
False positives: an exporter restarted inside the window can show 0 for a collector that has not had its first tick yet. The 30m pending window covers a normal restart; a restart loop shows up here, which is arguably correct.
Remediation: read cause= on the WARN collector completed with degraded
outcome line for that collector, or split graph2otel_scrape_outcomes_total by result. permission_denied on an endpoint that used to work means a consent grant was revoked — re-consent the app role named by the collector's RequiredPermissions(). permission_denied on a beta or preview endpoint usually means the tenant lost (or never had) the entitlement; Graph says so in the body ("Your tenant is not licensed for this
feature."), no grant will clear it, and the honest fix is to disable that collector for the tenant. Any other cause is a real failure — follow it from Self-obs → Collector health.
g2o-collector-watermark-stalled¶
A window collector's durable checkpoint watermark has stopped advancing: it is more than 20x that collector's own effective poll interval behind wall-clock. This is the #417 fingerprint, and it is the one fault where every other self-observability signal reads green while the collector ships nothing.
Recognise it by what it does NOT look like. Over the 11-day #417 freeze on m7kni: g2o-collector-staleness peaked at 1.008 against a threshold of 3; g2o-collector-degraded-sustained sat at exactly 1 every hour for 7 days; graph2otel_collector_availability reported state=healthy, reason=success; and nothing was logged, because nothing errored. The scrapes genuinely succeeded — the collector re-polled one frozen 15-minute window, re-fetched records already in its SeenIDs set, and deduped them. In graph2otel_record_outcomes_total that shows as fetched == mapped ==
deduped with the emitted series entirely ABSENT — entra.signins.interactive recorded 2013 / 2013 / 2013 over 7 days and no emitted series at all. A healthy scrape ratio is not evidence against this alert; it is part of the fault's shape.
Why the watermark and not the outcome counters. An expression over record_outcomes cannot separate this from a genuinely quiet tenant re-polling its overlap window and deduping every record, which is a normal steady state on a small tenant. The watermark can: logpipeline.Poll advances it to (window end − SafetyLag) even when the window drained zero records, so a quiet collector's watermark keeps moving at its poll interval and only a stalled window freezes it. It also catches a case no counter can see — a watermark frozen in the future, where the collector fetches nothing at all because its window is unreachable, so there are no outcomes to count. (If you do write an outcome-counter query while investigating, use unless, never == 0: == 0 never matches an absent series, and absent is exactly what emitted was.)
PAUSED, and the threshold is a placeholder. Unblock condition: observe (time() - graph2otel_collector_watermark_timestamp_seconds) /
graph2otel_collector_expected_interval_seconds across every window collector for at least one full week on a live tenant, take the per-collector maximum over that week, and set the threshold above the largest. That measurement could not be taken before this metric shipped. 20 is a guess sized to clear the per-collector SafetyLag, which is subtracted from the watermark, is not exported, and therefore inflates this ratio by an unknown amount that hurts the fastest collectors most. Enabling it before the measurement risks firing on correct data — and an alert that fires on correct data trains the reader to ignore it, which is precisely how the next 11-day freeze goes unnoticed.
No data: OK. Only window collectors that have drained at least one window report this metric. Blob consumers track a byte offset with no timestamp and never appear; a collector that has not yet completed a window is absent rather than reported as infinitely stale.
Evaluator error: Error. Both metrics carry exactly (tenant_id,
collector), so the division is a one-to-one vector match with no on()/ignoring() — an error here is a datasource or expression problem, not a tenant one.
False positives: a collector whose SafetyLag is a large fraction of its poll interval sits at a permanently elevated ratio. That is the measurement above, and the reason this ships paused.
Remediation: confirm the freeze first — read the collector's watermark on the admin status page (/api/status.json, per-collector checkpoint state) and against the on-disk checkpoint under the mounted checkpoints/ volume. If the watermark is genuinely frozen, restart the exporter: a restart re-reads the checkpoint and resumes from watermark − overlap, which clears a livelock but does not clear a corrupt or future-dated watermark. For a watermark ahead of now, or one that re-freezes after a restart, the checkpoint file itself is the problem — capture it before touching it, then delete that collector's checkpoint to force a cold start from its initial lookback, accepting the gap between the frozen watermark and the lookback horizon.
g2o-checkpoint-persist-errors¶
A WindowCollector's high-water mark is not reaching disk, so a restart re-polls (or, depending on the store, drops) an already-processed window. Paused by default because it needs a notification channel decision, not because it is noisy.
No data: OK.
Evaluator error: Error.
False positives: effectively none — the counter only increments on a real write failure. The rule fires on the first failed persist with no for delay, which is deliberate for a durability signal.
Remediation: check the checkpoint path is writable and has space. In Kubernetes this is almost always the Helm chart's default emptyDir: set persistence.enabled=true, because an emptyDir also silently loses every watermark on pod replacement. Confirm recovery on Self-obs → Collector health → Checkpoint persist error rate.
g2o-record-integrity-loss¶
At least one source record was dropped or errored in the last 15 minutes. dropped is deliberate rejection — most importantly a record with no parseable event time, which graph2otel refuses to stamp with the arrival clock. errored is decode or processing failure. Enabled by default: either outcome means fetched data did not become telemetry.
No data: OK.
Evaluator error: Error.
False positives: none in the strict sense, but a chronically malformed upstream produces a permanent low rate that is truthful and unfixable from graph2otel's side. Intentional filters and overlap dedupe are separate outcomes and do not fire this rule.
Remediation: open Self-obs → Record outcome accounting and split by outcome and collector. dropped on a log-shaped collector usually means a timestamp field changed shape upstream — check g2o-payload-type-mismatch and the Microsoft API drift panel. Sustained errored on one collector is a mapper bug: capture the raw payload and open an issue rather than tuning the alert.
g2o-record-attrs-truncated¶
At least one log record exceeded the backend's structured-metadata size limit in the last 15 minutes and had its largest attribute values shortened to fit (#419). This is content loss, not record loss — the record landed, and every attribute is still on it, but one or more values are truncated.
No data: OK.
Evaluator error: Error.
False positives: none. The counter only moves when a record genuinely did not fit, and graph2otel clips only what it must. Enabled at >0 because the measured rate is 2-3 records per day, which is not noisy, and because a clip is the only thing that names an oversized shape.
Remediation: this alert is a diagnostic, not a fire. Find the record:
attrs_truncated_keys names the fields that were shortened and attrs_truncated_bytes says how much was lost. A collector that appears here once is fine — the guard did its job. A collector that appears consistently has a source field that genuinely does not fit, and wants a cap at its mapper, where the mapper can decide what to keep, rather than a blind byte cut at the emitter boundary. attrs_dropped being present is louder: that record was so wide its attribute KEYS alone exceeded the budget, and it is missing dimensions rather than merely shortened ones.
g2o-record-over-horizon¶
Records were dropped because their event time was older than the backend's 7-day accept window, so sending them would have been rejected per-entry and lost anyway (#401). This is real record loss, and record_outcomes counts these as emitted — this counter, not that one, is the authoritative statement that they did not land.
No data: OK.
Evaluator error: Error.
False positives: not false measurements — every counted record really was dropped — but on a blob-derived stream the steady state is nonzero for a reason that needs no action. See below.
Paused by default, and the reason matters. A nonzero value is expected on a blob-derived stream. Those replay historical records, so a record can cross the 7-day window by ordinary aging with nothing misconfigured anywhere: #297 measured graph2otel's blob-ingested Intune stream at 3.31 days old at the newest, 5.97 median and 6.95 oldest. Enabling this rule at >0 on such a tenant pages on normal behaviour.
Unblock condition: open Self-obs → Backend accept window and read your own steady-state rate over a week. Raise the rule's threshold above it, then enable. On a Graph-only deployment with no blob ingest the steady state is genuinely zero and >0 is the right threshold as written.
Remediation: pair with Source-event lag at emission on the same board — a rising p95 there is the leading indicator of this counter starting to move. A sudden step usually means a stalled collector resumed and is replaying a backlog that has aged out; check the collector's checkpoint rather than the alert.
g2o-otlp-delivery-failing¶
The OTLP exporter's own callback reported that a batch did not reach the backend in the last 15 minutes, for the named signal (logs or metrics).
No data: OK.
Evaluator error: Error.
False positives: none. An export failure is never a normal steady state. A single transient network blip can produce one, and the SDK retries transport errors — so one isolated firing that does not recur is worth a glance, not an investigation. A sustained or repeating count is real, unrecovered loss.
This is the generic backstop, and that is the point. g2o-record-attrs-truncated and g2o-record-over-horizon each guard one known backend limit. This rule fires on any rejection class — a size limit, a label-count limit, an expired credential, a payload cap, a limit the backend introduces next year — without anyone having predicted it. Treat the per-limit rules as diagnostics that tell you which limit; treat this one as the thing that tells you a limit exists at all. It is what would have caught #419 on the day it began, rather than days later by grepping container logs.
Known blind spot — this rule's silence is not proof of health. Its own evidence travels through the metrics exporter, so a total metrics outage takes the counter with it and this rule goes quiet at exactly the wrong moment. It cannot be the metrics-path watchdog; g2o-collector-staleness, /readyz and the process-local admin status are. This limitation is why a delivery alert was originally forbidden outright (#268), and #421 narrowed that ban to what the reason actually supports: everything that reports itself is worth alerting on, and that is most of it — every logs-side failure (the metrics path is healthy throughout, which is exactly how #419's failures stayed queryable while being invisible), and every partial metrics-side rejection, whose accepted batches carry the counter.
Remediation: the rejection reason is not in the metric — it is in the process's stderr, on the otel sdk error line, which quotes the backend's response body verbatim:
That body names the limit and the measured value. From there:
- a size rejection means an oversized record the emitter budget did not catch — check
g2o-record-attrs-truncatedandMaxAttributeBytes; - a timestamp rejection means the horizon guard is set wider than the backend's real accept window — see
g2o-record-over-horizon; - a 401/403 is a credential problem, not a data problem;
- anything else is a limit graph2otel does not yet model, and wants an issue rather than a threshold change.
Cross-check Self-obs → OTLP delivery for whether the degradation cleared: the degraded gauge is 1 only while the latest failure has not been followed by a successful export, so a 0 there with a nonzero counter here means the exporter recovered but records in the failed batch are still gone.
g2o-payload-type-mismatch¶
A source-controlled optional field arrived with a different JSON type than the collector expects. The record was still emitted; this is report-only. Paused until your tenant's normal payload-shape baseline is known.
No data: OK.
Evaluator error: Error.
False positives: common on first deployment, which is why it ships paused. Some Microsoft fields legitimately carry more than one wire shape across workloads, and that does not make the record wrong.
Remediation: read field, expected_type and actual_type from the alert labels — the metric never carries field values, so there is nothing sensitive in them. If the new shape is legitimate, the fix is in graph2otel's expected-type set, not in the alert. Panel: Self-obs → Record outcome accounting → Payload type mismatches.
g2o-throttle-saturation¶
Microsoft Graph returned 429s on a workload and was still doing so 15 minutes later. The client-side rate limiter counts these itself, so this is a purpose-built counter rather than an inference from scrape errors. None of the throttled workloads reliably send Retry-After, so silent throttling degrades data freshness before anything else visibly breaks.
No data: OK. Zero throttle events is the healthy steady state.
Evaluator error: Error.
False positives: a burst at process start, when every snapshot collector races to fill its first poll. The for: 15m window is meant to filter that; widen it if your startup bursts routinely last longer.
Remediation: the alert's workload label names the ceiling being hit — reporting is 5 requests per 10s, Identity Protection 1/s per tenant across all applications sharing the tenant, Intune reports-export 48/min. Because the Identity Protection limit is tenant-wide, another application in your tenant can throttle graph2otel; check whether anything else started polling before assuming a graph2otel change. Otherwise lengthen the poll intervals of the collectors on that workload. Panel: Self-obs → Graph throttling and outbound HTTP.
g2o-throttle-budget-consumption¶
Graph's own x-ms-throttle-limit-percentage response header reported over 80% budget consumed for 15 minutes. Paused by default.
No data: OK, and this is the trap: the header is not guaranteed on every workload or every response, so absence here does not mean the budget is healthy. g2o-throttle-saturation is the primary signal; this is a companion for when the header happens to be present.
Evaluator error: Error.
False positives: a single sampled header value can read high without sustained pressure, hence the 15-minute window.
Remediation: as for g2o-throttle-saturation — reduce request rate on the named workload. Being over 80% with no 429s yet is the useful case: you have headroom to act before data freshness suffers.
g2o-mdca-uploads-stopped¶
A Defender for Cloud Apps Cloud Discovery stream has not parsed successfully in over 3 hours. This is the alert-on-silence signal a failure counter cannot produce: a dead uploader emits no failed parse tasks, so g2o-mdca-parse-failing stays green forever while data quietly stops.
No data: OK. A tenant with no Cloud Discovery streams legitimately emits no series. Once a stream has parsed once, its age gauge is always present and climbs when uploads stop, so absence and silence are genuinely different here.
Evaluator error: Error.
False positives: the 3-hour threshold is a placeholder for roughly 3x your upload cadence. On a daily uploader it fires constantly; replace it with your own number in seconds before trusting it.
Remediation: check the uploader itself first — an upload returns 200 {"success":true} the moment the blob lands, so a green uploader proves nothing about parsing. Then look at Defender → Defender for Cloud Apps for the last-success age per input_stream_id. A stream that was decommissioned in MDCA keeps its age gauge climbing; remove it from the collector's configuration rather than muting the rule.
g2o-mdca-parse-failing¶
At least one Cloud Discovery parse task failed in the last hour while the upload almost certainly reported HTTP 200. The template label names the failure — ..._BASELOGPARSER_UNEXPECTED_FORMAT means a malformed log format.
No data: OK.
Evaluator error: Error.
False positives: none: a parse failure is never normal. It is paired with g2o-mdca-uploads-stopped on purpose, because this rule cannot see a dead uploader.
Remediation: the failure is in the log format, not in MDCA — the same malformed line produced 22 consecutive silent parse failures on the tenant this collector was built for. Compare a captured sample against the parser template named in the label, fix the exporting appliance's format, and watch Defender → Defender for Cloud Apps → Discovery parse tasks return to is_success="true".
Portable detections (all paused)¶
These eleven ship in a separate folder and rule group from the health rules above, and every one of them is paused. None of their thresholds has been measured on more than one tenant, and each carries a tuning_required annotation naming the measurement it needs. A detection that fires on correct data is worse than no detection: it teaches responders to ignore the channel.
Every tuning_required note names a query on the hunting library that produces its missing measurement. A named measurement with no way to take it is a rule nobody can safely enable, so take the measurement first — that page is the instrument, this one is the response.
They are all Loki-backed, so their no-data and error semantics are shared: no data is OK because no matching records is the healthy steady state, and an evaluator error is Error because a Loki outage must not read as a quiet tenant.
g2o-detect-privileged-directory-change¶
An entra.directory_audit activity matched the high-risk set: application credential or secret added, admin consent granted, app-role or delegated permission granted, service principal or application created, directory-role member added, owner added, or a Conditional Access policy changed. The activity list is the portable, tenant-independent part of this rule and is tedious to reconstruct.
No data: OK.
Evaluator error: Error.
False positives: by design. Every activity in the list is also ordinary administrative work, so this fires on your own changes.
Remediation: inspect initiated_by and target_resources on the record and confirm the change was yours. Panel: Entra → Logs → Top directory audit activities. Before enabling, run the query over 30 days to learn your normal change rate, then decide between a review queue and a pager, or exclude specific initiators.
g2o-detect-security-alert-unresolved¶
An unresolved medium or high alert is open on the entra.security_alert stream. That stream carries alerts from every Microsoft source that surfaces through the security API — Defender for Endpoint, Defender for Cloud Apps and Entra ID Protection all arrive here, so one rule covers products that are separate consoles in the portal.
No data: OK.
Evaluator error: Error.
False positives: depends entirely on your Defender licensing and tenant size, not on the query. Volume, not correctness, is what makes this noisy.
Remediation: read title, category and service_source to see which product raised it, then work it in that product's console. Panel: Defender → Logs → Defender alerts. Measure your alert rate over 30 days before enabling; on a noisy tenant raise the threshold or narrow to severity=high.
g2o-detect-security-incident-active¶
An active medium or high incident — the correlation layer above individual alerts, grouping the alerts Microsoft believes are one attack.
No data: OK.
Evaluator error: Error.
False positives: it deliberately overlaps with the alert rule above, and a single security event will usually match both. Enabling both means being paged twice for one event.
Remediation: pick this or the alert rule, not both, after watching one real incident arrive and seeing which gives your responders the better entry point. The incident carries the correlation; the alert carries the detail. Panel: Defender → Logs → Alerts by severity and detection source.
g2o-detect-graph-403-burst¶
One application took more than 10 Graph authorization denials in 5 minutes. Permission probing looks like this, and so does a compromised identity exploring what it can reach.
No data: OK.
Evaluator error: Error.
False positives: an application that legitimately lost a consent grant produces exactly this shape — worth knowing either way. Needs entra.graph_activity, which arrives over blob ingest, so the rule can never fire without that transport configured.
Remediation: check app_id and the denied paths, then decide whether the caller should have the permission. Panel: Entra → Graph API activity. The threshold of 10 in 5 minutes came from one small tenant; measure your own per-application 403 baseline before enabling.
g2o-detect-interactive-signin-anomaly¶
A real user sign-in that Conditional Access refused, or that Entra ID Protection scored atRisk or confirmedCompromised.
Error 50097 "Device authentication is required" is excluded from the CA limb by default. A report-only Conditional Access policy is still evaluated, and a report-only grant the device cannot satisfy makes Entra stamp conditional_access_status=failure with 50097 on a sign-in that nothing blocked. Measured on a live tenant (2026-08-10): all 6 interactive 50097 records in 30 days had every enforced policy returning success, the only non-success entry a report-only compliant-device grant, and a success record under the same correlation_id about one second later. Report-only is the documented way to stage a CA policy, so any tenant rolling one out produces this.
No data: OK.
Evaluator error: Error.
False positives: Conditional Access failures include ordinary events such as a user declining or fumbling an MFA prompt, so even after the 50097 exclusion most tenants need a threshold above zero or a narrowing to risk_state alone. The risk states require Entra ID P2.
False negatives — read before enabling on a tenant that enforces device compliance. A 50097 that is not followed by a success is a genuine block, and on a tenant enforcing a compliant-device or hybrid-join grant that is the normal case. Loki cannot join two records on correlation_id, so the rule cannot tell an interrupt from a block — the same limitation that keeps impossible travel out of this pack. Put 50097 back if you enforce such a grant.
Remediation: check user_principal_name, app_display_name, status_error_code and ip_address on the record, then read appliedConditionalAccessPolicies on the sign-in in Entra to find which policy returned failure — graph2otel does not export that field, and without it a CA failure names no policy. A result of reportOnlyFailure there means the policy did not block anything. Panel: Entra → Logs → Failed sign-ins. Before enabling, run Which Conditional Access failures does your tenant produce, by error code to confirm the exclusion fits your tenant and to measure what is left. That hunt uses a 14-day window on purpose: a [30d] count_over_time exceeds the max query range on at least one Grafana Cloud Loki stack and returns empty rather than an error, which reads as a clean tenant. The tenant this rule came from adds a third clause for sign-ins outside its expected country; that is a per-tenant policy statement rather than a portable default, so it is not shipped — add an OR term filtering location_country_or_region against your own country code if you want it, and guard it with a presence check (location_country_or_region != "") too, because Loki reads a missing label as the empty string and a bare != fires on records carrying no location rather than on foreign ones.
g2o-detect-exchange-inbox-rule-change¶
An m365.audit record whose operation names an inbox-rule cmdlet: a mailbox rule was created, changed or removed. Business email compromise leaves one behind almost every time — a rule that files replies from finance, or anything mentioning invoices, into a folder the owner never opens, so the victim never sees the conversation running in their name. Microsoft calls the technique email hiding rules (MITRE T1564.008) and the Defender BEC playbook checks for it first.
No data: OK.
Evaluator error: Error.
False positives: constantly. Users create their own inbox rules, so on most tenants this is the noisiest rule in the pack. The rule parameters are not on the audit record, so it cannot tell a forwarding rule from housekeeping.
Remediation: read user_id, client_ip and modified_property_names, then look the rule up in the mailbox and confirm the owner made it. Panel: M365 → Top audited operations by workload. Before enabling, run Which audit operations does your tenant actually record — it gives you the rate and confirms your tenant spells the operation the way the regex expects.
g2o-detect-mailbox-permission-grant¶
An m365.audit operation granting mailbox, recipient or folder permission. Delegate access is durable, survives the owner's password reset, and is invisible to the owner, which is why granting it is a standard persistence and collection step once an attacker holds an administrative session (MITRE T1098.002).
No data: OK.
Evaluator error: Error.
False positives: shared mailboxes, resource calendars and migration tooling all produce legitimate grants, often in bursts.
Remediation: identify the grantee and the mailbox from the record, then confirm with the mailbox owner rather than with whoever made the change. Panel: M365 → Unified audit — which user, which operation. This is a separate rule from the inbox-rule detection on purpose even though both key on operation: an inbox rule is usually the owner's own doing and noisy, a delegation grant is an administrative act and rarer, and one threshold cannot serve both base rates.
g2o-detect-identity-risk-detection¶
An entra.risk_detection record at medium or high level whose risk_event_type is one of the detections worth waking up for. This stream answers the question the sign-in stream cannot: why a sign-in was risky. entra.signin carries risk_state, so a rule on it can say a session was at risk; risk_event_type exists only here and names the detection — impossible travel, an unfamiliar sign-in property, an anonymised or known-malicious address, credentials found in a public dump, a password-spray victim.
Impossible travel is worth calling out: it is a correlation Microsoft has already computed. Expressing it over raw sign-ins would need a join between two records and a distance calculation, and Loki cannot join — so reading Microsoft's verdict is not a shortcut, it is the only correct way to have this detection at all.
No data: OK. On a tenant without Entra ID P2 this is permanent: the endpoint answers with an empty collection, which reads exactly like a clean tenant.
Evaluator error: Error.
False positives: a corporate VPN or a mobile carrier NAT can produce travel-shaped and anonymised-address detections for an entirely ordinary user. leakedCredentials on a large tenant can arrive at a rate no responder can work.
Remediation: read mitre_techniques, ip_address, the location fields and user_principal_name, then confirm with the user before forcing a reset. Panel: Entra → Risk detections by event type and level. Before enabling, run Which risk detection types does Identity Protection raise here: the event-type list in the rule is Microsoft's published set, not one measured on this project's wire.
g2o-detect-workload-identity-risk¶
An entra.service_principal_risk_detection at medium or high level whose risk_state is atRisk or confirmedCompromised. The workload-identity half of Identity Protection, and the half that gets watched least: a service principal has no MFA to fall back on, its credential usually lives in a pipeline variable rather than a vault, and nothing prompts a human when it is used from somewhere new.
No data: OK, and normally permanent — this stream is silent on a healthy tenant, which makes a threshold of zero plausible and unmeasured at the same time. The endpoint returns real detections even without Workload Identities Premium (live-measured), so silence here is genuinely an absence of detections rather than a licence wall.
Evaluator error: Error.
False positives: a credential legitimately reused from a new build agent or a relocated runner can score anomalous.
Remediation: read service_principal_name, app_id and risk_detail, then rotate the credential rather than only dismissing the detection — dismissing clears the flag and leaves the secret. Panel: Entra → Risky service principals total (the gauge one layer up: it counts how many identities are risky now, where this rule fires on the detection that made one of them risky). This is the portable counterpart to pinning each automation identity to its own expected source address, which cannot ship here because those values are tenant infrastructure — see the pattern at the end of this page.
g2o-detect-legacy-auth-signin¶
An entra.signin whose client_app_used is a legacy protocol. Legacy protocols cannot present an MFA challenge, so a sign-in that succeeds over one has bypassed multi-factor authentication whatever the Conditional Access policy says. Password spraying targets them for exactly that reason.
This is deliberately separate from g2o-detect-interactive-signin-anomaly: that rule asks whether a sign-in looked suspicious, this one asks whether a channel exists that cannot be challenged — worth knowing even when every sign-in on it is legitimate.
No data: OK. On a tenant that already blocks legacy authentication, silence is the correct and permanent answer.
Evaluator error: Error.
False positives: a tenant with legitimate legacy clients still in migration fires this continuously. That is not a wrong measurement, but it belongs on a dashboard rather than a pager until the migration finishes.
Remediation: identify the account and client, then block the protocol with a Conditional Access policy rather than chasing individual sign-ins. Panel: Entra → Top failing sign-in sources (country, client app). Before enabling, run Which client apps sign in, and how much legacy protocol is left — the client names in the rule are Microsoft's spellings and have not been measured here. Add status_error_code=0`` to narrow to successful legacy sign-ins, which is the smaller and more urgent set.
g2o-detect-mail-remediation-failed¶
A defender.email_post_delivery record whose action_result is present and is not a success: Defender tried to remove a message it had already delivered, and the removal did not land. Zero-hour auto purge exists because a message can be reclassified as malicious after delivery — when the purge fails, the message is still sitting in an inbox Microsoft has already decided is dangerous, and nothing else tells you so. The alert says the threat was found; this record says whether it was actually removed.
No data: OK. Also the state when Defender advanced-hunting blob ingest is not configured, in which case the stream is absent rather than clean.
Evaluator error: Error.
False positives: a transient failure that a later retry resolves looks identical to a permanent one on a single record.
Remediation: read action_type, action_trigger and recipient_email_address, then remove the message by hand and confirm it is gone. Panel: Defender → Quarantine held messages total — the nearest surface, since both are about messages Defender has already judged malicious and acted on.
The filter has two terms for a reason worth remembering: a negative label filter also matches a record carrying no action_result at all, because LogQL treats a missing structured-metadata key as the empty string. The presence term ction_result=~`.+ is what keeps this from firing on every remediation record. Run Which post-delivery mail remediations succeed to see which values your tenant emits before enabling.
A pattern worth knowing that is not shipped¶
A workload identity that legitimately signs in from exactly one place — a CI runner, a scheduled sync job, a self-hosted integration — makes an almost-zero-false-positive detection, and it cannot be shipped because it is specific by nature:
{service_name="graph2otel"} | event_name=`entra.signin` | app_id=`<your application id>` | status_error_code!=`0`
plus an equivalent term for source address, ORed together and alerted above zero.
Detecting rather than preventing is a licensing decision worth stating plainly: Conditional Access IP-locking for workload identities requires Microsoft Entra Workload ID, a separate paid add-on. Without it there is no way to stop a leaked service-principal credential being used from anywhere, but the sign-in stream lets you notice within one evaluation interval at no extra licence cost. Build one rule per such application.