---
description: One section per shipped rule. Every rule under alerts/rules/ — health
  alerts and paused detections alike — carries a runbookurl annotation pointing at
  its…
---

# Alert runbooks

One section per shipped rule. Every rule under
[`alerts/rules/`](https://github.com/rknightion/graph2otel/tree/main/alerts/rules)
— health alerts and paused detections alike —
carries a `runbook_url` annotation pointing at its section here, so the link in a
notification lands on the guidance for that exact rule. Paused rules have
runbooks too: a paused rule is the one you are about to enable, and this page is
where you find out whether enabling it is safe.

Each section answers the four states a responder actually meets — it fired, it
went to no data, it errored, and it was wrong.

## How the links work

Every rule also carries `__dashboardUid__` + `__panelId__`, so Grafana renders a
**View panel** link straight to the panel that shows the same signal, plus a
`dashboard_path` annotation holding the same target as a deep link relative to
your Grafana host (`/d/graph2otel?dtab=<Tab>&viewPanel=<id>`). Both are generated
from `dashboards/graph2otel.json`, so they cannot name a panel or tab that does
not exist. Import that dashboard before the rules if you want the links live.

`runbook_url` and the panel annotations are generated by
`grafana/build_rules.py`. Renaming a section heading here breaks
`make grafana-check` rather than shipping a dead link.

## Rule-wide behaviour

**Evaluator errors** are never silent: every rule ships `execErrState: Error`, so
a datasource outage, a bad query or an expression failure surfaces as an
**Error** state rather than as a healthy rule. An erroring rule is not evidence
of a healthy tenant, and it is not evidence of an unhealthy one either — treat it
as "this rule told you nothing this interval" and check the datasource first.

**No data** is configured per rule, because absence means different things for
different signals. Most rules use `OK`: an empty result is the healthy steady
state for anything counting failures. `g2o-collector-staleness` is the one
exception and uses `Alerting`.

**Thresholds are starting points.** Where a number came from one tenant, the
section says so. Tune it against your own measurement rather than treating it as
a recommendation.

## graph2otel health alerts

### g2o-entra-cred-expiry-critical

An Entra application or service-principal credential (client secret or
certificate) lands in the `lt_7d` or `expired` bucket. When one of these expires,
sign-in or an integration breaks with no warning.

**No data:** `OK`. No soon-expiring credentials is the healthy state. It also
means the `entra.credential_expiry` collector may be off — check
**Self-obs → Collector availability** before concluding your inventory is clean.

**Evaluator error:** `Error`. The metric is a bucketed count, so an error here is
a datasource or expression problem, never a tenant problem.

**False positives:** a credential already scheduled for rotation is expiring on
purpose, and the count cannot tell that apart from a forgotten one. Tenants with
deliberately short-lived credentials sit in the bucket permanently.

**Remediation:** find the owning app from
**Entra → Applications, service principals and credentials**, then the entity
itself in the `entra.credential_expiry` log twin (`owner_type`,
`credential_type`, the owning object). Rotate the credential, or move it into an
automated rotation. If the alert is chronic because rotation is automated,
suppress by `owner_type` rather than raising the threshold — you still want the
unautomated ones.

### g2o-entra-cred-expiry-warning

The earlier tier of the rule above, on the `lt_30d` bucket. Paused by default.

**No data:** `OK`, same reasoning as the critical tier.

**Evaluator error:** `Error`.

**False positives:** more of them than the critical tier by construction. Any
credential renewed on a 30-day-or-shorter cycle keeps this bucket non-zero
forever, which is why it ships paused.

**Remediation:** before enabling, decide your rotation lead time. If it is under
30 days, this rule will be permanently firing and should stay paused; the `lt_7d`
critical tier is then your only credential alert. Otherwise enable it and treat
it as a work queue, not a page.

### g2o-intune-apple-token-expiry-critical

An Apple MDM token (APNS, VPP or DEP) has fewer than 14 days left. Paused by
default. Unlike the bucketed credential metrics this is a raw days-remaining
gauge over a small admin-configured token set, so the threshold is an exact day
count you can tune freely.

**No data:** `OK`. A tenant with no Apple MDM emits no series at all, which is
the normal case for a Windows-only fleet.

**Evaluator error:** `Error`.

**False positives:** rare — the gauge is a real countdown. The usual surprise is
the opposite: the token set is per `token_name`, so a decommissioned token that
Intune still lists keeps counting down and firing.

**Remediation:** APNS certificates renew in the Apple Push Certificates Portal
and must be renewed with **the same Apple ID** that created them, or every
enrolled iOS/macOS device has to re-enrol. VPP and DEP tokens renew in Apple
Business/School Manager. Confirm the new expiry on
**Intune → Tokens, certificates and connectors**.

### g2o-intune-cert-expiry-critical

Intune-managed certificates in a profile are in the `0d_7d` or `expired` bucket.
Paused by default; needs the beta `intune.certificates` collector.

**No data:** `OK`. The collector is opt-in, so no series is the default state.

**Evaluator error:** `Error`.

**False positives:** short-lived certificates by design (a SCEP profile issuing
weekly certificates) sit in the urgent bucket permanently. The bucket ladders
differ between the two expiry collectors and this catches people out: Intune
certificates use `0d_7d` / `7d_30d` / `30d_90d` / `over_90d` / `unknown`, Entra
credentials use `lt_7d` / `lt_30d` / `lt_90d` / `gt_90d` / `expired`.

**Remediation:** identify the profile from `cert_profile_name` on
**Intune → Tokens, certificates and connectors**, then check the issuing
authority — an expiring leaf set usually means the SCEP/PKCS connector or the CA
itself stopped issuing, not that thousands of devices independently expired.
`g2o-collector-staleness` on the connector collector is the corroborating signal.

### g2o-intune-compliance-ratio-low

The compliant fraction of the Intune fleet is under 90% for 30 minutes. The
`and >= 5` fleet-size guard stops a two-device pilot tenant paging on one
device.

**No data:** `OK`. No Intune compliance policies, or the `intune.compliance`
collector disabled, means no series.

**Evaluator error:** `Error`. This rule is a ratio of two sums, so a partial
datasource failure can error rather than produce a wrong number, which is the
intended behaviour.

**False positives:** a compliance re-evaluation window right after a policy
rollout drops devices out of `compliant` transiently. Devices that have not
checked in are counted as not compliant, so a fleet that is largely offline (a
holiday, a shift pattern) looks non-compliant without anything being wrong.

**Remediation:** open **Intune → Compliance and baselines** and split by
`state` and policy. If the drop is one policy, it is a policy change; if it is
spread evenly, it is a check-in problem — cross-check
`intune_devices_sync_staleness_seconds` on **Managed devices and inventory**. The
per-device detail is in the `intune.device` log twin. Widen `for` if your tenant
pushes policy frequently; raise the `>= 5` guard to match your smallest real
tenant.

### g2o-intune-compliance-noncompliant-spike

The non-compliant share rose more than 10 percentage points within an hour.
Companion to the ratio rule, for a large compliant fleet where the absolute ratio
takes a long time to cross 90%. Paused by default: the 1h/10pp thresholds have
not been validated against real fleet churn.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** likely, and the reason it is paused. Any bulk enrolment,
policy assignment change or maintenance window produces a step change that looks
identical to a regression.

**Remediation:** before enabling, run the expression over a month of your own
fleet history and pick a swing size your normal churn does not reach. Once
firing, treat it exactly like the ratio rule above — the difference is only in
how early it fires.

### g2o-collector-staleness

A collector has gone more than 3x **its own** effective poll interval without a
successful scrape. Interval-aware: the expression divides
`graph2otel_scrape_staleness_seconds` by
`graph2otel_collector_expected_interval_seconds` (the scheduler's resolved
interval, not the raw config value), so a 5-minute and a 24-hour collector each
get a correct threshold. The 3x multiplier tolerates one missed poll plus backoff
jitter; 2x was rejected because several workloads have mandatory client-side rate
limiters that make an occasional missed poll routine.

**No data:** `Alerting` — the one rule that alerts on absence. The whole query
returning zero rows means every collector's self-observability signal went dark
at once: the process died, or the only collector for a tenant was removed. It
does **not** mean one collector disappearing. Grafana evaluates this rule per
`(tenant_id, collector)` pair the query returns, so a deliberately removed or
disabled collector simply stops having a series and its instance resolves
silently. That is the correct outcome, not a gap.

**Evaluator error:** `Error`. A vector-match failure here would mean the two
metrics stopped carrying the same `(tenant_id, collector)` label pair, which is a
graph2otel bug worth seeing rather than swallowing.

**False positives:** a long-running Graph call near an interval boundary can push
one cycle over 3x. The `for: 10m` window plus the 3x margin absorb one slow cycle,
not zero.

**Remediation:** open
**Self-obs → Collector health** and read the scrape error rate by error type for
that collector. A throttled workload shows up on
**Graph throttling and outbound HTTP** as well — if `g2o-throttle-saturation` is
also firing, fix that first, since staleness is the symptom. Check the exporter's
own logs for the collector name.

**What this rule no longer covers:** a collector that hits a permanent 403 and
*declines* the run — it records `permission_denied`, returns no error, and stamps
last-success, so staleness stays flat and this rule stays silent. That is
deliberate: an endpoint the tenant is not licensed for can never be actioned, and
before #408 it paged critical forever. `g2o-collector-degraded-sustained` is the
rule that covers that collector now, at warning. A 403 the collector could *not*
handle still returns an error and still climbs staleness here, so a hard
authorization failure has not gone quiet.

### g2o-collector-degraded-sustained

A collector has not had one successful scrape in six hours. It is running and
reporting on schedule — this is not staleness — but every run comes back degraded
or failed. `graph2otel_scrape_success_ratio` is level-triggered (re-exported on
every OTLP interval, not only when a scrape finishes), so `max_over_time` over 6h
needs no interval arithmetic: a 24-hour collector whose last run succeeded holds
`1` across the whole window and never fires.

Warning rather than critical on purpose. The two causes are a revoked Graph
consent grant, which has already been broken for six hours by the time this
fires, and an endpoint the tenant is not licensed for, which cannot be actioned
at all. Neither is a 3am page.

**No data:** `OK`. A collector that is disabled or removed has no series, and its
silent disappearance is the correct outcome.

**Evaluator error:** `Error`. The expression is a single range aggregation over
one gauge, so an error is a datasource or expression problem, not a tenant one.

**False positives:** an exporter restarted inside the window can show `0` for a
collector that has not had its first tick yet. The 30m pending window covers a
normal restart; a restart loop shows up here, which is arguably correct.

**Remediation:** read `cause=` on the WARN `collector completed with degraded
outcome` line for that collector, or split
`graph2otel_scrape_outcomes_total` by `result`. `permission_denied` on an
endpoint that used to work means a consent grant was revoked — re-consent the app
role named by the collector's `RequiredPermissions()`. `permission_denied` on a
beta or preview endpoint usually means the tenant lost (or never had) the
entitlement; Graph says so in the body (`"Your tenant is not licensed for this
feature."`), no grant will clear it, and the honest fix is to disable that
collector for the tenant. Any other cause is a real failure — follow it from
**Self-obs → Collector health**.

### g2o-collector-watermark-stalled

A window collector's durable checkpoint watermark has stopped advancing: it is
more than 20x that collector's own effective poll interval behind wall-clock.
This is the **#417 fingerprint**, and it is the one fault where every other
self-observability signal reads green while the collector ships nothing.

Recognise it by what it does NOT look like. Over the 11-day #417 freeze on
m7kni: `g2o-collector-staleness` peaked at **1.008** against a threshold of 3;
`g2o-collector-degraded-sustained` sat at exactly **1** every hour for 7 days;
`graph2otel_collector_availability` reported `state=healthy, reason=success`;
and nothing was logged, because nothing errored. The scrapes genuinely
succeeded — the collector re-polled one frozen 15-minute window, re-fetched
records already in its `SeenIDs` set, and deduped them. In
`graph2otel_record_outcomes_total` that shows as **`fetched == mapped ==
deduped` with the `emitted` series entirely ABSENT** — `entra.signins.interactive`
recorded 2013 / 2013 / 2013 over 7 days and no `emitted` series at all. A
healthy scrape ratio is not evidence against this alert; it is part of the
fault's shape.

**Why the watermark and not the outcome counters.** An expression over
`record_outcomes` cannot separate this from a genuinely quiet tenant re-polling
its overlap window and deduping every record, which is a normal steady state on
a small tenant. The watermark can: `logpipeline.Poll` advances it to
`(window end − SafetyLag)` even when the window drained **zero** records, so a
quiet collector's watermark keeps moving at its poll interval and only a stalled
window freezes it. It also catches a case no counter can see — a watermark
frozen in the *future*, where the collector fetches nothing at all because its
window is unreachable, so there are no outcomes to count. (If you do write an
outcome-counter query while investigating, use `unless`, never `== 0`: `== 0`
never matches an absent series, and absent is exactly what `emitted` was.)

**PAUSED, and the threshold is a placeholder.** Unblock condition: observe
`(time() - graph2otel_collector_watermark_timestamp_seconds) /
graph2otel_collector_expected_interval_seconds` across every window collector
for at least one full week on a live tenant, take the per-collector maximum over
that week, and set the threshold above the largest. That measurement could not
be taken before this metric shipped. `20` is a guess sized to clear the
per-collector `SafetyLag`, which is subtracted from the watermark, is **not**
exported, and therefore inflates this ratio by an unknown amount that hurts the
fastest collectors most. Enabling it before the measurement risks firing on
correct data — and an alert that fires on correct data trains the reader to
ignore it, which is precisely how the next 11-day freeze goes unnoticed.

**No data:** `OK`. Only window collectors that have drained at least one window
report this metric. Blob consumers track a byte offset with no timestamp and
never appear; a collector that has not yet completed a window is absent rather
than reported as infinitely stale.

**Evaluator error:** `Error`. Both metrics carry exactly `(tenant_id,
collector)`, so the division is a one-to-one vector match with no
`on()`/`ignoring()` — an error here is a datasource or expression problem, not a
tenant one.

**False positives:** a collector whose `SafetyLag` is a large fraction of its
poll interval sits at a permanently elevated ratio. That is the measurement
above, and the reason this ships paused.

**Remediation:** confirm the freeze first — read the collector's watermark on
the admin status page (`/api/status.json`, per-collector checkpoint state) and
against the on-disk checkpoint under the mounted `checkpoints/` volume. If the
watermark is genuinely frozen, restart the exporter: a restart re-reads the
checkpoint and resumes from `watermark − overlap`, which clears a livelock but
does **not** clear a corrupt or future-dated watermark. For a watermark ahead of
now, or one that re-freezes after a restart, the checkpoint file itself is the
problem — capture it before touching it, then delete that collector's checkpoint
to force a cold start from its initial lookback, accepting the gap between the
frozen watermark and the lookback horizon.

### g2o-checkpoint-persist-errors

A `WindowCollector`'s high-water mark is not reaching disk, so a restart re-polls
(or, depending on the store, drops) an already-processed window. Paused by
default because it needs a notification channel decision, not because it is
noisy.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** effectively none — the counter only increments on a real
write failure. The rule fires on the first failed persist with no `for` delay,
which is deliberate for a durability signal.

**Remediation:** check the checkpoint path is writable and has space. In
Kubernetes this is almost always the Helm chart's default `emptyDir`: set
`persistence.enabled=true`, because an `emptyDir` also silently loses every
watermark on pod replacement. Confirm recovery on
**Self-obs → Collector health → Checkpoint persist error rate**.

### g2o-record-integrity-loss

At least one source record was `dropped` or `errored` in the last 15 minutes.
`dropped` is deliberate rejection — most importantly a record with no parseable
event time, which graph2otel refuses to stamp with the arrival clock. `errored`
is decode or processing failure. Enabled by default: either outcome means
fetched data did not become telemetry.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** none in the strict sense, but a chronically malformed
upstream produces a permanent low rate that is truthful and unfixable from
graph2otel's side. Intentional filters and overlap dedupe are separate outcomes
and do not fire this rule.

**Remediation:** open **Self-obs → Record outcome accounting** and split by
`outcome` and `collector`. `dropped` on a log-shaped collector usually means a
timestamp field changed shape upstream — check
`g2o-payload-type-mismatch` and the Microsoft API drift panel. Sustained
`errored` on one collector is a mapper bug: capture the raw payload and open an
issue rather than tuning the alert.

### g2o-record-attrs-truncated

At least one log record exceeded the backend's structured-metadata size limit in
the last 15 minutes and had its largest attribute values shortened to fit
([#419](https://github.com/rknightion/graph2otel/issues/419)). This is **content
loss, not record loss** — the record landed, and every attribute is still on it,
but one or more values are truncated.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** none. The counter only moves when a record genuinely did not
fit, and graph2otel clips only what it must. Enabled at `>0` because the measured
rate is 2-3 records per day, which is not noisy, and because a clip is the only
thing that names an oversized shape.

**Remediation:** this alert is a diagnostic, not a fire. Find the record:

```logql
{service_name="graph2otel"} | attrs_truncated = "true"
```

`attrs_truncated_keys` names the fields that were shortened and
`attrs_truncated_bytes` says how much was lost. A collector that appears here
once is fine — the guard did its job. A collector that appears **consistently**
has a source field that genuinely does not fit, and wants a cap at its mapper,
where the mapper can decide *what* to keep, rather than a blind byte cut at the
emitter boundary. `attrs_dropped` being present is louder: that record was so
wide its attribute KEYS alone exceeded the budget, and it is missing dimensions
rather than merely shortened ones.

### g2o-record-over-horizon

Records were dropped because their event time was older than the backend's 7-day
accept window, so sending them would have been rejected per-entry and lost anyway
([#401](https://github.com/rknightion/graph2otel/issues/401)). This is real record
loss, and `record_outcomes` counts these as `emitted` — this counter, not that
one, is the authoritative statement that they did not land.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** not false measurements — every counted record really was
dropped — but on a blob-derived stream the steady state is nonzero for a reason
that needs no action. See below.

**Paused by default, and the reason matters.** A nonzero value is *expected* on a
blob-derived stream. Those replay historical records, so a record can cross the
7-day window by ordinary aging with nothing misconfigured anywhere:
[#297](https://github.com/rknightion/graph2otel/issues/297) measured graph2otel's
blob-ingested Intune stream at 3.31 days old at the newest, 5.97 median and 6.95
oldest. Enabling this rule at `>0` on such a tenant pages on normal behaviour.

**Unblock condition:** open **Self-obs → Backend accept window** and read your own
steady-state rate over a week. Raise the rule's threshold above it, then enable.
On a Graph-only deployment with no blob ingest the steady state is genuinely zero
and `>0` is the right threshold as written.

**Remediation:** pair with **Source-event lag at emission** on the same board — a
rising p95 there is the leading indicator of this counter starting to move. A
sudden step usually means a stalled collector resumed and is replaying a backlog
that has aged out; check the collector's checkpoint rather than the alert.

### g2o-otlp-delivery-failing

The OTLP exporter's own callback reported that a batch did not reach the backend
in the last 15 minutes, for the named `signal` (`logs` or `metrics`).

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** none. An export failure is never a normal steady state. A
single transient network blip can produce one, and the SDK retries transport
errors — so one isolated firing that does not recur is worth a glance, not an
investigation. A sustained or repeating count is real, unrecovered loss.

**This is the generic backstop, and that is the point.** `g2o-record-attrs-truncated`
and `g2o-record-over-horizon` each guard one *known* backend limit. This rule
fires on **any** rejection class — a size limit, a label-count limit, an expired
credential, a payload cap, a limit the backend introduces next year — without
anyone having predicted it. Treat the per-limit rules as diagnostics that tell
you *which* limit; treat this one as the thing that tells you a limit exists at
all. It is what would have caught
[#419](https://github.com/rknightion/graph2otel/issues/419) on the day it began,
rather than days later by grepping container logs.

**Known blind spot — this rule's silence is not proof of health.** Its own
evidence travels through the **metrics** exporter, so a *total* metrics outage
takes the counter with it and this rule goes quiet at exactly the wrong moment.
It cannot be the metrics-path watchdog; `g2o-collector-staleness`, `/readyz` and
the process-local admin status are. This limitation is why a delivery alert was
originally forbidden outright
([#268](https://github.com/rknightion/graph2otel/issues/268)), and #421 narrowed
that ban to what the reason actually supports: everything that *reports itself*
is worth alerting on, and that is most of it — every logs-side failure (the
metrics path is healthy throughout, which is exactly how #419's failures stayed
queryable while being invisible), and every partial metrics-side rejection, whose
accepted batches carry the counter.

**Remediation:** the rejection reason is not in the metric — it is in the
process's stderr, on the `otel sdk error` line, which quotes the backend's
response body verbatim:

```bash
docker compose logs --since 1h 2>&1 | grep "otel sdk error"
```

That body names the limit and the measured value. From there:

- a **size** rejection means an oversized record the emitter budget did not
  catch — check `g2o-record-attrs-truncated` and `MaxAttributeBytes`;
- a **timestamp** rejection means the horizon guard is set wider than the
  backend's real accept window — see `g2o-record-over-horizon`;
- a **401/403** is a credential problem, not a data problem;
- anything else is a limit graph2otel does not yet model, and wants an issue
  rather than a threshold change.

Cross-check **Self-obs → OTLP delivery** for whether the degradation cleared: the
`degraded` gauge is 1 only while the latest failure has not been followed by a
successful export, so a 0 there with a nonzero counter here means the exporter
recovered but records in the failed batch are still gone.

### g2o-payload-type-mismatch

A source-controlled optional field arrived with a different JSON type than the
collector expects. The record was still emitted; this is report-only. Paused
until your tenant's normal payload-shape baseline is known.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** common on first deployment, which is why it ships paused.
Some Microsoft fields legitimately carry more than one wire shape across
workloads, and that does not make the record wrong.

**Remediation:** read `field`, `expected_type` and `actual_type` from the alert
labels — the metric never carries field *values*, so there is nothing sensitive
in them. If the new shape is legitimate, the fix is in graph2otel's expected-type
set, not in the alert. Panel:
**Self-obs → Record outcome accounting → Payload type mismatches**.

### g2o-throttle-saturation

Microsoft Graph returned 429s on a workload and was still doing so 15 minutes
later. The client-side rate limiter counts these itself, so this is a purpose-built
counter rather than an inference from scrape errors. None of the throttled
workloads reliably send `Retry-After`, so silent throttling degrades data
freshness before anything else visibly breaks.

**No data:** `OK`. Zero throttle events is the healthy steady state.

**Evaluator error:** `Error`.

**False positives:** a burst at process start, when every snapshot collector
races to fill its first poll. The `for: 15m` window is meant to filter that;
widen it if your startup bursts routinely last longer.

**Remediation:** the alert's `workload` label names the ceiling being hit —
reporting is 5 requests per 10s, Identity Protection 1/s per tenant across *all*
applications sharing the tenant, Intune reports-export 48/min. Because the
Identity Protection limit is tenant-wide, another application in your tenant can
throttle graph2otel; check whether anything else started polling before assuming
a graph2otel change. Otherwise lengthen the poll intervals of the collectors on
that workload. Panel:
**Self-obs → Graph throttling and outbound HTTP**.

### g2o-throttle-budget-consumption

Graph's own `x-ms-throttle-limit-percentage` response header reported over 80%
budget consumed for 15 minutes. Paused by default.

**No data:** `OK`, and this is the trap: the header is not guaranteed on every
workload or every response, so **absence here does not mean the budget is
healthy**. `g2o-throttle-saturation` is the primary signal; this is a companion
for when the header happens to be present.

**Evaluator error:** `Error`.

**False positives:** a single sampled header value can read high without
sustained pressure, hence the 15-minute window.

**Remediation:** as for `g2o-throttle-saturation` — reduce request rate on the
named workload. Being over 80% with no 429s yet is the useful case: you have
headroom to act before data freshness suffers.

### g2o-mdca-uploads-stopped

A Defender for Cloud Apps Cloud Discovery stream has not parsed successfully in
over 3 hours. This is the alert-on-silence signal a failure counter cannot
produce: a dead uploader emits no failed parse tasks, so `g2o-mdca-parse-failing`
stays green forever while data quietly stops.

**No data:** `OK`. A tenant with no Cloud Discovery streams legitimately emits no
series. Once a stream has parsed once, its age gauge is always present and climbs
when uploads stop, so absence and silence are genuinely different here.

**Evaluator error:** `Error`.

**False positives:** the 3-hour threshold is a placeholder for roughly 3x your
upload cadence. On a daily uploader it fires constantly; replace it with your own
number in seconds before trusting it.

**Remediation:** check the uploader itself first — an upload returns
`200 {"success":true}` the moment the blob lands, so a green uploader proves
nothing about parsing. Then look at
**Defender → Defender for Cloud Apps** for the last-success age per
`input_stream_id`. A stream that was decommissioned in MDCA keeps its age gauge
climbing; remove it from the collector's configuration rather than muting the
rule.

### g2o-mdca-parse-failing

At least one Cloud Discovery parse task failed in the last hour while the upload
almost certainly reported HTTP 200. The `template` label names the failure —
`..._BASELOGPARSER_UNEXPECTED_FORMAT` means a malformed log format.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** none: a parse failure is never normal. It is paired with
`g2o-mdca-uploads-stopped` on purpose, because this rule cannot see a dead
uploader.

**Remediation:** the failure is in the log format, not in MDCA — the same
malformed line produced 22 consecutive silent parse failures on the tenant this
collector was built for. Compare a captured sample against the parser template
named in the label, fix the exporting appliance's format, and watch
**Defender → Defender for Cloud Apps → Discovery parse tasks** return to
`is_success="true"`.

## Portable detections (all paused)

These eleven ship in a separate folder and rule group from the health rules
above, and **every one of them is paused**. None of their thresholds has been
measured on more than one tenant, and each carries a `tuning_required` annotation
naming the measurement it needs. A detection that fires on correct data is worse
than no detection: it teaches responders to ignore the channel.

Every `tuning_required` note names a query on the
[hunting library](../hunting.md) that produces its missing measurement. A named
measurement with no way to take it is a rule nobody can safely enable, so take
the measurement first — that page is the instrument, this one is the response.

They are all Loki-backed, so their no-data and error semantics are shared: no
data is `OK` because no matching records is the healthy steady state, and an
evaluator error is `Error` because a Loki outage must not read as a quiet tenant.

### g2o-detect-privileged-directory-change

An `entra.directory_audit` activity matched the high-risk set: application
credential or secret added, admin consent granted, app-role or delegated
permission granted, service principal or application created, directory-role
member added, owner added, or a Conditional Access policy changed. The activity
list is the portable, tenant-independent part of this rule and is tedious to
reconstruct.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** by design. Every activity in the list is also ordinary
administrative work, so this fires on your own changes.

**Remediation:** inspect `initiated_by` and `target_resources` on the record and
confirm the change was yours. Panel:
**Entra → Logs → Top directory audit activities**. Before enabling, run the query
over 30 days to learn your normal change rate, then decide between a review queue
and a pager, or exclude specific initiators.

### g2o-detect-security-alert-unresolved

An unresolved medium or high alert is open on the `entra.security_alert` stream.
That stream carries alerts from every Microsoft source that surfaces through the
security API — Defender for Endpoint, Defender for Cloud Apps and Entra ID
Protection all arrive here, so one rule covers products that are separate
consoles in the portal.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** depends entirely on your Defender licensing and tenant size,
not on the query. Volume, not correctness, is what makes this noisy.

**Remediation:** read `title`, `category` and `service_source` to see which
product raised it, then work it in that product's console. Panel:
**Defender → Logs → Defender alerts**. Measure your alert rate over 30 days
before enabling; on a noisy tenant raise the threshold or narrow to
`severity=high`.

### g2o-detect-security-incident-active

An active medium or high **incident** — the correlation layer above individual
alerts, grouping the alerts Microsoft believes are one attack.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** it deliberately **overlaps** with the alert rule above, and
a single security event will usually match both. Enabling both means being paged
twice for one event.

**Remediation:** pick this or the alert rule, not both, after watching one real
incident arrive and seeing which gives your responders the better entry point.
The incident carries the correlation; the alert carries the detail. Panel:
**Defender → Logs → Alerts by severity and detection source**.

### g2o-detect-graph-403-burst

One application took more than 10 Graph authorization denials in 5 minutes.
Permission probing looks like this, and so does a compromised identity exploring
what it can reach.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** an application that legitimately lost a consent grant
produces exactly this shape — worth knowing either way. Needs
`entra.graph_activity`, which arrives over blob ingest, so the rule can never
fire without that transport configured.

**Remediation:** check `app_id` and the denied paths, then decide whether the
caller should have the permission. Panel:
**Entra → Graph API activity**. The threshold of 10 in 5 minutes came from one
small tenant; measure your own per-application 403 baseline before enabling.

### g2o-detect-interactive-signin-anomaly

A real user sign-in that Conditional Access refused, or that Entra ID Protection
scored `atRisk` or `confirmedCompromised`.

Error `50097` "Device authentication is required" is **excluded from the CA limb
by default**. A **report-only** Conditional Access policy is still evaluated, and
a report-only grant the device cannot satisfy makes Entra stamp
`conditional_access_status=failure` with `50097` on a sign-in that nothing
blocked. Measured on a live tenant (2026-08-10): all 6 interactive `50097`
records in 30 days had every *enforced* policy returning `success`, the only
non-success entry a report-only compliant-device grant, and a success record
under the **same `correlation_id`** about one second later. Report-only is the
documented way to stage a CA policy, so any tenant rolling one out produces this.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** Conditional Access failures include ordinary events such as
a user declining or fumbling an MFA prompt, so even after the `50097` exclusion
most tenants need a threshold above zero or a narrowing to `risk_state` alone.
The risk states require Entra ID P2.

**False negatives — read before enabling on a tenant that enforces device
compliance.** A `50097` that is *not* followed by a success is a genuine block,
and on a tenant enforcing a compliant-device or hybrid-join grant that is the
normal case. Loki cannot join two records on `correlation_id`, so **the rule
cannot tell an interrupt from a block** — the same limitation that keeps
impossible travel out of this pack. Put `50097` back if you enforce such a grant.

**Remediation:** check `user_principal_name`, `app_display_name`,
`status_error_code` and `ip_address` on the record, then read
`appliedConditionalAccessPolicies` on the sign-in in Entra to find *which* policy
returned `failure` — graph2otel does not export that field, and without it a CA
failure names no policy. A `result` of `reportOnlyFailure` there means the policy
did not block anything. Panel:
**Entra → Logs → Failed sign-ins**. Before enabling, run
[Which Conditional Access failures does your tenant produce, by error code](../hunting.md)
to confirm the exclusion fits your tenant and to measure what is left. That hunt
uses a 14-day window on purpose: a `[30d]` `count_over_time` exceeds the max query
range on at least one Grafana Cloud Loki stack and returns **empty rather than an
error**, which reads as a clean tenant. The tenant
this rule came from adds a third clause for sign-ins outside its expected
country; that is a per-tenant policy statement rather than a portable default, so
it is not shipped — add an OR term filtering `location_country_or_region` against
your own country code if you want it, and guard it with a presence check
(`location_country_or_region != ""`) too, because Loki reads a missing label as
the empty string and a bare `!=` fires on records carrying no location rather
than on foreign ones.

### g2o-detect-exchange-inbox-rule-change

An `m365.audit` record whose `operation` names an inbox-rule cmdlet: a mailbox
rule was created, changed or removed. Business email compromise leaves one behind
almost every time — a rule that files replies from finance, or anything
mentioning invoices, into a folder the owner never opens, so the victim never
sees the conversation running in their name. Microsoft calls the technique email
hiding rules (MITRE T1564.008) and the Defender BEC playbook checks for it first.

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** constantly. Users create their own inbox rules, so on most
tenants this is the noisiest rule in the pack. The rule parameters are not on the
audit record, so it cannot tell a forwarding rule from housekeeping.

**Remediation:** read `user_id`, `client_ip` and `modified_property_names`, then
look the rule up in the mailbox and confirm the owner made it. Panel:
**M365 → Top audited operations by workload**. Before enabling, run
[Which audit operations does your tenant actually record](../hunting.md) — it gives
you the rate and confirms your tenant spells the operation the way the regex
expects.

### g2o-detect-mailbox-permission-grant

An `m365.audit` operation granting mailbox, recipient or folder permission.
Delegate access is durable, survives the owner's password reset, and is invisible
to the owner, which is why granting it is a standard persistence and collection
step once an attacker holds an administrative session (MITRE T1098.002).

**No data:** `OK`.

**Evaluator error:** `Error`.

**False positives:** shared mailboxes, resource calendars and migration tooling
all produce legitimate grants, often in bursts.

**Remediation:** identify the grantee and the mailbox from the record, then
confirm with the mailbox owner rather than with whoever made the change. Panel:
**M365 → Unified audit — which user, which operation**. This is a separate rule
from the inbox-rule detection on purpose even though both key on `operation`: an
inbox rule is usually the owner's own doing and noisy, a delegation grant is an
administrative act and rarer, and one threshold cannot serve both base rates.

### g2o-detect-identity-risk-detection

An `entra.risk_detection` record at medium or high level whose `risk_event_type`
is one of the detections worth waking up for. This stream answers the question
the sign-in stream cannot: **why** a sign-in was risky. `entra.signin` carries
`risk_state`, so a rule on it can say a session was at risk; `risk_event_type`
exists only here and names the detection — impossible travel, an unfamiliar
sign-in property, an anonymised or known-malicious address, credentials found in
a public dump, a password-spray victim.

Impossible travel is worth calling out: it is a correlation Microsoft has already
computed. Expressing it over raw sign-ins would need a join between two records
and a distance calculation, and **Loki cannot join** — so reading Microsoft's
verdict is not a shortcut, it is the only correct way to have this detection at
all.

**No data:** `OK`. On a tenant without Entra ID P2 this is permanent: the
endpoint answers with an empty collection, which reads exactly like a clean
tenant.

**Evaluator error:** `Error`.

**False positives:** a corporate VPN or a mobile carrier NAT can produce
travel-shaped and anonymised-address detections for an entirely ordinary user.
`leakedCredentials` on a large tenant can arrive at a rate no responder can work.

**Remediation:** read `mitre_techniques`, `ip_address`, the location fields and
`user_principal_name`, then confirm with the user before forcing a reset. Panel:
**Entra → Risk detections by event type and level**. Before enabling, run
[Which risk detection types does Identity Protection raise here](../hunting.md): the
event-type list in the rule is Microsoft's published set, not one measured on this
project's wire.

### g2o-detect-workload-identity-risk

An `entra.service_principal_risk_detection` at medium or high level whose
`risk_state` is `atRisk` or `confirmedCompromised`. The workload-identity half of
Identity Protection, and the half that gets watched least: a service principal
has no MFA to fall back on, its credential usually lives in a pipeline variable
rather than a vault, and nothing prompts a human when it is used from somewhere
new.

**No data:** `OK`, and normally permanent — this stream is silent on a healthy
tenant, which makes a threshold of zero plausible and unmeasured at the same
time. The endpoint returns real detections even without Workload Identities
Premium (live-measured), so silence here is genuinely an absence of detections
rather than a licence wall.

**Evaluator error:** `Error`.

**False positives:** a credential legitimately reused from a new build agent or a
relocated runner can score anomalous.

**Remediation:** read `service_principal_name`, `app_id` and `risk_detail`, then
**rotate the credential** rather than only dismissing the detection — dismissing
clears the flag and leaves the secret. Panel:
**Entra → Risky service principals total** (the gauge one layer up: it counts how
many identities are risky now, where this rule fires on the detection that made
one of them risky). This is the portable counterpart to pinning each automation
identity to its own expected source address, which cannot ship here because those
values are tenant infrastructure — see the pattern at the end of this page.

### g2o-detect-legacy-auth-signin

An `entra.signin` whose `client_app_used` is a legacy protocol. Legacy protocols
cannot present an MFA challenge, so a sign-in that succeeds over one has bypassed
multi-factor authentication whatever the Conditional Access policy says. Password
spraying targets them for exactly that reason.

This is deliberately separate from `g2o-detect-interactive-signin-anomaly`: that
rule asks whether a sign-in looked suspicious, this one asks whether a channel
exists that cannot be challenged — worth knowing even when every sign-in on it is
legitimate.

**No data:** `OK`. On a tenant that already blocks legacy authentication, silence
is the correct and permanent answer.

**Evaluator error:** `Error`.

**False positives:** a tenant with legitimate legacy clients still in migration
fires this continuously. That is not a wrong measurement, but it belongs on a
dashboard rather than a pager until the migration finishes.

**Remediation:** identify the account and client, then block the protocol with a
Conditional Access policy rather than chasing individual sign-ins. Panel:
**Entra → Top failing sign-in sources (country, client app)**. Before enabling,
run [Which client apps sign in, and how much legacy protocol is left](../hunting.md)
— the client names in the rule are Microsoft's spellings and have not been
measured here. Add `status_error_code=`0`` to narrow to *successful* legacy
sign-ins, which is the smaller and more urgent set.

### g2o-detect-mail-remediation-failed

A `defender.email_post_delivery` record whose `action_result` is present and is
not a success: Defender tried to remove a message it had already delivered, and
the removal did not land. Zero-hour auto purge exists because a message can be
reclassified as malicious after delivery — when the purge fails, the message is
still sitting in an inbox Microsoft has already decided is dangerous, and nothing
else tells you so. The alert says the threat was found; this record says whether
it was actually removed.

**No data:** `OK`. Also the state when Defender advanced-hunting blob ingest is
not configured, in which case the stream is absent rather than clean.

**Evaluator error:** `Error`.

**False positives:** a transient failure that a later retry resolves looks
identical to a permanent one on a single record.

**Remediation:** read `action_type`, `action_trigger` and
`recipient_email_address`, then remove the message by hand and confirm it is gone.
Panel: **Defender → Quarantine held messages total** — the nearest surface, since
both are about messages Defender has already judged malicious and acted on.

The filter has two terms for a reason worth remembering: a negative label filter
also matches a record carrying no `action_result` at all, because LogQL treats a
missing structured-metadata key as the empty string. The presence term
``action_result=~`.+``` is what keeps this from firing on every remediation
record. Run
[Which post-delivery mail remediations succeed](../hunting.md) to see which values
your tenant emits before enabling.

## A pattern worth knowing that is not shipped

A **workload identity that legitimately signs in from exactly one place** — a CI
runner, a scheduled sync job, a self-hosted integration — makes an
almost-zero-false-positive detection, and it cannot be shipped because it is
specific by nature:

```logql
{service_name="graph2otel"} | event_name=`entra.signin` | app_id=`<your application id>` | status_error_code!=`0`
```

plus an equivalent term for source address, ORed together and alerted above zero.

Detecting rather than preventing is a licensing decision worth stating plainly:
Conditional Access IP-locking for workload identities requires Microsoft Entra
Workload ID, a separate paid add-on. Without it there is no way to *stop* a
leaked service-principal credential being used from anywhere, but the sign-in
stream lets you *notice* within one evaluation interval at no extra licence cost.
Build one rule per such application.