Blob ingest: reading Azure Storage instead of Graph¶

Some signals have no Microsoft Graph read endpoint at all — Graph cannot see them, no matter how the poller is tuned. They exist only as Azure Monitor diagnostic-settings output. graph2otel reads them from an Azure Storage account.

This is the one place graph2otel reads from outside Graph, and it is opt-in: set a tenant's blob_ingest.account_url and the blob collectors register; leave it unset (the default) and they do not exist. A deployment with no storage account is unaffected.

The Azure SDK lives behind one interface (blobpipeline.Source, implemented in internal/blobpipeline/azblob_adapter.go). No collector imports an Azure type.

What it is for¶

The sign-in collectors¶

The three shipped sign-in collectors emit entra.signin — the same event name and the same attributes as the polled streams, because the diagnostic-settings properties object is the Graph signIn resource (verified field-for-field against live samples of all four sign-in categories). They share one mapper with the polled path, so the two sources are indistinguishable downstream and an attribute added for one is automatically right for the other.

MicrosoftServicePrincipalSignInLogs is not a duplicate of ServicePrincipalSignInLogs. Live-verified 2026-07-16: every sampled record in the former was owned by Microsoft's own tenant (f8cdef31-…) and every record in the latter by the local tenant, with zero sign-in ids overlapping. The two partition cleanly by app ownership, which is why the polled entra.signins.service_principal — restricted to your own service principals — can never surface the first-party half.

The .blob suffix marks a collector whose polled twin exists; the two are separate config keys with separate intervals. They are not Experimental (configuring blob_ingest.account_url is already the opt-in, and unlike the polled twins these are v1.0-stable sources), and they declare Entra ID P1 so a Free tenant gets a stated skip rather than a silent empty container.

Running a .blob collector and its polled twin ships each sign-in twice. The polled twins are Experimental and therefore off by default, so this needs a deliberate act; if you do both, dedupe downstream on the id attribute, which is identical across the two sources.

Blob is not a general replacement for polling. Metrics want freshness and blob is floored at ~4 minutes; logs want completeness and blob has no throttle ceiling. But a collector's source is graph XOR blob, never both — #131 examined a "dual-ship" mode (Graph for metrics, blob for logs) and closed rejecting it: the log-shaped collectors emit zero metrics, so dual-ship is empty for every signal it would list, and mutual exclusion is enforced in config (#144's ConflictsWith). The one genuinely dual-capable signal (intune.devices) is tracked in #132.

What it costs¶

~£12.90/month on a small tenant with the Entra categories and the Defender XDR advanced-hunting tables streaming. There is no standing charge — that was the point.

Read straight off the bill rather than modelled: Cost Management billed the account £1.53 month-to-date for 2026-07-16..21 (the account is new; billing starts on the 16^th), and the last full day cost £0.4242 → £12.91/month at that run-rate. [live-measured 2026-07-22, #228]

The bill is almost entirely write operations, not storage: 3.0 GB resident costs pennies — in fact £0.00, it lands inside the free-tier grant — while the AppendBlock calls that put it there are ~£11.45/month. Listing is billed at the write rate and is not a rounding error: ~9,400 ListBlobs/day is ~£1.30/month, about 10% of the total. Storage is ~£0.0145/GiB-mo, writes £0.0447/10K (confirmed exactly against the bill: 32.7743 × £0.0447 = £1.4650), reads ~£0.0036/10K, and blob deletes are free.

Do not assume a mean append size — measure it. The 7.3 KB per append figure this doc used to quote was measured on the Entra categories alone ([live-measured 2026-07-17, #137]). Once the Defender advanced-hunting tables started streaming, the blended account-wide mean measured 35,590 B on 2026-07-21 (2,994.5 MB Ingress over 84,139 AppendBlock ops) — 4.9× higher, and still moving: 10.5 KB on 07-16, 18.2 KB on 07-17, 25.5 KB on 07-18, 35.6 KB on 07-21 as more tables came online. Defender batches far larger records than Entra does. scripts/storage-report.py hardcoded the 7.3 KB constant and back-solved the op count from resident bytes, which over-stated the bill 4.7× (£59.39 modelled vs £12.91 billed) until #228; it now reads the Transactions metric filtered to ApiName eq 'AppendBlock' — an exact count of billable appends, since an append blob supports no other write — and calibrates the mean from Ingress / AppendBlock each run. [live-measured 2026-07-22, #228]

The previous ~£3.07/month headline was Entra-only scope, not a wrong measurement: it covered MicrosoftGraphActivityLogs plus the service-principal sign-in categories at ~5.0 GB/month, before the Defender XDR tables were switched on. The increase to ~£12.90 is mostly added scope (ingress is now ~2.1 GB/day), not drift on the original categories. Compare like for like before reading it as a regression.

The Azure free-account 12-month grant is immaterial — do not model it. It expires 2026-09-11 and every meter that matters is already past its allowance (write ops at 3,377%). Total value at real volume: 10K free writes £0.045 + 20K free list £0.089 + 20K free reads £0.007 + 5 GB stored £0.044 + 15 GB egress ~£0.16 ≈ £0.35/month, under 3% of the bill. Expiry moves the number by pennies. [live-measured 2026-07-22, #228]

Cost scales with graph2otel's own collector count, not just tenant size. Every Graph call a collector makes writes a MicrosoftGraphActivityLogs record, which the blob path then pays to ingest — so graph2otel is 59.9% of its own MGAL volume (14,404 of 24,048 records carry the poller's own appId; top URIs /groups/$count, /devices/$count, /servicePrincipals). Enabling a Graph-polled collector bills twice: once for the poll, once for ingesting the MGAL record it created, so a 60-collector tenant costs more than a 20-collector tenant of the same size. The opt-in self-exhaust exclusion (#154) is the lever; the self-share also means any "who is calling Graph in this tenant" reading of MGAL is ~60% the observer. A second, unrelated concentration distorts the bill the same way: one Defender TVM scan-agent SP is 96.4% of ServicePrincipalSignInLogs.

Earlier headline figures here were each honest when taken and each superseded. £0.85/month (2026-07-16) was measured mid-backfill, before the volume drivers above were in play; £3.07 (2026-07-17) was Entra-only scope. Both were modelled op counts. The current £12.90 is read off Cost Management. The lesson that keeps recurring: don't guess at the batching; read the counter. Transactions filtered to ApiName eq 'AppendBlock' is a direct count of billable append operations, because an append blob supports no other write.

Log Analytics and Event Hub were both evaluated against this and closed (#89). Neither correction reopens either — the blob figure rose because scope and volume grew, and both alternatives are priced on that same volume. At Entra-only scope #89 ranked blob £3.07 < Event Hub £8.32 < Log Analytics £10.88 against live-queried uksouth prices. Blob is priced on write operations while LA is priced on GB ingested, so growing the volume hurts LA strictly harder: scaling #89's LA per-GB price to today's ~63 GB/month of ingress puts LA in the low hundreds of £/month against blob's £12.90. Event Hub's entire measured advantage was 12 seconds of latency, and the ~4-minute delay is Entra-side — upstream of where the transport forks — so a faster destination cannot buy back time already spent. Full evaluation: #89. [blob live-measured 2026-07-22 (#228); LA/EH re-rank is #89's docs-priced meters scaled by measured volume, NOT re-quoted live — re-quote before citing a specific LA number]

Excluding graph2otel's own exhaust (exclude_self)¶

Because ~60% of MicrosoftGraphActivityLogs is graph2otel calling Graph (above), there is an opt-in filter that drops the poller's own records before they are emitted: exclude_self, default off. It is a tenant-level key (a sibling of client_id, not under blob_ingest) because the same "self" spans transports — it filters both the blob feed and the Graph-polled service-principal sign-in stream with one flag (#176):

tenants:
  - tenant_id: "11111111-1111-1111-1111-111111111111"
    client_id: "<optional expected poller app id>"
    exclude_self: true
    blob_ingest:
      account_url: "https://myaccount.blob.core.windows.net"

• Self-only, by proved appId. A record is dropped if and only if its actor appId equals the non-empty appid proved from the Graph access token issued to that tenant-pinned credential. That is the poller's authenticated application identity (live-verified: the poller's application ID is exactly the MGAL appId, 14,404 records matched, #154). Any other appId — including Microsoft's own first-party service principals — always passes untouched.
• Per-tenant proof. One deployment polling many directories resolves the identity once from each token whose requested tenant and returned tid match that entry's hyphenated directory GUID, then shares the result across the Graph and blob paths. The ambient DefaultAzureCredential still selects one application identity for the process.
• Which transports it covers (live-measured 2026-07-19, #176):
• Blob categories that carry an appId — MicrosoftGraphActivityLogs (entra.graph_activity) and the service-principal sign-in categories. This is the material saving (~60% of MGAL). Categories with no appId (e.g. AuditLogs) are never filtered.
• The Graph-polled entra.signins.service_principal stream — the poller's token acquisitions are service-principal sign-ins, so its appId appears here too. The self-share is small (~1.1% of that stream, vs 59.9% on the blob MGAL feed), so this is filtered for completeness rather than volume; the one tenant flag simply covers it too.
• m365.activity is out of scope — the poller does no Exchange/SharePoint operations, so its exhaust in the default UAL subscription is 0% (0/131 SharePoint, 0/119 Exchange records carried the poller; only the non-default Audit.AzureActiveDirectory had ~1.2%). No m365.activity self filter is built.
• Loud, never silent. Every dropped record increments a self-obs counter (_total on the Prometheus side), labeled collector: graph2otel.blob.self_excluded on the blob path, graph2otel.logpipeline.self_excluded on the Graph-polled path — so a quieter dashboard is visible and alertable rather than looking like breakage. On the blob path the bytes are still consumed, so the byte-offset cursor advances exactly as for any other dropped record.
• Resolving "self". client_id in YAML is an optional non-secret consistency assertion only. It never selects a credential or controls this comparison. If it disagrees with the token's proved appid, graph2otel warns once and uses the authenticated ID; a record matching only the stale configured value still passes. If token acquisition or claim decoding cannot prove a non-empty appid, filtering fails open: every record is retained and startup emits one bounded warning for the tenant.

Poll cadence — blob_ingest.interval (#425)¶

Listing is the one part of the bill graph2otel itself controls. Per the measured breakdown above, ~9,400 ListBlobs/day is ~£1.30/month, about 10% of the total — billed at the write rate, not the read rate. That is a straight multiple of the poll cadence: halving the tick doubles it.

The default is 15 minutes, and turning it up costs you nothing observable. The freshness floor is Azure-side: Azure Monitor writes hour-partitioned blobs and appends to the current hour on its own cadence, and blob-derived records were measured at 3.3–7.0 days of event-time lag (#297). Against days, the difference between a 5- and a 15-minute list does not show up in the data — only on the invoice.

tenants:
  - tenant_id: "..."
    blob_ingest:
      account_url: "https://myaccount.blob.core.windows.net"
      interval: 30m            # optional; default 15m

Precedence, loosest to tightest:

The two collectors this does NOT slow down, and why¶

entra.graph_activity and the entra.signins.* blob streams are the only blob collectors that emit metrics as well as logs. They are pinned to blobpipeline.MetricDerivingInterval (5m) and blob_ingest.interval does not move them.

Their tick is an input to the metric-recency gate (#128). Derive runs only for records whose event time is inside blob_ingest.metric_recency_window (default 20m); anything older takes the log path only, so a backfilled event is never credited to "now" under cumulative temporality. A record appended just after one tick is not read until the next, so its age at the gate is up to tick + Azure write latency. Azure's steady-state blob latency is ~5m, so:

At 15m the tail of every tick's batch would stop counting toward metrics. Silently — the logs would still be complete, so nothing would look broken while those two metric streams undercounted. Raising the window instead was considered and rejected: it re-admits older backfill into cumulative counters, which is the precise bug #128 exists to prevent.

Keeping two of ~35 collectors fast costs a small fraction of the listing saving. If you do want them slower, use the per-collector interval: override — explicit enough to be a deliberate act rather than a side effect of a cost tweak — and raise metric_recency_window to match, or accept the undercount knowingly.

TestBlobIntervalMatchesWhetherTheCollectorDerivesMetrics gates the pairing off DerivesMetrics() rather than a list of collector names, so adding Derive to a log-only collector fails the build instead of quietly gating its new metrics away.

Setup¶

1. Create a storage account (StorageV2, Hot, LRS is fine; disable public blob access) with a lifecycle rule deleting blobs after N days. Retention past your log backend's own reject-old-samples age is pointless — Loki would refuse the records anyway — so 7 days is a reasonable default.
2. Create the diagnostic settings pointing at it. Entra categories live under the tenant-level microsoft.aadiam provider, Intune categories under microsoft.intune. These are two separate settings, not one.
3. Grant the DATA-plane role Storage Blob Data Reader to graph2otel's app registration, scoped to that account. See the traps below — this is not optional, and getting it wrong looks like success.
4. Set the account URL in config:

tenants:
  - tenant_id: "11111111-1111-1111-1111-111111111111"
    blob_ingest:
      account_url: "https://myaccount.blob.core.windows.net"

No credential goes in config: the process's ambient DefaultAzureCredential identity is reused, and the SDK requests the storage audience itself.

graph2otel is read-only on the account — it cannot write or delete a blob. Retention belongs entirely to the lifecycle rule; see "Why read-only" below.

Traps¶

Everything here was verified live against a real tenant on 2026-07-16. Several items contradict Microsoft's own documentation, which is why this list exists.

The blob layout is not what the docs say¶

DOCUMENTED:  resourceId=/tenants/<tid>/providers/microsoft.aadiam/y=/m=/d=/h=/m=00/PT1H.json
ACTUAL:      tenantId=<tid>/y=2026/m=07/d=16/h=13/m=00/PT1H.json

Every published Microsoft example is subscription-scoped. A tenant-level (microsoft.aadiam) resource uses tenantId=<guid>/ instead. Coding the listing prefix to the docs yields a collector that lists zero blobs and reports success forever.

The container is insights-logs-<category-lowercased>. Records are JSON Lines with CRLF terminators, in append blobs.

Data-plane RBAC fails in a way that looks like success¶

Owner grants blob container list/create — those are control-plane Actions. Reading blob content is a DataAction, and needs Storage Blob Data Reader. So an under-privileged identity lists blobs happily and 403s only on the read.

(The Event Hub variant is worse, if you ever go that way: without Azure Event Hubs Data Receiver the SDK returns 0 events with no error at all while the hub reports hundreds.)

The poller CAN read the microsoft.aadiam diagnostic settings (control plane)¶

live-measured 2026-07-23, #238: GET https://management.azure.com/providers/microsoft.aadiam/diagnosticSettings?api-version=2017-04-01 (ARM-audience token, the poller's own client-assertion cert) → 200, setting name graph2otel, 26 categories / 21 enabled. So reading the tenant-level microsoft.aadiam diagnostic setting is authorized by the poller's Entra roles, not by Azure RBAC — it needs no extra grant and does NOT violate the identity split. Corrects a #134 comment that claimed reading it "would require a different identity". The boundary is narrow and real: providers/microsoft.intune/diagnosticSettings and the storage account resource itself both 403 AuthorizationFailed for the same identity — so it is specifically microsoft.aadiam, not ARM generally. diagnosticSettingsCategories 400s; the setting object's own logs[] array is the only enumerable list. This makes a diagnostic-settings census gate possible (enabled-but-unread containers vs registered blob collectors) — tracked on #238.

A closed hour's blob keeps growing — and nothing tells you when it stops¶

Blobs are partitioned by event time, and on enablement Azure backfills history into those hour buckets progressively, oldest-first. While backfill is working on hour N, that blob grows regardless of how long ago hour N closed — an h=00 blob was observed still being appended to 13 hours later. Once backfill passes an hour, that hour freezes.

So a "this hour is complete" state does exist, but nothing signals when it is reached, and it is not derivable from the clock. Hence:

• The cursor is a byte offset per blob, never a timestamp watermark. A watermark cannot express this; an offset can, because append blobs never rewrite a byte.
• The consumer re-checks every blob it has seen on every tick, not just newer ones. A walk-forward-and-forget consumer silently loses every late-arriving record.
• This is affordable: the lifecycle rule bounds the set to ~168 blobs per category, an unchanged blob costs a size comparison rather than a read, and in steady state only one or two blobs actually grow per tick.

Backfill on enablement is good news, incidentally: turning the destination on recovers history rather than starting from zero.

Resolved (#137, [live-measured 2026-07-17, n=1 tenant, 19h window]): backfill on this tenant ended 2026-07-16T17:00Z; in steady state a bucket freezes ~2–8 minutes after its hour closes, and no closed bucket was observed growing across a ~19h window (16 settled hours, 6 categories). That is not proof it can never grow — a genuinely rare late record on a larger or more distributed tenant would not necessarily surface in 19h here — so the re-check-every-blob design stays: correct under both answers, and cheap. A settle-horizon optimisation (stop re-reading blobs older than N) is explicitly not built: below a 13h horizon it would have silently dropped real backfill data on this very tenant on 2026-07-16. Evidence class: cheap to reopen, not armored.

Azure delivers at-least-once: ~2.7% (MGAL) / ~4% (sign-ins) of records arrive more than once¶

Measured live on a clean backfill-free window (2026-07-17, [live-measured, #137]), and cross-checked against the backfill window — the two agreed within noise, so at-least-once re-delivery is a steady-state property of Azure's delivery, not a backfill artifact. The sign-in family clusters near 4%, higher than MGAL's 2.7%, and one event was observed delivered four times (an earlier measurement, on ~4–12× less data, saw a max of three and a flat ~2.3–2.8%):

AuditLogs showed 0/73 dupes across both windows, but n=73 is too small to tell exactly-once from a ~3% rate (expected ~2 dupes) — do not record it as exactly-once. A downstream dedupe must not assume at-most-two copies: multiplicity reaches ×4.

A re-delivered event is written as a separate line, usually into the same hour blob, with a byte-identical properties payload and a fresh envelope time. One h=04 blob carried the same sign-in at line 15 (envelope 04:09:50) and line 20 (envelope 04:16:16) — identical id, createdDateTime, correlationId, and uniqueTokenIdentifier, 6.4 minutes apart.

This is not a cursor bug and it is not fixable by the cursor. Both copies are real, distinct bytes; a byte-offset cursor consuming them exactly once is behaving correctly. Verified directly: across a cold start plus a restart, for all 1,035 emitted ids, the number of times graph2otel emitted a record exactly equalled the number of times Azure wrote it — no over-emission, no loss.

The consequence is real: blob-sourced collectors ship Azure's duplicates through to your backend. The polled path does not have this problem — logpipeline carries a seen-id set in its checkpoint and dedupes. blobpipeline has no equivalent, so ~2.7% (MGAL) / ~4% (sign-ins) of blob-sourced records are duplicates.

Decision (#138): dedupe downstream, not in the engine. Engine-side dedupe is the wrong trade here. It would need a seen-id set that (a) is unbounded — Azure re-delivers across the full 7-day retention with no natural window to bound it by (MGAL alone is ~150k rows/7d), unlike logpipeline's overlap window; (b) persists across restart, growing the byte-offset checkpoint without limit; and © is correct across hour blobs (re-deliveries cross blob boundaries, so per-blob dedup is insufficient). Against that, the engine is provably exact today — a bounded/lossy set risks dropping genuinely distinct events to catch a 2.7–4% duplication that the backend deduplicates for free and exactly. Every blob-sourced record already carries its identity attribute as structured metadata (id for sign-ins, request_id for Graph activity), the duplicates are byte-identical, and the rate is steady-state — so downstream dedupe costs graph2otel nothing, cannot go stale, and has no memory-bound failure mode. The recipe (LogQL for counts, store-side distinct for raw export, with the ×4 multiplicity caveat) is in signals.md.

Field types are inconsistent within a single record¶

• durationMs is a string ("497815") at the top level and an int (497815) inside properties — on the same record. Bind to properties.
• level is "Informational" on every MicrosoftGraphActivityLogs record, including the 500s (verified across a 335-record sample spanning 200/201/204/400/401/403/404/500). Deriving severity from it marks every server error INFO, permanently. Use properties.responseStatusCode.
• level is a numeric string ("4") on SignInLogs — so a severity mapper shared across categories would be wrong twice over. Map per category.
• resourceId casing differs per category (/TENANTS/…/MICROSOFT.AADIAM vs /tenants/…/Microsoft.aadiam). Never match on it.

Timestamp binding differs per category family — and only parsed instants tell the truth¶

The envelope time is the ingestion time. Whether it equals the event time depends on the category, and there are at least three patterns (live-verified 2026-07-16, #135):

Rules that fall out:

• Sign-in categories bind to properties.createdDateTime, with NO fallback to time — the gap is variable (0 of ~700 records matched), so a fallback silently backdates records by a random couple of minutes. MGAL and AuditLogs bind to time and are correct to. Copying a neighboring collector's timestamp rule onto a new category is the trap — verify per category.
• Compare timestamps as PARSED INSTANTS, never as strings. AuditLogs' time and properties.activityDateTime differ in serialization (7-digit fraction + Z vs 6-digit + +00:00) — a string comparison reports "never equal" and points at the wrong binding; as instants the delta is 0.000s on every record.
• The envelope is consistent within a category family, not across families. The three sign-in categories share one schema (zero type conflicts, additive optional fields only) and correctly share one mapper; MGAL vs sign-ins agree on nothing. Verify agreement against live samples before sharing a mapper.
• The blob properties object IS the Graph resource — verified field-for-field for all four sign-in categories (properties = the signIn resource; same id as the polled record) and for AuditLogs (properties.id byte-identical to the polled checkpoint's seen ids). Reuse the polled mapper; never write a second one. Two categories is not a universal rule though — check per category before assuming.
• Within the Defender advanced-hunting family the three clocks do not spread consistently per TABLE [live-measured 2026-07-23, #233]. Every Defender table binds to properties.Timestamp with no fallback, which stays correct — but do not build a shared assertion on how far the envelope clocks trail it. On EmailEvents' live record all three differ (Timestamp 14:08:56, _TimeReceivedBySvc 14:08:56 as serialized but distinct in the record, envelope time 14:11:24); on EmailPostDeliveryEvents' live record _TimeReceivedBySvc is the same instant as properties.Timestamp, and only the envelope time (88s later) differs. A test asserting "the event time differs from both envelope clocks" passes on one table and can only fail on the other. The spread is a per-table property, not a family invariant.
• Records are JSON Lines with CRLF terminators; drop the properties.__UDI_RequiredFields_* keys (Microsoft-internal plumbing).

An empty container is not evidence of a fault¶

Microsoft documents up to 24 hours before data appears on a newly configured destination. In practice first data landed in ~15 minutes, but do not "fix" a working diagnostic setting during the first day. Azure also creates the insights-logs-<category> containers on first write, not at setting-creation time, so a missing container means "nothing written yet", not "misconfigured".

Why read-only¶

graph2otel never deletes. The original design was read-and-delete, on the theory that removing consumed data controls cost. It does not: the saving is £0.002/month, and buying it would have required write access plus a "delete closed hours only" rule.

That rule would have destroyed live data — the h=00 blob above looked safely closed for 13 hours while Azure was still writing to it. Dropping delete was right for a reason nobody had identified when the decision was made (#89).

The property that falls out: nothing graph2otel does can destroy data it has not read. The only deleter is the lifecycle rule, on a clock you set.

Related¶

• #89 — the transport evaluation (Log Analytics vs Event Hub vs Storage), the live measurements, and every decision above with its reasoning.
• #131 — the dual-ship proposal (Graph for metrics, blob for logs), closed as rejected: source is graph XOR blob per collector, enforced by #144. The one real dual candidate is #132.
• #128 — deriving metrics from blob-sourced events, recency-gated so backfilled events cannot corrupt a cumulative counter.
• #106 — raw Defender hunting-table ingest. Storage is a supported destination for the Defender streaming API and Log Analytics is not, so it can share this transport — but it must verify Defender's own blob write cadence rather than inherit #89's latency numbers.