Alerting¶

Classify by what is installed¶

The most useful judgement in the whole deployment, and the one that decides whether anyone still reads the alerts in a month.

source_type distinguishes a package that is installed on disk from one merely declared in a lockfile. Combined with root_kind, that gives three tiers with genuinely different responses:

Tier 3 is the one people get wrong, and it is the bulk of the volume. A pnpm-lock.yaml vendored inside a Go module under ~/go/pkg/mod names packages that are not installed, will never be installed by anything on the machine, and come straight back if you delete them, because the file is part of a read-only module cache. Paging on those trains people to ignore the alert.

The declared-only list:

npm-lockfile   pnpm-lockfile   yarn-lockfile   bun-lockfile
rubygems-gemfile-lock          composer-lock
go-mod         go-sum          skill-lock      mcp-config

Is a lockfile-only match ever exploitable?¶

Worth answering explicitly, because it decides whether tier 3 is "ignore" or "ignore for now".

A malicious package declared inside a read-only module cache is not exploitable: nothing is installed, no node_modules exists, the tree is mode-0444, and it returns on the next dependency fetch. Removing it achieves nothing.

But a malicious release installed inside a vendored tree is real exposure. The distinction that matters is lockfile versus installed, not first-party versus third-party. Tier 3 is not "this came from someone else's code, therefore fine" - it is "nothing was installed, therefore nothing runs".

Verify the list against live data¶

sum by (source_type) (count_over_time({source="bumblebee", record_type="package"}
  | json source_type="source_type" [1h]))

Do not write this list from memory. Enumerate it from live data and re-check it after every scanner upgrade.

Guard the list, because it lives in three places¶

The same alternation ends up in your alert rules (what pages), your recording rules (what the metrics count), and your dashboard JSON (what a human reads). Those three cannot see each other.

When they drift, the same finding sits in different tiers depending on where you look, and nothing errors.

Ship a check that asserts they match, run it before deploying, and negative-test it - a consistency check that has never been seen to fail is indistinguishable from one that always passes.

Source: src/grafana/check-tier-consistency.py.

Better still, generate all three from one source. The check is the pragmatic version.

The seven rules¶

Three of those seven alert on the monitoring, not on findings. That ratio is not an accident: on this deployment the likeliest bad outcome is not a missed malicious package, it is a fleet that stopped looking and did not mention it.

Source: src/grafana/alert-rules.py.

Rules worth explaining¶

BumblebeeCatalogIneffective¶

The failure mode that looks exactly like good news. A host scanning with no usable catalog emits scan_summary records indistinguishable from a clean host - status: complete, timed_out: false, findings_emitted: 0 - so without this rule, total loss of detection reads as a quiet week.

min by (host) (last_over_time({source="bumblebee", record_type="catalog_health"}
  | json catalog_effective="catalog_effective" | unwrap catalog_effective [25h]))

BumblebeeScanTimingOut¶

A rate, gated on having enough scans for a rate to mean anything:

(sum by (host, profile) (count_over_time({...record_type="scan_summary"}
   | json timed_out="timed_out" | timed_out="true" [48h]))
 /
 sum by (host, profile) (count_over_time({...record_type="scan_summary"} [48h])))
and
(sum by (host, profile) (count_over_time({...record_type="scan_summary"} [48h])) >= 8)

A count threshold only catches a profile that never finishes at all. The measured baseline before the daemon split was ~0.23 - roughly a quarter of scans silently under-reporting - which a "more than 10 in 48h" rule never tripped.

The >= 8 floor stops a laptop that was mostly switched off from tripping this on two unlucky runs.

BumblebeeHealthTelemetryMissing¶

The companion rule that makes the other three trustworthy. Covered in silent failures; the short version is that any rule deriving series from a field is blind to hosts that do not emit it, and those hosts do not even show as NoData.

Two settings that matter on laptops¶

NoDataState: OK on every rule. These are laptops. A machine that is switched off, on a plane, or in a drawer must not page. This is also why the health-telemetry rule exists - with NoData suppressed everywhere, absence needs a rule whose presence condition catches it.

A lookback longer than the scan interval. With four-hourly scans, a 12h lookback keeps finding-based alerts steady instead of flapping once per cycle. The cost is that resolution after remediation takes up to 12h; say so in the alert description so nobody re-investigates a resolved alert that has not cleared yet.

Route on labels your policy tree actually matches¶

Check the live notification policy tree before choosing labels. A plausible-looking team=security label that matches no route falls through to the default receiver, which is usually an inbox nobody reads. Confirm the label you chose actually selects the route you meant.

Write descriptions for the person woken at 3am¶

The alert description is the only thing a responder has. Include: what the failure actually means, why nothing else went red, the likely causes in order of probability, the exact commands to run on the host, and the query that verifies a fix.

That is verbose for a dashboard and correct for a pager. The most valuable line in most of them is the one explaining why the rest of the system still looks healthy, because that is the fact that otherwise costs an hour.