Exposure catalogs¶

Findings come from matching installed packages against catalogs of known-malicious releases. Getting the catalogs onto the endpoint is the easy part. Getting a coherent set onto the endpoint is where this bites.

The failure: mixed schema versions emit nothing¶

Bumblebee's catalog loader requires every file in the directory to declare the same schema_version. Mix them and it exits 2 before scanning.

Live-proved on v0.1.2:

That third row is the dangerous one. There are no package records, no findings, and no scan_summary either. Every other failure in this deployment at least leaves a record saying the scan happened. This one leaves nothing, so a dashboard filtered to "scans that ran" just shows one fewer host - the same shape as a laptop that is switched off.

--findings-only with no catalog at all fails identically.

Why a fleet drifts into it¶

The release tarball ships its own threat_intel/ catalogs, matched to that binary. You will also want external catalogs on a faster refresh cycle, because the whole point is to learn about a malicious release quickly.

Those two move independently:

• Upstream bumps the bundled catalogs when it changes the schema, in lockstep with the binary.
• Your published catalogs are on their own cadence, at whatever schema you last generated.

So an upstream release that bumps threat_intel/ to a new schema turns "copy every valid catalog file into one directory" into an outage, on the next installer cycle, on every host at once. Nothing you did changed; the version you pinned changed underneath the assumption.

Coherent selection on the endpoint¶

Rather than trusting that the sources agree, group the candidates by schema_version and stage exactly one group:

for each candidate catalog file:
    parse it, read schema_version, count entries
    discard anything unparseable or unsupported

group by schema_version
rank groups: newest schema first, then most entries
stage group[rank] into catalog/<mode>_<profile>/ via atomic rename

Then prove it loads before anything depends on it:

"$BIN" scan --profile baseline --root "$PROBE_DIR" --max-duration 5s \
       --exposure-catalog "$STAGED" --output file --output-file /dev/null \
  || try_next_rank

A throwaway scan against an empty directory. It costs a few seconds and it is the difference between finding out now and finding out from an absence of findings later.

If every rank fails, run without a catalog and say so, rather than not running at all. Drop both the catalog argument and --findings-only, because --findings-only with no catalog is itself an exit 2.

Source: src/endpoint/catalog-select.py, with tests. The test worth reading is test_mixed_schemas_never_staged_together.

Emit catalog health yourself¶

The scanner cannot tell you it had nothing to look with. A catalog-less scan reports status: complete, timed_out: false, findings_emitted: 0 - which is character-for-character what a clean host reports.

So the wrapper emits its own record:

{
  "record_type": "catalog_health",
  "catalog_effective": 1,
  "catalog_files": 14,
  "catalog_entries": 33789,
  "catalog_schema_version": "0.1.0",
  "catalog_files_rejected": 0,
  "catalog_schema_conflict": false,
  "catalog_age_hours": 2.9,
  "catalog_age_by_source": { "osv": 2.9, "datadog": 5.5, "ghsa": 5.5 },
  "catalog_sources": 3,
  "fda": 1,
  "fda_probe_files": 1169
}

catalog_effective: 0 is the page-worthy one: detection is entirely off on that host while every other signal stays green. catalog_schema_conflict tells you why without an SSH session.

Per-source ages matter because sources fail independently. One stale feed among three is a partial loss of coverage that a single aggregate age would hide.

Staleness is its own alert¶

A catalog that stops refreshing is a blind detector: scans keep succeeding and keep reporting clean against a frozen list, and nothing else in the pipeline goes red.

Alert on max by (host) (catalog_age_hours) > 18 when your generation cadence is 4-hourly. Two causes dominate:

1. Your catalog generation stopped. If it runs on GitHub Actions, note that scheduled workflows are auto-disabled after 60 days of repository inactivity - a catalog repo that is working correctly is exactly the repo nobody commits to.
2. Endpoints are failing to fetch and falling back to their last-good cache. Which is the correct behaviour, and is why it is silent.

Fetching, and last-good caching¶

Refresh before each scan, but never let a fetch failure stop a scan:

• Download to a temp file, validate it parses, then atomically rename into place.
• On any failure, keep the previous file and carry on. A network blip must not disable detection.
• Track age per source and emit it, so "we have been running on cache for six hours" is visible.

The corollary: a host can run indefinitely on a stale cache and look perfectly healthy. The staleness alert above is the only thing that catches it.

Check that you deploy every catalog you generate¶

Cross-check the asset list your CI publishes against the list your endpoint fetches. There is no error when they disagree - the scan runs happily against whatever subset arrived.

The catalogs most easily missed are the supplementary ones: editor-extension advisories, and malware advisories whose identifiers the primary source does not alias.