Failure Harness¶

The failure harness runs a locally built exporter against a locally built HTTPS replay origin. It never calls Meraki. A run fails closed unless tests/harness/corpus/manifest.json exists and every fixture has a matching SHA-256 and LIVE-VERIFIED provenance. Synthetic unit-test metadata is explicitly SHAPE-ASSUMED and cannot drive a run.

make failure-harness-validate
make failure-harness-run MODE=baseline
uv run python -m tests.harness.runner --build-exporter run --all-modes
uv run python -m tests.harness.runner --build-exporter --target-operation getOrganizationDevices run --mode slow_valid
uv run python -m tests.harness.runner --build-exporter observe-duration
uv run python -m tests.harness.runner --build-exporter observe-shutdown

The runner builds the proxy from the local exporter tag with pulls disabled, records the resolved exporter image ID as a provenance label, and then runs both images by their local sha256: IDs. It creates a fresh CA/certificate with replay-origin in its SAN, copies the verified corpus into a temporary runtime directory, invokes Compose, probes /health and /metrics, retains the raw metrics response plus redacted plans, logs, journals, probes (when available), and aggregate evidence, and always runs compose down --volumes. If that shutdown times out, exits nonzero, or leaves exact-project resources behind, the runner force-removes only containers and networks bearing the exact run's Compose-project label, verifies that none remain, and records why the fallback was needed in a redacted teardown-<mode>.json artifact and aggregate evidence. This retention also happens when a later mode fails, so completed modes and the failed mode's partial evidence are not lost. Compose cannot pull images, uses an internal network, publishes no host ports, and disables Watchtower on both services. Health and metrics probes run inside the exporter container and their output is retained as host-side evidence.

Modes are baseline, trusted_tls, unauthorized, forbidden, both seconds and HTTP-date 429 Retry-After variants, timeout, stall, tls_failure, html, slow_valid, real TCP reset, and dns_failure. Each emits a monotonic-timestamped journal decision. A target SDK operation can be selected with --target-operation; verified non-target fixtures continue normally, while an unrecorded or mismatched request fails closed. slow_valid additionally requires a two-second delay and a subsequent matched verified fixture for the faulted route. stall requires a barrier-entry and a recorded deterministic hold before it passes. reset requires the proxy abort record and an exporter-side transport failure log. The 429 modes retain the response-sent Retry-After header in the redacted journal: seconds mode is exactly 2, while HTTP-date mode must be an HTTP-date value. Timeout/stall requests enter a deterministic barrier and are released only when the origin process is torn down.

Blocked SDK worker shutdown observation¶

observe-shutdown is a distinct native replay command, not a normal fault-mode run. It starts the timeout replay with the SDK executor constrained to one worker, selects one LIVE-VERIFIED corpus operation, and waits for the proxy's barrier:entered journal record. It resolves the exporter container by its exact Compose project and service labels, records wall-clock and monotonic times, and sends SIGTERM directly to PID1 with docker kill --signal TERM. PID1 must remain running while the barrier is closed. The runner then writes HARNESS_BARRIER_RELEASE without stopping the origin, uses bounded docker wait, and requires exit code zero within the chart's 150-second default grace.

shutdown-observation.json retains redacted raw journal and PID1 logs, parsed barrier evidence, timestamps, image and manifest provenance, exit timing, and lifecycle markers. It fails closed unless the journal proves request:entered < TERM < barrier:released and the structured lifecycle order is OTel metrics, DNS executor, SDK executor, SDK session/client close, serving executor, then Shutdown complete (tracing, OTel logging, and data-log markers may appear between OTel and DNS in their documented order). A failure still retains the fullest redacted partial candidate. Separate bounded teardown-shutdown-observation.json evidence proves no resources with that exact project label remain. The journal records the origin container's monotonic clock while TERM timing is host-monotonic, so causal order uses the journal line boundary captured immediately before TERM; the artifact retains both clocks independently and never compares their epochs.

DeviceCollector duration observation¶

observe-duration is a separate native replay command, not a fault mode. It starts a clean baseline replay, requires every manifest route to have a verified-fixture match and HTTP 200 response, and requires non-empty organization, network, device, and availability cache-population evidence. It then takes two idle snapshots at least 250 ms apart. Their histogram, collector status, device_availability scheduler state, journal length, and exporter-log length must be identical. The disposable Compose environment alone enables JSON DEBUG logging so the proof can inspect structured inventory-cache events; it does not alter production logging.

The runner records wall-clock and monotonic observation boundaries and captures raw /metrics plus authenticated /status?format=json. It then waits for the next natural scheduler wrapper invocation; it deliberately does not use the manual force endpoint, because force bypasses the selected profile and would call product-family routes absent from the four-route corpus. The disposable Compose environment supplies a fixed explicitly non-secret token solely for the protected status request.

It writes redacted duration-observation.json with the pre/post raw responses, histogram count and sum, total_runs, total_successes, total_failures, is_running, journal boundary, elapsed time, and calculated mean. It also retains the image IDs, manifest provenance, raw and parsed journal/log boundaries, parsed scheduler state, corpus-backed product-series evidence, and all deltas. If a gate fails after startup, the fullest partial candidate is retained with its stage and redacted error; teardown evidence remains a separate artifact.

Native acceptance is deliberately exact: one histogram observation, one run, one success, no failures, an advancing collector last_success_time, a positive sum, and a positive mean no greater than the observation elapsed time. The wrapper must use warm caches: structured post-boundary logs need finite non-negative cache ages and hits for organizations, networks, and devices, with no miss, update, or invalidation. It must log non-empty device processing while availability remains not due. The device_availability attempt and success state must therefore remain unchanged. This establishes that the histogram measures every successful top-level collector wrapper, including cached processing cycles where no endpoint group is due; it is not an API-latency histogram. The journal suffix must be exactly empty: any post-boundary origin request fails the proof. Teardown remains bounded and must leave no resources bearing the exact Compose-project label.

This proves only DeviceCollector; it does not exercise ClientsCollector. The command fails closed if a newly due endpoint group reaches a route that is absent from the LIVE-VERIFIED corpus. Capture and sanitise that route under an explicit read-only evidence grant; never relax the journal gate or invent a response merely to make the observation pass.

The GitHub Actions workflow is manual-only. In Actions → Failure harness → Run workflow, provide all to exercise every mode, or provide one of the mode names above to run only that replay. Its redacted .failure-harness-artifacts upload is attempted even when the selected run fails.

Refreshing the corpus¶

Only a separately authorised capture may refresh it. Capture into an untracked temporary directory; never commit raw responses, keys, certificates, addresses, contact details, or credentials. Run sanitize_capture_set once for the complete required operation set so its shared placeholder state preserves organization/network/device joins:

sanitize_capture_set({
    "getOrganization": organization,
    "getOrganizationNetworks": networks,
    "getOrganizationDevices": devices,
    "getOrganizationDevicesAvailabilities": availabilities,
})

It rejects missing required operations and credential fields, replaces MACs with locally administered stable values and IPs with RFC 5737/RFC 3849 values, and redacts location/user-controlled strings. Review output, calculate fixture digests, record the capture provenance, then run make failure-harness-validate before committing only sanitized JSON.