H100 HGX DCGM telemetry¶

The dcgm integration emits the pinned dcgm-exporter 4.8.4 / DCGM 4.6.1 profile for H100 SXM HGX pools with driver 595.91.07. Select a pool explicitly with gpu_pools; unsupported products, drivers and framebuffer declarations fail closed. The signal contract owns field names, types, units, health metadata and profile provenance.

Configuration¶

Declare the pool's shared operating.constraints: exact driver_version, requested SM/memory clocks, full_demand_power_w, power_fault_demand_w, and optional software_thermal_clock_cap / power_clock_cap objects (sm, memory, MHz). Caps are explicit synthetic controller setpoints. They are not a vendor response curve. See GPU clock constraints for strict causal thresholds.

The integration accepts profile: stock (default), dcp, or fault-detection, and mode: host (default) or kubernetes. It requires an explicit finite memory_temperature_offset_c; no HBM maximum is assumed. Optional engine fractions and PCIe/NVLink full-utilization traffic rates describe the operator's workload, not a device capacity. Their default zero means no declared activity of that kind.

The focused fixture shows a complete declared HGX pool and integration. It deliberately exercises a full-load training allocation and thermal/power incidents outside shipped blueprints. Do not use its stress inputs as production sizing defaults.

Profiles and identity¶

Stock and alternate DCP each have 24 baseline families and samples per GPU. The selected fault profile has 36 baseline families and 46 samples per GPU because the health family has eleven watches. Event-created Xid/clock totals expand the inventory. Raw Xid is absent before an event; retired-page rows omitted in the capture remain omitted. P2P, MIG, NVSwitch and Grace telemetry is not implemented.

Host identity is fixture-owned. Kubernetes mode adds the actual worker pod, namespace and container while it owns the GPU, then removes those dimensions on release. It does not emit pod_uid. Public runner tests verify the join against KSM pod output. Other GPU compute collection adapters still have separate admission requirements; the shipped AI Factory compute cluster remains emit: false.

Incidents and time¶

Shared gpu_cooling_fault and gpu_power_constraint drive the clock constraints; shared bus-loss and NVLink-error faults preserve topology targeting. Collection-only dcgm_ecc_error_growth and dcgm_health_failure model the documented ECC/health observations. gpu_xid_burst uses the configured sourced Xid code and declared burst size. Modeled Xid codes are 48, 63, 64 and 74; GPU-wide fatal code 95 is not admitted by the nonfatal burst scenario. See incidents for the observables.

The default collection interval is 5 seconds and event window is 300 seconds, matching the retained capture; metric emission stays at one minute. Duration and energy counters integrate the previous observed synthetic state, starting at zero on the first observation. Window counts and rising-edge totals are different: startup-active clocks do not create an edge, and expiring a window never clears a cumulative total. For pinned exporter 4.8.4, an empty source window has a zero COUNT row, while a window containing valid zero clock masks has no COUNT row. The model preserves that distinction after its first observed clock reason. A generator restart starts a new model run; driver-reload and exporter-only restart scenarios are not modeled.

The baseline operating reference remains 87 C. Explicit fault scenarios may exceed the 89 C slowdown threshold but must stay strictly below 95 C shutdown. Equality at 89 C and a cap alone do not activate thermal limiting; power requires demand strictly above the configured budget plus a reducing cap. Both may coexist.