Checkpoint foundations¶

This increment adds protocol acknowledgement evidence, loss-aware flush barriers and an optional filesystem persistence primitive. It does not enable counter restoration, durable delivery queues, takeover replay or historical backfill. Generator restart still starts a new cumulative-state run. No checkpoint environment variables, chart volumes or standalone Docker mount requirements are introduced.

Existing Kubernetes ConfigMaps continue to hold small control-state documents. They are not repurposed for per-series state or serialized telemetry payloads. The selected future checkpoint direction is an operator-owned shared filesystem, enabled only after runtime state coverage, transaction boundaries and actual shared-mount semantics are verified.

Protocol acknowledgement¶

GET /control/status includes last_acknowledgement on each sink-history entry. The object contains a closed status, unit, expected, accepted and rejected count. No receiver diagnostic text is exposed. Counts describe one final push attempt, not cumulative unique backend ingestion. last_success_ms and lane readiness retain their existing transport meaning; neither is a checkpoint watermark.

OTLP traces, metrics and logs parse bounded protobuf or JSON export responses. A standard HTTP 200 full-success response acknowledges the request. Partial rejection is an error and is never retried; warnings with zero rejected items remain successful. Malformed, negative, oversized or over-count responses cannot become accepted receipts. JSON unknown fields are ignored per the protocol, but ambiguous duplicate fields are rejected. Response reads are bounded to 4 MiB. Legacy non-200 2xx transport behavior remains compatible while acknowledgement is unverified or deferred. Units are spans, data points or log records.

Remote Write v2 checks the written sample, histogram and exemplar headers against the actual encoded request. Its unit is rw2_elements, the sum of those three categories, which can differ from the existing logical-series last_items count. Missing categories can mean zero on non-202 responses when written headers are present. A legacy success with no written headers stays transport-successful but explicitly unverified. HTTP 202 with missing or incomplete counts is deferred, not rejected and not blindly retried. A contradictory 2xx written count fails the push without retrying an already partially accepted request.

These are receiver-acceptance facts, not final Grafana ingestion/readback proof. Other sink lanes remain unverified by this classifier. Backend queries remain necessary when validating a deployment. Sources: OTLP response semantics and Remote Write v2 written-count semantics.

Flush barriers¶

A delivery queue retains the first failure from background sends until the next explicit Flush barrier reports it. A later successful batch cannot erase that failure. The next reporting interval starts after that barrier, and concurrent barriers serialize with a context-bounded wait. Producers must still be quiescent while flushing.

This change does not add a WAL or resend discarded batches. Queue dropped-item accounting counts items removed after failed sends; it can conservatively exceed known remote rejection when a batch was partially accepted or its outcome was ambiguous. Protocol counts are kept separately. DrainJoined proves sender termination, not successful remote acceptance.

Optional filesystem primitive¶

internal/checkpoint.FileStore is a developer-facing building block, not a configured runtime backend. It implements a small pluggable store interface and stores opaque caller-versioned bytes in immutable SHA-256-named payload files and publishes a checksummed manifest through fsynced atomic rename. Files are mode 0600 in a dedicated directory created as mode 0700; existing directories must not be group/world writable. Existing unbound data, unsupported format, destination or configuration mismatch, corrupt data, and symlink payloads fail closed.

A binding identifies deployment, destination/configuration fingerprints and payload format. It contains no credentials. Acquiring an epoch preserves the prior payload, but an initialized manifest without a payload is not a restored counter snapshot. Read.Exists=false means an observably empty directory; it does not prove that this logical deployment has never had a checkpoint. Runtime enrollment must distinguish an explicitly authorized first run from a lost/replaced volume. Read failures never become empty state.

Mutations require the existing HA authority gate. A nil-gate handle opens only existing storage and cannot create a directory, epoch or payload; AlwaysLeader is only appropriate for explicitly single-writer non-HA use. A store instance acquires one epoch. Both epoch and revision participate in CAS, and an old instance cannot adopt a newer epoch merely by re-reading its manifest. The actual lease-backed gate must authorize epoch acquisition and each write; an incremented integer alone is not leadership authority.

Linux and macOS use an exclusive filesystem lock. Its wait is bounded by the caller's deadline and an internal five-second ceiling. Filesystem syscalls themselves are not interruptible by that context: the actual shared provider must support reliable locking, hard links, atomic rename and file/directory fsync, and must be validated with the runtime crash-fence contract before multi-node HA is admitted. Unsupported platforms reject the backend. Local subprocess proofs are not proof of a Kubernetes RWX provider under network failure.

Explicit limits bound payload bytes, retained unique-inode file content and directory entries. They are not a filesystem-block usage estimate. There is no automatic pruning or garbage collection. Capacity failures preserve the last published manifest. Failed or cancelled publications retain temporary files for inspection and count against capacity. An interrupted first epoch acquisition leaves unbound data and requires explicit operator recovery; automatic reinitialization is refused. There is no automatic cleanup on cancellation; successful immutable publication removes only the byte-identical temporary hard link. An I/O failure after publication may be ambiguous: re-read the manifest rather than assuming rollback or blindly retrying.

Remaining runtime work¶

Every selected emitter and auxiliary state path still needs explicit versioned snapshot/restore support. Exact serialized payloads, post-tick state and scheduler positions must commit before sending, with durable per-fragment outcomes and legitimate leader epochs. Replaying from old randomized ticks is not equivalent to replaying committed bytes. Partial rejection must become an explicit gap, not a whole-request retry. Enrollment evidence outside a potentially lost payload mount is still required before runtime initialization is admitted. Unknown/deferred acceptance cannot advance a durable watermark.

Full takeover, timestamp-bounded backfill, retention/compaction and out-of-order acceptance on the selected live stack remain separate acceptance work. No existing deployment is migrated or silently opted into these foundations.