Log shipping¶

The exporter can ship structured OPNsense events (firewall log lines, IDS alerts, audit entries, and similar) to Loki, separately from the metrics it exposes at /metrics. This is opt-in and off by default: it runs only when --logs.enabled is set.

Log shipping is a long-lived background pipeline, not a scrape-time collector. Registered sources poll OPNsense event APIs on their own cadence, records pass through a bounded in-memory queue, and an emitter ships batches to the configured sink. It is independent of OTLP metrics export (--otlp.enabled): metrics and logs are gated by separate flags and neither turns the other on.

High-cardinality event data (IP addresses, ports, Suricata SIDs, domains) is shipped as log body and Loki structured metadata - never as a metric and never as a Loki label. The only labels are the resource identity, plus opnsense.source and opnsense.subsystem if you promote them (see Loki label model).

The pipeline is implemented in internal/logship/ on GitHub; parsers live one file per program, so adding a parser is a self-contained contribution.

Sinks¶

Select the sink with --logs.sink:

• otlp (default) - ships over OTLP logs. One sink covers both the Grafana Cloud OTLP gateway (which routes /v1/logs to Loki) and a self-hosted Loki 3.x native OTLP endpoint. It reuses the exporter's existing --otlp.* transport family (endpoint, protocol, headers, TLS/mTLS, and the Grafana Cloud shortcut), so no separate endpoint configuration is needed. Because the transport is shared but the gates are separate, the --otlp.* flags configure the logs endpoint even when --otlp.enabled (metrics export) is off.
• stdout - writes one compact JSON line per event to standard output. This is the zero-dependency path for container/Kubernetes setups where a node log collector already ships stdout.

When --logs.sink=otlp and no OTLP endpoint is resolvable (no --otlp.endpoint, no Grafana Cloud endpoint, and no OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_LOGS_ENDPOINT environment variable), startup fails with an error naming the missing flag rather than silently shipping nothing.

Sources¶

Syslog receiver (--logs.syslog.enabled)¶

The receiver is a push source: OPNsense forwards its logs to the exporter and the exporter enriches them from the API. It supersedes the old firewall and diaglog poll lanes, which spawned configd on the box and re-read log files the firewall will happily push. See Syslog receiver for the full setup, including the target you must configure on the firewall.

In short: it listens for RFC5424 or RFC3164 syslog over UDP, TCP and/or TLS (port 5514 by default), parses filterlog records into structured fields, ships every other program as a generic record, and enriches everything it can from the OPNsense API. It also derives low-cardinality Prometheus counters (opnsense_log_events_*) from what it parses, and can optionally sample high-volume raw lines away once their metric is counted - see Derived metrics and sampling and TLS transport.

Enrichment is the reason this lives in the exporter rather than a generic syslog collector. The exporter already holds an authenticated API client, so it can resolve what Alloy, Vector or rsyslog structurally cannot:

The rule description matters more than it looks. A filterlog rule id is either a rule UUID (for rules you wrote) or a content hash (for the auto-generated ones - anti-lockout, default-deny, bogon blocks, DHCP-allow). The config-level rule inventory only contains the former, so it cannot label the majority of the lines a real box emits. list_rule_ids resolves both.

Lookups read a lock-free snapshot refreshed on its own goroutine; the receive path never makes an API call. Enrichment failure never drops a record - a cold or stale snapshot ships the line unenriched.

Fidelity: the parser uses filterlog's true nine-field TCP tail (srcport, dstport, datalen, tcpflags, seq, ack, window, urg, options). OPNsense's own log reader declares eight, which mislabels the TCP window as the urgent pointer and drops the options entirely - so the receiver recovers data the API path silently loses.

Caveat: the filter log records the first packet of a flow only - it is an event stream, not flow accounting. Do not read event counts as byte/connection totals.

Zenarmor (--logs.zenarmor.enabled)¶

The exporter can also pose as an Elasticsearch node and receive Zenarmor's reporting data directly - a second push source, and the first high-volume one. See Zenarmor receiver for the full setup, including the GUI fields on the Zenarmor side and a warning about a similarly-named but destructive install-time Elasticsearch option you must not confuse this with.

In short: it accepts Elasticsearch _bulk writes on port 9200 by default, ships all six of Zenarmor's reporting families (conn, dns, tls, http, alert, sip - mapped to opnsense.subsystem values flow, dns, tls, web, ids, voip) as enriched OTLP logs, and derives a Prometheus counter (opnsense_log_events_zenarmor_total) from every document. Volume is substantial - measured at ~39 docs/sec on a live box, ~4–6 GB/day of raw JSON - so prefer cutting families in Zenarmor's own indexes setting over --logs.zenarmor.families: data cut at the source never crosses the wire.

IDS (Suricata EVE alerts)¶

--logs.ids.enabled (off by default; requires --logs.enabled) ships full Suricata EVE alert records - not just the aggregate counts the IDS metrics collector exposes (opnsense_ids_recent_alerts, gated separately by --exporter.enable-ids-alerts). It polls POST api/ids/service/query_alerts, the same endpoint the metrics collector's opt-in alert-activity series already uses, reading up to the newest 500 eve rows per poll (the same cap the metrics collector's alert count is a floor against).

• Body is the compact JSON of the complete raw eve alert record exactly as Suricata wrote it - every field it emits, not a reconstructed subset. This includes fields the exporter never parses into typed metadata (the nested flow object, app_proto, event_type, filepos/fileid, …).
• Structured metadata (never a label): alert_sid, alert_action, src_ip, dest_ip, in_iface, proto, signature.
• Severity is warn for a blocked alert, info otherwise.
• Cursor: the record's own timestamp is the true cursor, with a flow_id+alert_sid dedupe ring scoped to records sharing the cursor's exact timestamp. filepos/fileid are never used to cursor - log rotation shifts fileids, so they are fragile across restarts.
• Gap accounting: query_alerts is a windowed, saturating read. If a poll returns a full 500-row window whose oldest row is still newer than the prior cursor, the read could not reach back far enough to cover everything since the last poll - some alerts in that range were never observed. This is accepted, bounded loss: the source ships one synthetic gap record (event=gap_detected structured metadata, warn severity, a JSON body naming the gap bounds) instead of silently dropping it, so the loss is visible and queryable in Loki (e.g. {service_name="opnsense-exporter"} | opnsense_source="ids" | json | event="gap_detected").
• First poll: with no prior cursor (fresh start, or --logs.state-file not set/empty/corrupt), the whole initial window ships as a startup catch-up rather than being silently skipped or treated as a gap.

CrowdSec (--logs.crowdsec.enabled)¶

Ships CrowdSec alert and decision records. There is no native syslog path for these - the plugin registers no syslog scope, so alerts and decisions live only in the local API (LAPI). Off by default; enable with --logs.crowdsec.enabled (requires --logs.enabled). Silent when the os-crowdsec plugin is absent. Polls at a 60s floor regardless of --logs.poll-interval - each poll is a full cscli alerts/decisions dump (one configd exec each), so polling faster buys nothing at homelab/SMB event volumes.

• Cursor: alert ids and decision ids are each a separate, server-side monotonic counter. The source tracks the highest id shipped per record kind and ships only rows whose id is greater on the next poll - a plain id-diff, no timestamp windowing. On a cold start every currently-active alert/decision is shipped once, so enabling the source surfaces current state instead of silently starting from a blank slate.
• Body: compact JSON of the alert or decision (kind, id, scope_value, scenario, plus alert-only decisions/created or decision-only alert_id/action/expiration/events_count).
• Attributes (structured metadata, never labels): scenario, value (the scope:ip the alert/decision concerns - high cardinality, so metadata only, never a label), country, as (both often empty without a GeoIP database configured), plus decisions (alerts: a type:count summary, e.g. ban:1) or decision_type and duration (decisions: the CrowdSec action and the remaining-duration string, e.g. 693h46m29s).
• Timestamps: alerts carry an RFC3339 created field, used as the record's timestamp. Decisions carry no absolute timestamp (only a remaining-duration string), so the record is stamped at emit time.

Unbound (per-query DNS log)¶

--logs.unbound.enabled (default false) ships a pi-hole-style per-query DNS log: domain, client, action (Pass/Block/Drop), resolution source (Recursion/Local/Local-data/Cache), blocklist/policy enrichment and DNSSEC status per query - data unavailable anywhere else in OPNsense's API (syslog only carries raw unbound daemon lines). It requires Unbound reporting/statistics to be enabled on the firewall, and raises the poll floor to 15s (IntervalSource) because each poll spawns python+pandas+DuckDB on the box (~1s CPU).

Accepted sampling loss - read this before enabling on a busy resolver. Unbound's query-log backend (api/unbound/overview/search_queries) has no true cursor: without a per-client filter it only ever returns the newest 1000 rows across the whole resolver, newest first. This source reconstructs a best-effort cursor from each row's time (unix seconds) plus a dedup fingerprint for rows sharing the same second - the uuid field is part of the documented schema but has been observed to always be null in practice, so it is used only when present. On a resolver sustaining more than roughly 1000 queries between two polls, the oldest rows in the next page will all be newer than the previous cursor - a full discontinuity - meaning some number of queries fell out of the window before this exporter ever fetched them. That loss is never silent: it is counted via opnsense_exporter_logs_possible_gap_total{source="unbound"} (see Self-metrics) every time it is detected. Homelab/SMB query volumes (a handful to ~100 qps) are fine; a busy enterprise resolver should not enable this source.

Loki structured metadata for this source: client, domain, qtype, action, query_source, rcode, blocklist, dnssec_status. (Unbound's own source field - where the answer came from - is deliberately mapped to the query_source attribute rather than a bare source, to keep it clear of this pipeline's own opnsense.source stamp; see Loki label model.) The log body is a compact JSON encoding of the full row, including fields not promoted to structured metadata (family, resolve_time_ms, ttl, policy).

Loki label model¶

Loki promotes only OTLP resource attributes to index labels. Scope and log attributes can never become labels - otlp_config has no index_label action for them, whatever the tenant config says. Cardinality discipline therefore falls out of where an attribute is put, not out of policy:

On the resource (promotable - a closed, code-defined set):

The last two are Zenarmor-only: they are absent on every other source's records, and absent rather than empty, so they never pool unrelated records into one empty-valued stream. opnsense.device_category is clamped to a code-defined vocabulary before it reaches the resource - the receiver is a push endpoint, so an unrecognised value is a sender minting streams rather than data, and it folds into Zenarmor's own other bucket. opnsense.interface needs no clamp because it is resolved from the exporter's own view of the firewall's interface config: an interface the box does not have simply does not resolve. Note it is the interface description space, not the kernel device space (igb0, ixl0) - the two are disjoint.

Each lane's own verbatim keys (device.category, interface.name) stay on the record untouched, so queries already written against them keep working either way.

service.name and service.instance.id are indexed because they are on Loki's default promotion list. No host or SDK resource detectors are attached, so nothing else can leak into that set. The two custom keys are namespaced (opnsense.source, opnsense.subsystem) so they can never collide with a semconv/Loki-reserved key; Loki mangles the dot, so in LogQL they read opnsense_source and opnsense_subsystem.

On the record (structured metadata - never promotable): everything else. program, action, rule_id, rule_description, IPs, ports, MACs, hostnames, SIDs, tcp_*, dhcp_*, auth_*. Note program in particular: it comes off the syslog wire and any process on the firewall can pick its own tag with logger(1), so it is deliberately kept where it cannot become a label.

Promoting the opnsense.* attributes (optional)¶

Out of the box they all land in structured metadata, so you filter with |:

{service_name="opnsense-exporter"} | opnsense_subsystem="firewall" | action="block"

That scans every chunk for the instance. To turn it into a stream selection instead, promote them on the Loki side. Promotion is a tenant-side opt-in and costs the exporter nothing: it changes nothing about what is shipped, only how Loki indexes what arrives. Promote none, some or all of them, in any combination.

Self-hosted, put this in the Loki config:

limits_config:
  otlp_config:
    resource_attributes:
      # leave ignore_defaults false, or service.name stops being a label too
      attributes_config:
        - action: index_label
          attributes:
            - opnsense.source
            - opnsense.subsystem
            - opnsense.action
            - opnsense.device_category
            - opnsense.interface

On Grafana Cloud there is no config file, but the same otlp_config is settable per-tenant through the OTLP config self-serve API - PUT the resource_attributes block above to /loki/api/v1/config/limits/otlp_config with your Loki write token. No support ticket. The change is queued and can take a couple of business days to apply.

Then the same query becomes:

{service_name="opnsense-exporter", opnsense_subsystem="firewall"} | action="block"

Cost: bounded, because every one of these is a closed set. The combinations are a product, but most never occur - a printer does not appear on the WAN, a syslog record has no device at all - so the real figure is far below the theoretical one: 79 distinct streams per instance measured over an hour on a live box with all five promoted. Promoting fewer costs proportionally less: 16 with the original three.

Promotion does not reduce ingest cost. Loki bills on volume, and labels only change what a query has to scan.

A promoted attribute moves out of structured metadata and into the label, so | opnsense_subsystem=… stops matching once {opnsense_subsystem=…} starts - switch your queries when you switch the config. This applies per attribute, so a partial promotion leaves the rest filterable with | exactly as before.

Two different limits get conflated here, and conflating them invents a scarcity that is not there. max_label_names_per_series (default 15) bounds the label names on a single stream - it is per stream, so another exporter writing to the same tenant does not consume any of it, because its streams never carry an opnsense_* label. The exporter's widest stream carries seven. Separately, the tenant's promoted-attribute list is one shared list, so adding entries to it is a coordinated change - but adding another exporter's attributes cannot affect an opnsense_* stream. See OTLP attribute contract.

Do not promote anything else. src_ip as a label is one stream per address: the classic Loki cardinality footgun, and the reason the exporter keeps it out of reach.

Delivery semantics¶

Stated honestly, because this pipeline is pull-based over a lossy source:

• Within a run: at-least-once. Each source tracks its own cursor and a dedup ring, so rotation overlap does not duplicate and normal operation does not lose. The sink exports each batch synchronously and reports the real outcome: a record is counted opnsense_exporter_logs_shipped_total only once the OTLP endpoint has acknowledged it, never merely on enqueue. When one resource partition succeeds and a sibling fails, the acknowledged partition is removed and only the retryable remainder is sent again. Permanent HTTP/gRPC failures and OTLP partial-success rejections are terminal: they are counted once under reason="rejected" and are not retried. Transient failures are retried in memory (the records stay queued, never re-fetched) until the endpoint recovers or --logs.ship-max-attempts is reached, so a transient outage is ridden out for as long as the bounded queue behind it holds. Every loss path is counted: queue overflow, an oversized input record, a terminal destination rejection, exhausted retries, or a batch abandoned during shutdown. Each attempt that does not fully deliver is visible as opnsense_exporter_logs_ship_errors_total - degraded but never silent.
• Across restarts: at-most-once by default. Cursors are in memory, so a restart resumes from now. The in-memory retry does not survive a process restart: records polled but not yet delivered when the process exits mid-outage are lost (counted ship_failed if a graceful shutdown abandons them, unrecoverable on a hard kill). Set --logs.state-file to persist cursors (atomic JSON, rewritten only when a cursor changes) for best-effort resume across restarts.
• Never exactly-once.
• One path per log type: do not both ship a log type through this pipeline and forward the same type via native syslog - that double-ships. Pick one path per log type.
• One logs-enabled instance per firewall: running multiple logs-enabled replicas against the same firewall double-ships.

Debug capture¶

When a receiver reports a signal the exporter does not model - a Zenarmor Elasticsearch endpoint it does not implement, a document whose family or shape it does not recognise, a syslog line from a program with no parser - that signal is counted but its payload is not kept. Debug capture is an opt-in escape hatch that dumps those unmodelled payloads to disk as NDJSON, so you can see the real data and decide to model it or drop it without a packet capture on a multi-GB/day stream.

It captures only signals the exporter does not already handle - never the parsed, shipped stream:

Enable it with a shared directory plus a per-receiver toggle:

A per-receiver toggle with no --logs.debug-capture.dir set is a startup error - it would read as "on" but write nowhere. When Zenarmor capture is on, the receiver's "unhandled endpoint" warning is suppressed (the capture file carries the same signal); the logs_rejected_total{reason="unhandled_endpoint"} counter still fires, so the rate is never lost.

Files land as <dir>/<receiver>/capture-NNN.ndjson, rotating at 32 MiB. The --logs.debug-capture.max-bytes cap governs the whole directory, counting bytes left by previous runs: when it is reached capture stops and keeps the oldest samples (the first time an unknown appears is the sample worth having) - it never deletes to make room, so a debug capture can never fill the disk. Capture writes are best-effort and never block ingest: if the disk cannot keep up, entries are dropped and counted rather than stalling a receiver.

The capture files carry real network data - client addresses, DNS queries, TLS SNI, HTTP hosts - so they are written 0600 and should be treated as sensitive. Point the directory at a writable bind mount and remove the captures once you are done with them. For the containerised exporter (runs as UID/GID 65532 nonroot):

# docker-compose.yml
services:
  opnsense-exporter:
    volumes:
      - ./capture:/capture
    environment:
      OPNSENSE_EXPORTER_LOGS_DEBUG_CAPTURE_DIR: /capture
      OPNSENSE_EXPORTER_LOGS_ZENARMOR_DEBUG_CAPTURE: "true"

mkdir -p ./capture && sudo chown 65532:65532 ./capture   # match the nonroot container user

Watch capture activity with logs_debug_captured_total{receiver,kind} and logs_debug_capture_dropped_total{receiver,reason} (see Self-metrics).

Configuration¶

The pipeline flags are listed in the Configuration reference. The complete top-level pipeline set is --logs.enabled, --logs.sink, --logs.poll-interval (floor 5s), --logs.buffer-size, --logs.buffer-max-bytes, --logs.batch-max, --logs.max-record-bytes, --logs.ship-max-attempts, --logs.ship-concurrency, --logs.max-metric-keys, and --logs.state-file. Per-source --logs.<source>.enabled flags are documented alongside each source above as they land.

Sink throughput¶

The OTLP sink ships one resource partition per wire request — a partition being one distinct (source, subsystem, action, device_category, interface) tuple in the batch. That is what makes a partial-success rejection attributable to a single source, and it is not negotiable. It does mean a batch costs one round-trip per distinct partition, so throughput is governed by round-trips rather than by record count:

records/sec  =  ship-concurrency x (batch-max / partitions per batch) / latency

Two levers follow from that shape, and they multiply:

• --logs.ship-concurrency (default 8) exports that many partitions at once instead of strictly one after another.
• --logs.batch-max (default 5000) amortises the fixed per-partition round-trip. Distinct partitions plateau with batch duration while records keep growing linearly, so a larger batch costs barely more round-trips and carries far more records per round-trip.

Raise --logs.buffer-size if bursts still outrun the sink; it absorbs a burst but does nothing for sustained rate. If the queue is chronically deep rather than spiky, the sink rate is the problem, not the buffer.

A batch is bounded by bytes as well as by --logs.batch-max, at half the OTLP exporter's serialized-request ceiling. That bound is not configurable and is not about memory — the queue's own --logs.buffer-max-bytes covers memory. It exists because an oversized request is refused by the exporter before it reaches the wire, which yields no delivery outcome to classify, so the batch would be retried to exhaustion against bytes that can never be accepted. Note that gzip does not raise that ceiling: it is checked against the uncompressed protobuf. A single record larger than the bound is still shipped on its own rather than stalling the queue behind it; --logs.max-record-bytes is the bound that rejects those, at ingest.

Over HTTP the client's idle-connection pool is sized to the concurrency automatically. Note that an endpoint which does not negotiate HTTP/2 needs one TCP connection per concurrent flush — the Grafana Cloud OTLP gateway is HTTP/1.1-only, so this matters there. The gRPC transport multiplexes every flush over one connection and is unaffected.

Self-metrics¶

The pipeline exposes its own health metrics (visible at /metrics and on the Log Shipping dashboard tab). Every series carries the stable opnsense_instance identity label, matching the collector metrics. The remaining labels are listed exactly below; source, reason, stage, table, receiver, kind, and rule are not substitutes for that box identity.

• opnsense_exporter_logs_shipped_total{source} - records the sink confirmed exported (an acknowledged OTLP delivery, not merely enqueued).
• opnsense_exporter_logs_dropped_total{source,reason} - records dropped before delivery. Its closed reason values are overflow (the count or byte queue bound evicted the oldest record), record_too_large (rejected at ingest), rejected (terminal destination refusal), ship_failed_permanent (the configured retry bound was exhausted), and ship_failed (shutdown abandoned the retryable remainder).
• opnsense_exporter_logs_ship_errors_total - attempts that did not fully deliver. Acknowledged partitions are removed before retry, so this is an attempt counter rather than a record-loss counter.
• opnsense_exporter_logs_poll_errors_total{source} - source poll failures.
• opnsense_exporter_logs_last_received_timestamp_seconds{source} - exporter-clock time when this source most recently admitted a record to the queue: source-input liveness, not an origin event timestamp.
• opnsense_exporter_logs_last_exported_timestamp_seconds{source} - exporter-clock time when the sink most recently acknowledged a record from this source: delivery liveness, not queue admission or an origin event timestamp.
• opnsense_exporter_logs_queue_length / opnsense_exporter_logs_queue_capacity - record depth and configured record-count capacity; opnsense_exporter_logs_queue_bytes / opnsense_exporter_logs_queue_max_bytes are retained bytes and the aggregate byte budget (a max of 0 disables the byte bound).
• opnsense_exporter_logs_parse_errors_total{source,stage} - lines that failed to parse. Its closed stages are syslog envelope and Zenarmor bulk / document. These are not dropped: they ship with their raw body, so this counts fidelity lost, not data lost.
• opnsense_exporter_logs_rejected_total{source,reason} - receiver input refused before shipping. Closed syslog reasons are peer, oversized, filtered, sampled, conn_limit, tls_timeout, tls_auth_failed, and tls_deadline_error; closed Zenarmor reasons are peer, auth, unhandled_endpoint, body, overloaded, index_limit, unknown_family, filtered, self_traffic, and excluded.
• opnsense_exporter_logs_enrich_misses_total{table} - enrichment lookups that missed. A steady rate on table=rules means the snapshot is behind the box's ruleset.
• opnsense_exporter_logs_enrich_refresh_errors_total{table} - failed enrichment refreshes. The previous snapshot keeps serving, so records still ship - enriched with increasingly stale data.
• opnsense_exporter_logs_enrich_last_refresh_timestamp_seconds{table} - when each lookup table last refreshed successfully. Alert on time() - ... to catch a silently-stale cache.
• opnsense_exporter_logs_possible_gap_total{source} - a bounded-window source found no continuity with its previous cursor, so an unknown amount may have been skipped.
• opnsense_exporter_logs_resource_capped_total - records shipped without their opnsense.* index labels because the resource-label budget was exceeded; the record still arrives, but label-scoped queries under-report.
• opnsense_exporter_logs_debug_captured_total{receiver,kind} - unmodelled signals written to the debug-capture dir (see Debug capture). Zero unless capture is enabled.
• opnsense_exporter_logs_debug_capture_dropped_total{receiver,reason} - capture entries dropped rather than written: reason=buffer_full (disk could not keep up), reason=cap_reached (--logs.debug-capture.max-bytes hit; capture paused), reason=write_error, reason=duplicate_shape (a repeat of a message shape already captured this window - see below).
• opnsense_exporter_logs_zenarmor_excluded_total{rule} - Zenarmor records dropped by each configured exclusion rule; this is docs-only because it is a deliberate operator-configured blind spot, not a general pipeline-health panel.
• opnsense_exporter_logs_zenarmor_bulk_requests_total / opnsense_exporter_logs_zenarmor_bulk_bytes_total - accepted Zenarmor _bulk request count and decompressed request bytes; these are docs-only receiver-volume diagnostics.

opnsense_log_events_observation_dropped_total{reason} is not a Log Shipping pipeline self-metric. It belongs to the separate Log-derived Events collector's bounded receiver-to-collector handoff. It is documented in the generated metric reference and charted in the Derived Metric Budget row of the health dashboard's Log Shipping tab, with the closed reason handoff_full. The rest of that collector's counters are firewall data, so they sit on the operational dashboard beside the subsystem each one describes (filterlog events on Firewall & PF, tunnel lifecycle on VPN, and so on).

Why the syslog capture keeps one example per shape¶

The syslog lane captures any line whose program has no parser, and on a real firewall that is most lines: one reference box produced 60,073 entries and 31 MB in a day, 70% of it three repeated shapes (arpresolve failures, a chatty cron job, unbound blocklist updates). The byte cap governs the whole directory and stops when reached, so that one lane would fill it and permanently starve the NetFlow and Zenarmor captures sharing it - and a starved capture looks exactly like a quiet one.

So the syslog lane keeps the first example of each message shape per 15-minute window and counts the rest as reason=duplicate_shape. A shape is the message with its varying parts collapsed, so arpresolve: ... for 86.31.203.106 and the same line for another address are one shape. A shape never seen before is always captured immediately, and the suppressed count is what tells you the difference between a quiet lane and a busy one - a small capture file no longer means silence.

The NetFlow and Zenarmor lanes are not deduplicated. Both are bounded by design - a couple of datagrams at startup and nothing after - so a window would throw away samples they need to be complete.

See also¶

• Native Log Export - the syslog-ng/Alloy/NetFlow alternative to this pipeline, and the decision matrix for choosing between the two paths per log type.
• opnsense_exporter_logs_possible_gap_total{source} - possible sampling gaps detected by a source whose only view of its data is a bounded window (e.g. the unbound source's latest-1000-row DNS query log, see Sources above): incremented when a poll's page shows no continuity with the previous cursor, meaning an unknown amount of data was skipped between polls.