Log shipping¶

The exporter can ship structured OPNsense events (firewall log lines, IDS alerts, audit entries, and similar) to Loki, separately from the metrics it exposes at /metrics. This is opt-in and off by default: it runs only when --logs.enabled is set.

Log shipping is a long-lived background pipeline, not a scrape-time collector. Registered sources poll OPNsense event APIs on their own cadence, records pass through a bounded in-memory queue, and an emitter ships batches to the configured sink. It is independent of OTLP metrics export (--otlp.enabled): metrics and logs are gated by separate flags and neither turns the other on.

High-cardinality event data (IP addresses, ports, Suricata SIDs, domains) is shipped as log body and Loki structured metadata - never as a metric and never as a Loki label. The only labels are the resource identity, plus opnsense.source and opnsense.subsystem if you promote them (see Loki label model).

The pipeline is implemented in internal/logship/ on GitHub; parsers live one file per program, so adding a parser is a self-contained contribution.

Sinks¶

Select the sink with --logs.sink:

• otlp (default) - ships over OTLP logs. One sink covers both the Grafana Cloud OTLP gateway (which routes /v1/logs to Loki) and a self-hosted Loki 3.x native OTLP endpoint. It reuses the exporter's existing --otlp.* transport family (endpoint, protocol, headers, TLS/mTLS, and the Grafana Cloud shortcut), so no separate endpoint configuration is needed. Because the transport is shared but the gates are separate, the --otlp.* flags configure the logs endpoint even when --otlp.enabled (metrics export) is off.
• stdout - writes one compact JSON line per event to standard output. This is the zero-dependency path for container/Kubernetes setups where a node log collector already ships stdout.

When --logs.sink=otlp and no OTLP endpoint is resolvable (no --otlp.endpoint, no Grafana Cloud endpoint, and no OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_LOGS_ENDPOINT environment variable), startup fails with an error naming the missing flag rather than silently shipping nothing.

Sources¶

Syslog receiver (--logs.syslog.enabled)¶

The receiver is a push source: OPNsense forwards its logs to the exporter and the exporter enriches them from the API. It supersedes the old firewall and diaglog poll lanes, which spawned configd on the box and re-read log files the firewall will happily push. See Syslog receiver for the full setup, including the target you must configure on the firewall.

In short: it listens for RFC5424 or RFC3164 syslog over UDP, TCP and/or TLS (port 5514 by default), parses filterlog records into structured fields, ships every other program as a generic record, and enriches everything it can from the OPNsense API. It also derives low-cardinality Prometheus counters (opnsense_log_events_*) from what it parses, and can optionally sample high-volume raw lines away once their metric is counted - see Derived metrics and sampling and TLS transport.

Enrichment is the reason this lives in the exporter rather than a generic syslog collector. The exporter already holds an authenticated API client, so it can resolve what Alloy, Vector or rsyslog structurally cannot:

The rule description matters more than it looks. A filterlog rule id is either a rule UUID (for rules you wrote) or a content hash (for the auto-generated ones - anti-lockout, default-deny, bogon blocks, DHCP-allow). The config-level rule inventory only contains the former, so it cannot label the majority of the lines a real box emits. list_rule_ids resolves both.

Lookups read a lock-free snapshot refreshed on its own goroutine; the receive path never makes an API call. Enrichment failure never drops a record - a cold or stale snapshot ships the line unenriched.

Fidelity: the parser uses filterlog's true nine-field TCP tail (srcport, dstport, datalen, tcpflags, seq, ack, window, urg, options). OPNsense's own log reader declares eight, which mislabels the TCP window as the urgent pointer and drops the options entirely - so the receiver recovers data the API path silently loses.

Caveat: the filter log records the first packet of a flow only - it is an event stream, not flow accounting. Do not read event counts as byte/connection totals.

Zenarmor (--logs.zenarmor.enabled)¶

The exporter can also pose as an Elasticsearch node and receive Zenarmor's reporting data directly - a second push source, and the first high-volume one. See Zenarmor receiver for the full setup, including the GUI fields on the Zenarmor side and a warning about a similarly-named but destructive install-time Elasticsearch option you must not confuse this with.

In short: it accepts Elasticsearch _bulk writes on port 9200 by default, ships all six of Zenarmor's reporting families (conn, dns, tls, http, alert, sip - mapped to opnsense.subsystem values flow, dns, tls, web, ids, voip) as enriched OTLP logs, and derives a Prometheus counter (opnsense_log_events_zenarmor_total) from every document. Volume is substantial - measured at ~39 docs/sec on a live box, ~4–6 GB/day of raw JSON - so prefer cutting families in Zenarmor's own indexes setting over --logs.zenarmor.families: data cut at the source never crosses the wire.

IDS (Suricata EVE alerts)¶

--logs.ids.enabled (off by default; requires --logs.enabled) ships full Suricata EVE alert records - not just the aggregate counts the IDS metrics collector exposes (opnsense_ids_recent_alerts, gated separately by --exporter.enable-ids-alerts). It polls POST api/ids/service/query_alerts, the same endpoint the metrics collector's opt-in alert-activity series already uses, reading up to the newest 500 eve rows per poll (the same cap the metrics collector's alert count is a floor against).

• Body is the compact JSON of the complete raw eve alert record exactly as Suricata wrote it - every field it emits, not a reconstructed subset. This includes fields the exporter never parses into typed metadata (the nested flow object, app_proto, event_type, filepos/fileid, …).
• Structured metadata (never a label): alert_sid, alert_action, src_ip, dest_ip, in_iface, proto, signature.
• Severity is warn for a blocked alert, info otherwise.
• Cursor: the record's own timestamp is the true cursor, with a flow_id+alert_sid dedupe ring scoped to records sharing the cursor's exact timestamp. filepos/fileid are never used to cursor - log rotation shifts fileids, so they are fragile across restarts.
• Gap accounting: query_alerts is a windowed, saturating read. If a poll returns a full 500-row window whose oldest row is still newer than the prior cursor, the read could not reach back far enough to cover everything since the last poll - some alerts in that range were never observed. This is accepted, bounded loss: the source ships one synthetic gap record (event=gap_detected structured metadata, warn severity, a JSON body naming the gap bounds) instead of silently dropping it, so the loss is visible and queryable in Loki (e.g. {service_name="opnsense2otel"} | opnsense_source="ids" | json | event="gap_detected").
• First poll: with no prior cursor (fresh start, or --logs.state-file not set/empty/corrupt), the whole initial window ships as a startup catch-up rather than being silently skipped or treated as a gap.

CrowdSec (--logs.crowdsec.enabled)¶

Ships CrowdSec alert and decision records. There is no native syslog path for these - the plugin registers no syslog scope, so alerts and decisions live only in the local API (LAPI). Off by default; enable with --logs.crowdsec.enabled (requires --logs.enabled). Silent when the os-crowdsec plugin is absent. Polls at a 60s floor regardless of --logs.poll-interval - each poll is a full cscli alerts/decisions dump (one configd exec each), so polling faster buys nothing at homelab/SMB event volumes.

• Cursor: alert ids and decision ids are each a separate, server-side monotonic counter. The source tracks the highest id shipped per record kind and ships only rows whose id is greater on the next poll - a plain id-diff, no timestamp windowing. On a cold start every currently-active alert/decision is shipped once, so enabling the source surfaces current state instead of silently starting from a blank slate.
• Body: compact JSON of the alert or decision (kind, id, scope_value, scenario, plus alert-only decisions/created or decision-only alert_id/action/expiration/events_count).
• Attributes (structured metadata, never labels): scenario, value (the scope:ip the alert/decision concerns - high cardinality, so metadata only, never a label), country, as (both often empty without a GeoIP database configured), plus decisions (alerts: a type:count summary, e.g. ban:1) or decision_type and duration (decisions: the CrowdSec action and the remaining-duration string, e.g. 693h46m29s).
• Timestamps: alerts carry an RFC3339 created field, used as the record's timestamp. Decisions carry no absolute timestamp (only a remaining-duration string), so the record is stamped at emit time.

Unbound (per-query DNS log)¶

--logs.unbound.enabled (default false) ships a pi-hole-style per-query DNS log: domain, client, action (Pass/Block/Drop), resolution source (Recursion/Local/Local-data/Cache), blocklist/policy enrichment and DNSSEC status per query - data unavailable anywhere else in OPNsense's API (syslog only carries raw unbound daemon lines). It requires Unbound reporting/statistics to be enabled on the firewall, and raises the poll floor to 15s (IntervalSource) because each poll spawns python+pandas+DuckDB on the box (~1s CPU).

Accepted sampling loss - read this before enabling on a busy resolver. Unbound's query-log backend (api/unbound/overview/search_queries) has no true cursor: without a per-client filter it only ever returns the newest 1000 rows across the whole resolver, newest first. This source reconstructs a best-effort cursor from each row's time (unix seconds) plus a dedup fingerprint for rows sharing the same second - the uuid field is part of the documented schema but has been observed to always be null in practice, so it is used only when present. On a resolver sustaining more than roughly 1000 queries between two polls, the oldest rows in the next page will all be newer than the previous cursor - a full discontinuity - meaning some number of queries fell out of the window before this exporter ever fetched them. That loss is never silent: it is counted via opnsense_exporter_logs_possible_gap_total{source="unbound"} (see Self-metrics) every time it is detected. Homelab/SMB query volumes (a handful to ~100 qps) are fine; a busy enterprise resolver should not enable this source.

Loki structured metadata for this source: dns.client_label, domain, qtype, action, query_source, rcode, blocklist, dnssec_status, plus src.ip and its enrichment (src.hostname, src.mac, src.scope) on the minority of records where the client value is an address - see below. (Unbound's own source field - where the answer came from - is deliberately mapped to the query_source attribute rather than a bare source, to keep it clear of this pipeline's own opnsense.source stamp; see Loki label model.) The log body is a compact JSON encoding of the full row, including fields not promoted to structured metadata (family, resolve_time_ms, ttl, policy).

This lane cannot be joined to flow records by IP. OPNsense collapses the client address and its resolved hostname into one column before serialising the query log - stats.py handle_details() picks the hostname when its client table has one, then drops the address entirely, on every code path - so the raw client IP is not present anywhere in the API response. What arrives is therefore sometimes a hostname (camden.rob-knight.net), sometimes a bare IPv4 or IPv6 literal, sometimes a non-FQDN name (poly-office) or localhost, and sometimes empty.

The exporter does not guess. That value ships as dns.client_label - always set when non-empty, and the field to group a per-client panel on. src.ip is set only when the value already parses as an address (roughly 3% of rows on a box with reverse lookups working), and only then is it enriched to src.hostname / src.mac / src.scope from the exporter's own ARP/DHCP-lease tables. So src.ip on this source is either a real address or absent, never a hostname - but on most rows it is absent, and a query joining DNS to NetFlow, Zenarmor or firewall records by src.ip will match only that minority.

Reverse-mapping the hostname back to an address was measured against a live 1000-row page and rejected: it covered 35% of queries, missed the busiest client entirely (a static-IP host with no lease and no ARP hostname), and was four-way ambiguous on the second busiest. If you need IP-keyed DNS correlation, use the syslog per-query route below - Unbound's own daemon log carries the raw client address and never substitutes it.

Separately, the syslog receiver's unbound parser structures a plainer slice of this same data - query name/type/class and the matched local-zone - straight off Unbound's daemon log, with no blocklist, policy or DNSSEC detail. See Structured parsers for what it captures.

Unbound per-query DNS via syslog (--logs.syslog.unbound-per-query.enabled)¶

The second per-query DNS route. Unbound can log every query and every reply to its own daemon log; this parses that output instead of polling OPNsense's reporting database. The two routes are complementary and mutually exclusive - the exporter refuses to start with both enabled, because both log the same queries and running both ships two Loki records per query, silently doubling every per-query panel.

Which one to pick:

Firewall prerequisites. In Unbound's advanced settings, enable log-replies (and optionally log-queries) and log-tag-queryreply. The last one is not optional: without it Unbound tags these lines info: rather than query:/reply:, and the parser will not match them. Upstream defaults it off.

Prefer log-replies alone. A reply line carries every field a query line does plus four more (rcode, resolve time, from-cache flag, response size), so enabling only log-replies halves the ingest for strictly more data. log-queries is worth adding only if you specifically need evidence that a query arrived and was never answered.

Performance. Upstream's man unbound.conf warns that log-queries "Prints one line per query to the log ... Note that it takes time to print these lines which makes the server (significantly) slower", and says the same of log-replies. This is a synchronous write in the resolution hot path, per query, per worker thread. Measured on a homelab resolver at up to roughly 60x its ~6 qps baseline, enabling both settings had no detectable effect on loss, latency or throughput, and every saturation counter stayed at zero. That test cannot clear a genuinely busy resolver - the cost is per query, so it is invisible at these rates and would be orders of magnitude larger where it matters. Do not cite it to justify enabling this on a high-volume resolver.

The measurable cost is log volume: about 2.05 lines per DNS query, which on the test box was 1.14M lines/day and took DNS from 5.6% to 41% of all syslog.

Loki structured metadata: dns.query_name, dns.query_type, dns.query_class, src.ip (plus enrichment), unbound.thread and dns.event (query or reply) on every record; dns.rcode, dns.resolve_time_seconds, dns.cached and dns.response_size_bytes on reply records; dst.ip, dst.port and net.transport when log-destaddr is also on. Records land under opnsense_source="syslog" and opnsense_subsystem="dns", alongside the other syslog parsers - select them with {opnsense_source="syslog", opnsense_subsystem="dns"} | program="unbound".

Read the flag, not the timing, to identify a cache hit: a hit is always 0.000000 seconds, but a fast uncached reply is not.

When this route is off, these lines are not dropped - they ship as generic records exactly as they did before the parser existed. The firewall sends them either way; the flag decides whether the exporter structures them.

Flow log records (netflow, merged)¶

A push source layered on the separate NetFlow/Zenarmor flow pipeline (--flow.enabled, --flow.netflow.enabled, --flow.zenarmor - see NetFlow & flow metrics): the same correlator that builds the netflow_* metrics also ships one OTLP log record per correlated flow, stamped opnsense.source = netflow (NetFlow only) or merged (a NetFlow fragment joined with a matching Zenarmor conn document). Gated on three settings rather than its own --logs.<source>.enabled flag: --logs.enabled, --flow.enabled, and --flow.log-mode (per_flow, the default, ships the logs; off keeps deriving the metrics without them). See Flow log records for the full attribute set, timestamp semantics and the --flow.max-logs-per-window flood guard.

Loki label model¶

Loki promotes only OTLP resource attributes to index labels. Scope and log attributes can never become labels - otlp_config has no index_label action for them, whatever the tenant config says. Cardinality discipline therefore falls out of where an attribute is put, not out of policy:

On the resource (promotable - a closed, code-defined set):

The 26 opnsense.subsystem values are the syslog receiver's 22-entry program→subsystem map (internal/logship/syslog/registry.go) plus the four Zenarmor-only families that don't overlap it: flow, tls, web, voip (its other two, dns and ids, already exist in the syslog map).

The last two are Zenarmor-only: they are absent on every other source's records, and absent rather than empty, so they never pool unrelated records into one empty-valued stream. opnsense.device_category is clamped to a code-defined vocabulary before it reaches the resource - the receiver is a push endpoint, so an unrecognised value is a sender minting streams rather than data, and it folds into Zenarmor's own other bucket. opnsense.interface needs no clamp because it is resolved from the exporter's own view of the firewall's interface config: an interface the box does not have simply does not resolve. Note it is the interface description space, not the kernel device space (igb0, ixl0) - the two are disjoint.

Each lane's own verbatim keys (device.category, interface.name) stay on the record untouched, so queries already written against them keep working either way.

service.name and service.instance.id are indexed because they are on Loki's default promotion list. No host or SDK resource detectors are attached, so nothing else can leak into that set. The two custom keys are namespaced (opnsense.source, opnsense.subsystem) so they can never collide with a semconv/Loki-reserved key; Loki mangles the dot, so in LogQL they read opnsense_source and opnsense_subsystem.

On the record (structured metadata - never promotable): everything else. program, action, rule_id, rule_description, IPs, ports, MACs, hostnames, SIDs, tcp_*, dhcp_*, auth_*. Note program in particular: it comes off the syslog wire and any process on the firewall can pick its own tag with logger(1), so it is deliberately kept where it cannot become a label.

Promoting the opnsense.* attributes (optional)¶

Out of the box they all land in structured metadata, so you filter with |:

{service_name="opnsense2otel"} | opnsense_subsystem="firewall" | action="block"

That scans every chunk for the instance. To turn it into a stream selection instead, promote them on the Loki side. Promotion is a tenant-side opt-in and costs the exporter nothing: it changes nothing about what is shipped, only how Loki indexes what arrives. Promote none, some or all of them, in any combination.

Self-hosted, put this in the Loki config:

limits_config:
  otlp_config:
    resource_attributes:
      # leave ignore_defaults false, or service.name stops being a label too
      attributes_config:
        - action: index_label
          attributes:
            - opnsense.source
            - opnsense.subsystem
            - opnsense.action
            - opnsense.device_category
            - opnsense.interface

On Grafana Cloud there is no config file, but the same otlp_config is settable per-tenant through the OTLP config self-serve API - PUT the resource_attributes block above to /loki/api/v1/config/limits/otlp_config with your Loki write token. No support ticket. The change is queued and can take a couple of business days to apply.

Then the same query becomes:

{service_name="opnsense2otel", opnsense_subsystem="firewall"} | action="block"

Cost: bounded, because every one of these is a closed set. The combinations are a product, but most never occur - a printer does not appear on the WAN, a syslog record has no device at all - so the real figure is far below the theoretical one: 79 distinct streams per instance measured over an hour on a live box with all five promoted. Promoting fewer costs proportionally less: 16 with the original three.

Promotion does not reduce ingest cost. Loki bills on volume, and labels only change what a query has to scan.

A promoted attribute moves out of structured metadata and into the label, so | opnsense_subsystem=… stops matching once {opnsense_subsystem=…} starts - switch your queries when you switch the config. This applies per attribute, so a partial promotion leaves the rest filterable with | exactly as before.

Two different limits get conflated here, and conflating them invents a scarcity that is not there. max_label_names_per_series (default 15) bounds the label names on a single stream - it is per stream, so another exporter writing to the same tenant does not consume any of it, because its streams never carry an opnsense_* label. The exporter's widest stream carries seven. Separately, the tenant's promoted-attribute list is one shared list, so adding entries to it is a coordinated change - but adding another exporter's attributes cannot affect an opnsense_* stream. See OTLP attribute contract.

Do not promote anything else. src_ip as a label is one stream per address: the classic Loki cardinality footgun, and the reason the exporter keeps it out of reach.

Delivery semantics¶

Stated honestly, because this pipeline is pull-based over a lossy source:

• Within a run: at-least-once. Each source tracks its own cursor and a dedup ring, so rotation overlap does not duplicate and normal operation does not lose. The sink exports each batch synchronously and reports the real outcome: a record is counted opnsense_exporter_logs_shipped_total only once the OTLP endpoint has acknowledged it, never merely on enqueue. When one resource partition succeeds and a sibling fails, the acknowledged partition is removed and only the retryable remainder is sent again. Permanent HTTP/gRPC failures and OTLP partial-success rejections are terminal: they are counted once under reason="rejected" and are not retried. Transient failures are retried in memory (the records stay queued, never re-fetched) until the endpoint recovers or --logs.ship-max-attempts is reached, so a transient outage is ridden out for as long as the bounded queue behind it holds. Every loss path is counted: queue overflow, an oversized input record, a terminal destination rejection, exhausted retries, or a batch abandoned during shutdown. Each attempt that does not fully deliver is visible as opnsense_exporter_logs_ship_errors_total - degraded but never silent.
• Across restarts: at-most-once by default. Cursors are in memory, so a restart resumes from now. The in-memory retry does not survive a process restart: records polled but not yet delivered when the process exits mid-outage are lost (counted ship_failed if a graceful shutdown abandons them, unrecoverable on a hard kill). Set --logs.state-file to persist cursors (atomic JSON, rewritten only when a cursor changes) for best-effort resume across restarts.
• Never exactly-once.
• One path per log type: do not both ship a log type through this pipeline and forward the same type via native syslog - that double-ships. Pick one path per log type.
• One logs-enabled instance per firewall: running multiple logs-enabled replicas against the same firewall double-ships.

Debug capture¶

When a receiver reports a signal the exporter does not model - a Zenarmor Elasticsearch endpoint it does not implement, a document whose family or shape it does not recognise, a syslog line from a program with no parser - that signal is counted but its payload is not kept. Debug capture is an opt-in escape hatch that dumps those unmodelled payloads to disk as NDJSON, so you can see the real data and decide to model it or drop it without a packet capture on a multi-GB/day stream.

It captures only signals the exporter does not already handle - never the parsed, shipped stream:

Enable it with a shared directory plus a per-receiver toggle:

A per-receiver toggle with no --logs.debug-capture.dir set is a startup error - it would read as "on" but write nowhere. When Zenarmor capture is on, the receiver's "unhandled endpoint" warning is suppressed (the capture file carries the same signal); the logs_rejected_total{reason="unhandled_endpoint"} counter still fires, so the rate is never lost.

Files land as <dir>/<receiver>/capture-NNN.ndjson, rotating at 32 MiB. The --logs.debug-capture.max-bytes cap governs the whole directory, counting bytes left by previous runs: when it is reached capture stops and keeps the oldest samples (the first time an unknown appears is the sample worth having) - it never deletes to make room, so a debug capture can never fill the disk. Capture writes are best-effort and never block ingest: if the disk cannot keep up, entries are dropped and counted rather than stalling a receiver.

The capture files carry real network data - client addresses, DNS queries, TLS SNI, HTTP hosts - so they are written 0600 and should be treated as sensitive. They do not carry reusable authentication credentials: Authorization, Proxy-Authorization, Cookie, Set-Cookie, X-Api-Key and X-Auth-Token header values are redacted to a fixed placeholder before a captured request ever reaches disk, whatever receiver or auth scheme captured it. Point the directory at a writable bind mount and remove the captures once you are done with them. For the containerised exporter (runs as UID/GID 65532 nonroot):

# docker-compose.yml
services:
  opnsense2otel:
    volumes:
      - ./capture:/capture
    environment:
      OPN2OTEL_LOGS_DEBUG_CAPTURE_DIR: /capture
      OPN2OTEL_LOGS_ZENARMOR_DEBUG_CAPTURE: "true"

mkdir -p ./capture && sudo chown 65532:65532 ./capture   # match the nonroot container user

Watch capture activity with logs_debug_captured_total{receiver,kind} and logs_debug_capture_dropped_total{receiver,reason} (see Self-metrics).

Configuration¶

The pipeline flags are listed in the Configuration reference. The complete top-level pipeline set is --logs.enabled, --logs.sink, --logs.poll-interval (floor 5s), --logs.buffer-size, --logs.buffer-max-bytes, --logs.batch-max, --logs.max-record-bytes, --logs.max-export-bytes, --logs.ship-max-attempts, --logs.ship-concurrency, --logs.max-metric-keys, and --logs.state-file. Per-source --logs.<source>.enabled flags are documented alongside each source above as they land.

Sink throughput¶

The OTLP sink ships one resource partition per wire request — a partition being one distinct (source, subsystem, action, device_category, interface) tuple in the batch. That is what makes a partial-success rejection attributable to a single source, and it is not negotiable. It does mean a batch costs one round-trip per distinct partition, so throughput is governed by round-trips rather than by record count:

records/sec  =  ship-concurrency x (batch-max / partitions per batch) / latency

Two levers follow from that shape, and they multiply:

• --logs.ship-concurrency (default 8) exports that many partitions at once instead of strictly one after another.
• --logs.batch-max (default 5000) amortises the fixed per-partition round-trip. Distinct partitions plateau with batch duration while records keep growing linearly, so a larger batch costs barely more round-trips and carries far more records per round-trip.

Raise --logs.buffer-size if bursts still outrun the sink; it absorbs a burst but does nothing for sustained rate. If the queue is chronically deep rather than spiky, the sink rate is the problem, not the buffer.

A batch is bounded by bytes as well as by --logs.batch-max, by two bounds that answer different questions. Neither is about memory — the queue's own --logs.buffer-max-bytes covers memory.

The transport bound is half the OTLP exporter's serialized-request ceiling. It is not configurable. It exists because an oversized request is refused by the exporter before it reaches the wire, which yields no delivery outcome to classify, so the batch would be retried to exhaustion against bytes that can never be accepted. Note that gzip does not raise that ceiling: it is checked against the uncompressed protobuf.

The ingest-rate bound is --logs.max-export-bytes (default 1MiB), and it is the one that normally binds. A Loki tenant's ingestion limit is a bytes-per-second budget, which the transport ceiling knows nothing about: a 7.5MB push is worth many seconds of that budget and is rejected atomically however long the exporter waits, while being comfortably under the 32MiB transport bound. Set it to roughly one second of the destination's per-tenant bytes/sec budget. 0 falls back to the transport bound alone.

A single record larger than either bound is still shipped on its own rather than stalling the queue behind it; --logs.max-record-bytes is the bound that rejects those, at ingest.

Rate limits, splitting and pacing¶

An HTTP 429 or a gRPC RESOURCE_EXHAUSTED is classified separately from an ordinary transient failure, because the correct response is different. A generic 503 means the endpoint could not take the request right now, so resending the same bytes is right. A rate limit means the request was too much at once for a bytes/sec budget, so a byte-identical resend is guaranteed to fail again.

On a rate limit the emitter therefore halves the number of records it puts into one delivery attempt (floored at 32) and paces subsequent attempts by the advertised Retry-After — both the delay-seconds and the HTTP-date form are parsed, clamped to 30s so a misconfigured header cannot park the emitter — or by a 2s floor when nothing is advertised. The reduced size is carried forward across batches and walks back up geometrically once deliveries are clean again, so a sustained burst above the tenant's limit degrades into steady paced delivery rather than into whole batches dropped plus collateral queue overflow.

The gRPC path detects the rate limit but does not read an advertised delay (google.rpc.RetryInfo would need a new direct module dependency); the 2s pacing floor applies there instead.

Worst-case time spent on one undeliverable batch is bounded and accounts for both retry layers — the pipeline's --logs.ship-max-attempts and the OTLP exporter's own in-request retry, which is pinned to a 5s max-elapsed rather than the SDK's default of one minute so the two compose instead of multiplying:

t_worst  =  attempts x (sdk_max_elapsed + export_timeout)  +  (attempts - 1) x max_wait
         =  10 x (5s + 10s) + 9 x 30s  =  420s

420s is the pathological case where the destination advertises the maximum Retry-After on every attempt; with none advertised it is 195s. Attempts are counted across the whole batch rather than per chunk, so splitting never multiplies the bound.

Over HTTP the client's idle-connection pool is sized to the concurrency automatically. Note that an endpoint which does not negotiate HTTP/2 needs one TCP connection per concurrent flush — the Grafana Cloud OTLP gateway is HTTP/1.1-only, so this matters there. The gRPC transport multiplexes every flush over one connection and is unaffected.

Self-metrics¶

The pipeline exposes its own health metrics (visible at /metrics and on the Log Shipping dashboard tab). Every series carries the stable opnsense_instance identity label, matching the collector metrics. The remaining labels are listed exactly below; source, reason, stage, table, receiver, kind, and rule are not substitutes for that box identity.

• opnsense_exporter_logs_shipped_total{source} - records the sink confirmed exported (an acknowledged OTLP delivery, not merely enqueued).
• opnsense_exporter_logs_dropped_total{source,reason} - records dropped before delivery. Its closed reason values are overflow (the count or byte queue bound evicted the oldest record), record_too_large (rejected at ingest), rejected (terminal destination refusal), ship_failed_permanent (the configured retry bound was exhausted), and ship_failed (shutdown abandoned the retryable remainder).
• opnsense_exporter_logs_ship_errors_total - attempts that did not fully deliver. Acknowledged partitions are removed before retry, so this is an attempt counter rather than a record-loss counter.
• opnsense_exporter_logs_poll_errors_total{source} - source poll failures.
• opnsense_exporter_logs_last_received_timestamp_seconds{source} - exporter-clock time when this source most recently admitted a record to the queue: source-input liveness, not an origin event timestamp.
• opnsense_exporter_logs_last_exported_timestamp_seconds{source} - exporter-clock time when the sink most recently acknowledged a record from this source: delivery liveness, not queue admission or an origin event timestamp.
• opnsense_exporter_logs_queue_length / opnsense_exporter_logs_queue_capacity - record depth and configured record-count capacity; opnsense_exporter_logs_queue_bytes / opnsense_exporter_logs_queue_max_bytes are retained bytes and the aggregate byte budget (a max of 0 disables the byte bound).
• opnsense_exporter_logs_parse_errors_total{source,stage} - lines that failed to parse. Its closed stages are syslog envelope and Zenarmor bulk / document. These are not dropped: they ship with their raw body, so this counts fidelity lost, not data lost.
• opnsense_exporter_logs_rejected_total{source,reason} - receiver input refused before shipping. Closed syslog reasons are peer, oversized, filtered, sampled, conn_limit, tls_timeout, tls_auth_failed, and tls_deadline_error; closed Zenarmor reasons are peer, auth, unhandled_endpoint, body, overloaded, index_limit, unknown_family, filtered, self_traffic, and excluded.
• opnsense_exporter_logs_enrich_misses_total{table} - enrichment lookups that missed. A steady rate on table=rules means the snapshot is behind the box's ruleset.
• opnsense_exporter_logs_enrich_refresh_errors_total{table} - failed enrichment refreshes. The previous snapshot keeps serving, so records still ship - enriched with increasingly stale data.
• opnsense_exporter_logs_enrich_last_refresh_timestamp_seconds{table} - when each lookup table last refreshed successfully. Alert on time() - ... to catch a silently-stale cache.
• opnsense_exporter_logs_enrich_seam_reads_total{endpoint,outcome} - attempts to source an enrichment input from the shared-result seam rather than the OPNsense API. A hit is a request the firewall never received, because a metrics collector had already decoded that endpoint on its own poll; a miss means the refresher fetched it itself. Expect a hit ratio at or near 1 for every endpoint listed. An endpoint sliding toward all-misses means its owning collector is disabled, its poll is failing, or its plugin was removed - enrichment is unaffected (it falls back to fetching, exactly as it did before the seam existed), but the duplicate requests are back.
• opnsense_exporter_logs_possible_gap_total{source} - a bounded-window source found no continuity with its previous cursor, so an unknown amount may have been skipped.
• opnsense_exporter_logs_resource_capped_total - records shipped without their opnsense.* index labels because the resource-label budget was exceeded; the record still arrives, but label-scoped queries under-report.
• opnsense_exporter_logs_debug_captured_total{receiver,kind} - unmodelled signals written to the debug-capture dir (see Debug capture). Zero unless capture is enabled.
• opnsense_exporter_logs_debug_capture_dropped_total{receiver,reason} - capture entries dropped rather than written: reason=buffer_full (disk could not keep up), reason=cap_reached (--logs.debug-capture.max-bytes hit; capture paused), reason=write_error, reason=duplicate_shape (a repeat of a message shape already captured this window - see below).
• opnsense_exporter_logs_zenarmor_excluded_total{rule} - Zenarmor records dropped by each configured exclusion rule; this is docs-only because it is a deliberate operator-configured blind spot, not a general pipeline-health panel.
• opnsense_exporter_logs_zenarmor_bulk_requests_total / opnsense_exporter_logs_zenarmor_bulk_bytes_total - accepted Zenarmor _bulk request count and decompressed request bytes; these are docs-only receiver-volume diagnostics.

opnsense_log_events_observation_dropped_total{reason} is not a Log Shipping pipeline self-metric. It belongs to the separate Log-derived Events collector's bounded receiver-to-collector handoff. It is documented in the generated metric reference and charted in the Derived Metric Budget row of the health dashboard's Log Shipping tab, with the closed reason handoff_full. The rest of that collector's counters are firewall data, so they sit on the operational dashboard beside the subsystem each one describes (filterlog events on Firewall & PF, tunnel lifecycle on VPN, and so on).

Why the syslog capture keeps one example per shape¶

The syslog lane captures any line whose program has no parser, and on a real firewall that is most lines: one reference box produced 60,073 entries and 31 MB in a day, 70% of it three repeated shapes (arpresolve failures, a chatty cron job, unbound blocklist updates). The byte cap governs the whole directory and stops when reached, so that one lane would fill it and permanently starve the NetFlow and Zenarmor captures sharing it - and a starved capture looks exactly like a quiet one.

So the syslog lane keeps the first example of each message shape per 15-minute window and counts the rest as reason=duplicate_shape. A shape is the message with its varying parts collapsed, so arpresolve: ... for 198.51.100.106 and the same line for another address are one shape. A shape never seen before is always captured immediately, and the suppressed count is what tells you the difference between a quiet lane and a busy one - a small capture file no longer means silence.

The NetFlow and Zenarmor lanes are not deduplicated. Both are bounded by design - a couple of datagrams at startup and nothing after - so a window would throw away samples they need to be complete.

See also¶

• Native Log Export - the syslog-ng/Alloy/NetFlow alternative to this pipeline, and the decision matrix for choosing between the two paths per log type.
• opnsense_exporter_logs_possible_gap_total{source} - possible sampling gaps detected by a source whose only view of its data is a bounded window (e.g. the unbound source's latest-1000-row DNS query log, see Sources above): incremented when a poll's page shows no continuity with the previous cursor, meaning an unknown amount of data was skipped between polls.