Architecture¶

tailscale2otel is a single static Go binary that bridges the Tailscale observability surface to any OpenTelemetry backend. It polls the Tailscale API (and optionally receives streamed logs or webhooks), runs each data source through a typed conversion pipeline, and pushes the resulting metrics and logs through OTLP, Prometheus pull, or stdout. Optional traces describe the exporter’s own collection and delivery work.

High-level data flow¶

app.New fans out over the configured tailnets: each gets its own *tailnetRuntime (client, enrichment cache, scheduler, flow/audit processors), and every runtime feeds a per-tailnet telemetry.Provider inside one process-wide telemetry.ProviderSet. Under provider: headscale there is no fan-out - a single runtime talks to Headscale instead.

flowchart LR
    subgraph Sources ["Per tailnet (N runtimes)"]
        A[Tailscale API]
        B[Log stream\nSplunk-HEC]
        C[Webhook receiver]
    end

    HS[Headscale API\nprovider: headscale]

    subgraph Collectors ["internal/collector (one per source, per runtime)"]
        D[devices / users / keys\nsettings / acl / dns\nservices / contacts / webhooks\nposture / log_stream / nodemetrics]
        E[flowlogs / auditlogs\nwindow collectors]
    end

    subgraph Processors ["Per-runtime processors"]
        F[flowlog.Processor\naudit.Processor]
        G[enrich.DeviceCache]
        R[rdns.Cache\nexternal-IP PTR lookups]
    end

    subgraph Telemetry ["internal/telemetry.ProviderSet"]
        H[per-tailnet Provider\n+ process Provider]
    end

    I[(OTLP endpoint\nGrafana Cloud)]
    P[(Prometheus /metrics\noptional pull listener)]

    A --> D
    A --> E
    HS -.-> D
    B --> F
    C --> F
    E --> F
    D --> H
    F --> G
    F --> R
    G --> H
    R --> H
    H --> I
    H -.-> P

Collectors and processors emit only through the telemetry.Emitter interface - they never touch the OTEL SDK directly. This keeps OTLP a deployment concern, not a business-logic concern.

Control-plane providers: Tailscale or Headscale¶

By default tailscale2otel talks to Tailscale's hosted API. Setting provider: headscale points it at a self-hosted Headscale control plane instead, via internal/hsapi behind the same internal/provider abstraction - the collectors and processors described above are unaware of which backend is in play.

Headscale's API is narrower than Tailscale's, so under provider: headscale only the devices, users, keys, acl, and nodemetrics collectors run; the Tailscale-only collectors (flowlogs, auditlogs, services, webhooks, contacts, posture_integrations, log_stream, settings, dns) auto-disable, and several device/user signals are reduced or omitted. See Configuration → headscale for the full list of what's affected and the required connection settings.

Composition root - internal/app¶

app.New is where everything gets wired together. On startup it:

1. Resolves the configured tailnets (cfg.ResolvedTailnets() - one tailscale: block or a tailnets: list) and builds a telemetry.ProviderSet: one process-level OTEL provider (no tailscale.tailnet attribute; process/global self-obs) plus one per-tailnet provider (each stamps tailscale.tailnet=<name> and gets a distinct service.instance.id, so tailnets never collide on a shared Grafana Cloud backend).
2. Builds process-level shared dependencies (buildProcessDeps): the reverse-DNS cache (internal/rdns, gated by enrichment.reverse_dns.enabled), the webhook↔audit cross-dedup set (internal/dedup), and the release/version-check fetchers (internal/release, gated by version_checks.self.enabled / version_checks.devices.enabled).
3. Builds the collection machinery, branching on provider:
4. tailscale (default): one *tailnetRuntime per resolved tailnet. Each runtime gets its own internal/tsapi client (OAuth preferred, with refreshing access tokens, or a static API key), its own enrich.DeviceCache, its own flowlog.Processor / audit.Processor pair (so both the poll path and the stream/webhook path feed identical conversion logic within that tailnet), and its own collector registry + scheduler (internal/collector).
5. headscale: a single runtime backed by internal/hsapi behind the internal/provider abstraction (provider.Headscale) - no per-tailnet fan-out; it shares the process emitter rather than getting its own tailnet provider.
6. Starts any enabled receivers (internal/stream for Splunk-HEC, internal/webhook).
7. Launches the admin HTTP server (health probes, status page, POST /api/rdns/purge, optional pprof) when admin.enabled.
8. Launches the standalone Prometheus pull-endpoint server (internal/app/metrics.go) when prometheus.enabled, serving the merged per-provider gatherers from the ProviderSet.
9. Starts continuous profiling (Pyroscope push agent) when configured; failure to reach Pyroscope is non-fatal.

Run then starts, per active runtime, its scheduler loop plus the enabled release-check background loops (selfRelease.Run / tsRelease.Run) driving tailscale2otel.update_available and the device version-skew metrics.

The file internal/app/collectors.go is the canonical list of what is registered and under what config gates. Start here when you want to understand how a new data source would be added.

The default mode is singleton. Kubernetes coordination elects one active process for the whole fleet using a Lease. Standbys keep admin and process-metrics listeners running; listener Services select the leader. With Kubernetes coordination enabled, checkpoint.store: kubernetes shares sharded cursors through ConfigMaps. Enrichment, dedup and local views remain process-local.

Collector types and scheduling¶

internal/collector defines two interfaces:

• SnapshotCollector - a point-in-time read, called on a fixed interval. Used by devices, users, keys, settings, acl, dns, services, contacts, webhooks, posture_integrations, log_stream, and nodemetrics. Some retain state between ticks, including ACL ETags, previous snapshots, lifecycle observations and dedup sets.
• WindowCollector - a time-windowed read, called with an explicit [from, to] range. Used by flowlogs and auditlogs. Each run returns a high-water mark that becomes the from of the next window, enabling resumable polling across restarts.

Each collector runs in its own goroutine with a small randomised start-up stagger (bounded at 3 s by default) to spread initial API requests. Collisions remain possible. A panic or transient error in one tick is recovered and logged; it never stops the scheduler or any other collector.

Log sources - pick one per log type¶

For flow logs and audit logs there are three ingestion sources:

• Poll (source: poll): the window collector periodically calls the Tailscale Logs API and processes the batch.
• Stream (source: stream): tailscale2otel runs a built-in Splunk-HEC-compatible receiver; Tailscale pushes records to it in near-real time.
• Object store (source: objectstore): tailscale2otel reads the exports Tailscale writes to an S3-compatible bucket, with checkpointed batch ingestion and backfill.

Choose one path per log type. source: both is accepted but can double-count records. A best-effort bounded FIFO de-duplicate set (internal/dedup) can suppress exact duplicates across paths, but it is a failsafe, not a substitute for correct configuration. The app logs a WARNING at startup if both paths are active for the same log type. See Streaming & Webhooks for receiver setup.

All three sources feed the same flowlog.Processor / audit.Processor, so the emitted signals are identical regardless of which source delivers the record. Webhooks are a separate event feed; they are not a flow/audit source choice.

Device enrichment¶

Two enrichment layers feed source/destination naming on flow and audit records; the device cache resolves tailnet addresses, reverse-DNS resolves external ones.

• internal/enrich.DeviceCache - populated by the devices collector, in-memory, one instance per tailnetRuntime. Maps IP addresses and node IDs within that tailnet to human-readable device names. The flow-log and audit processors consult this cache first when annotating records and metrics with source.address / destination.address labels; a hit resolves to the device name, a miss on an in-tailnet-looking address resolves to unknown, and an address outside the tailnet resolves to external.
• internal/rdns.Cache (opt-in, enrichment.reverse_dns.enabled) - an async, bounded reverse-DNS (PTR) cache that only ever runs for addresses the device cache already bucketed as external (flowlog.Processor calls it exactly there - see internal/flowlog/processor.go). Lookups never block the hot path: a miss returns immediately and the resolved PTR name becomes available on a later sighting once the background lookup completes. Positive and negative results are cached with separate TTLs (enrichment.reverse_dns.cache_ttl / negative_ttl), bounded by max_entries. It is process-level shared infra (one cache, not one per tailnet) built in app.buildProcessDeps and wired into every runtime's flowlog.Processor in newRuntime (internal/app/tailnetruntime.go). It emits its own self-obs metrics (tailscale.rdns.cache.lookups, .evictions, .overflows, .entries, .capacity, tailscale.rdns.queries) and can be cleared on demand via POST /api/rdns/purge (see Self-observability and the admin status page below).

Checkpointing¶

Window collectors persist their high-water marks via internal/collector.CheckpointStore. By default this is a file, written atomically on each successful tick to checkpoint.file_path (default /var/lib/tailscale2otel/checkpoints.json - see configuration.md for the authoritative value). On a clean restart the collector resumes from the last saved mark rather than re-fetching the full history; on cold start (no checkpoint) it applies the configured initial_lookback window. In-memory cursor checkpointing is also available (useful for streamed deployments where poll cursors are unused). ACL revision and audit provenance use the independent checkpoint.evidence_store selector and share the same atomic file when durable, so choosing memory cursors does not reset semantic evidence on restart. Memory cursor state remains disposable and may cold-start after a restart.

If a window collection fails the high-water mark is not advanced, so the same window is retried on the next tick.

Version / release checks¶

internal/release.Fetcher is a cached, fail-open fetcher for an external "latest release" string plus version parse/compare helpers (release.Parse, release.Less), shared by two independent, config-gated background loops built in app.buildProcessDeps and started from Run:

• version_checks.self.enabled - a.selfRelease polls the tailscale2otel GitHub releases feed and drives tailscale2otel.update_available (comparing the running build version to the latest tagged release).
• version_checks.devices.enabled - a.tsRelease polls the latest stable Tailscale client release and feeds the devices collector's per-device / fleet version-skew metrics (flagging devices more than version_checks.devices.outdated_minor_threshold minor versions behind).

Both fetchers make plain outbound HTTPS calls (no Tailscale auth), cache their result for version_checks.cache_ttl, and are fail-open: an unavailable version source does not stop collection. Version-check status signals distinguish success from an unavailable source. Both loops run independently of self_observability.enabled: an operator can want update alerts with broad self-obs off.

Self-observability and the admin status page¶

tailscale2otel emits its own health signals as OTLP metrics (see Metrics for the full catalog):

• tailscale2otel.scrape.* - per-collector duration, success/failure counts, last-run timestamp, staleness (seconds since last success), and budget (last duration ÷ interval; ≥ 1 means risk of interval overrun), tagged with tailscale.collector.
• tailscale2otel.up - overall heartbeat gauge.
• tailscale2otel.series.* - per-source-metric active time-series count (series.active, pinned at the cap), the effective cap itself (series.limit, omitted when unlimited), and a 0/1 overflow flag (series.overflowing) that fires when excess series are silently dropped into otel_metric_overflow. Together these let you alert on cardinality cap hits without hardcoding the limit in PromQL.
• tailscale2otel.api.requests / api.retries - Tailscale API call counters, plus tailscale2otel.api.duration - a per-request latency histogram (with trace exemplars when tracing.enabled).
• Export-cost & ingest volume (C8): tailscale2otel.export.datapoints / export.log_records (the DPM/log-cost proxy) and tailscale2otel.ingest.records / ingest.size (per poll/stream/webhook/objectstore path; ingest.size counts decoded payload bytes and the poll path emits none), plus tailscale2otel.series.by_group.
• Health (C9): tailscale2otel.config.warnings / config.valid (runtime view of Validate()/Warnings()), the standard process.uptime / process.cpu.time, and checkpoint health (checkpoint.disk.size, checkpoint.persist.age). The dedup failsafe exposes tailscale2otel.dedup.hits (duplicates suppressed per set).
• Receivers: the Splunk-HEC and webhook receivers each emit *.inflight and *.request.duration alongside their record counters (see Streaming & Webhooks).

When tracing.enabled is set, a TracerProvider is also built in app.New and the scheduler, Tailscale API client, and receivers emit spans (one root span per scrape cycle, child spans per API request, one span per receiver request) over the same otlp.* endpoint.

In addition, an admin HTTP server (on by default, 127.0.0.1:9091) serves:

• / - an HTML status page with live collector health, cardinality table, the metrics/log catalog, discovered node-metrics targets, and a redacted config view.
• /api/status.json - the same data as JSON, for programmatic access. This and the other read-only JSON endpoints (/api/config.json, /api/cardinality.json, /api/flows.json, /api/flows/export.json, /api/events.json) each carry a top-level schema_version integer and are published/versioned artifacts - see docs/api/compatibility.md for the additive-vs-breaking policy and how CI enforces it (#323).
• /flows and /events expose the local flow and event explorers. Flow export is available as JSON and CSV at /api/flows/export.json and /api/flows/export.csv. Disabled views return 404.
• /api/support-bundle.zip downloads bounded diagnostics, including a redacted process-log tail. Device inventory requires ?include_devices=1.
• POST /api/rdns/purge - the admin server's only mutating endpoint: clears the reverse-DNS cache (internal/rdns.Cache). Method-gated (405 + Allow: POST on anything but POST), gated by the same admin-token auth as / and /api/status.json (requireAdminAuth), and additionally same-origin-checked (sameOrigin, internal/app/admin_status.go) as CSRF hardening: a browser request carrying Sec-Fetch-Site must be same-origin/none, or (falling back for clients that don't send it) an Origin header must match the request Host; a request with no Origin / Sec-Fetch-Site at all (e.g. curl) is allowed through since the admin-token gate is the primary control for those. A cross-origin browser request gets 403 cross-origin request forbidden. Responds 200 application/json with {"purged": <int>, "enabled": <bool>} - enabled reports whether reverse-DNS is configured at all (enrichment.reverse_dns.enabled); when it is false, purged is always 0.

• /healthz and /readyz - liveness and readiness probes.

  /healthz is process-only and answers 200 as long as the process is running: a component failure means "stop sending me traffic", not "restart me", and conflating the two turns a bad config into a crash loop.

  In singleton mode and on a coordinated leader, /readyz returns 503 while any collector has not completed its first tick, or once a long-running component has terminally failed - the stream and webhook receivers, the admin and Prometheus listeners, and the ingress WAL. A merely degraded collector (a failed tick, overdue, a stuck checkpoint) deliberately does not gate readiness; pulling the pod for a partial fault would take the whole exporter down.

  A coordinated standby becomes Ready when coordination starts, without running collectors. Services use the leader label to route active traffic.

  The status page derives its health verdict from that same component state, so the probe and the page cannot disagree, and /api/status.json carries a components[] array naming each subsystem with enabled, failed and the failure reason. The overall health is degraded whenever any of them has failed.

  OTLP export failure is the one thing that degrades health without affecting readiness. delivery[] reports, per signal (metrics, logs, traces), what the exporters actually shipped - attempts, failures, the current failure streak, the last success and failure times, the last attempt's duration, and an error class. Three consecutive failures marks a signal failing, which makes overall health degraded. It deliberately does not make /readyz return 503: a backend outage would otherwise pull every replica out of rotation at once, turning one vendor's bad afternoon into a cascading outage.

  delivery[] counts what came back; throughput counts what was handed to the exporters. Keep them distinct - the page's "last export" used to be the freshest collector success, which reported healthy data flow while every OTLP request was failing.

  last_error_class is one of timeout, canceled, unauthenticated, rate_limited, unavailable, invalid, other. The backend's response text is never surfaced: an OTLP error can carry the response body, which is exactly where an echoed credential or signed URL would be. - /debug/pprof - optional, requires profiling.pprof.enabled: true, which in turn requires both admin.enabled: true and admin.auth.token to be set (heap/goroutine dumps can expose in-memory secrets) - see security.md.

The status page is entirely self-contained - no CDN or external assets - so it renders on air-gapped tailnets.

Prometheus pull endpoint¶

A second, independent HTTP listener - off by default and enabled by prometheus.enabled, delivery.mode: prometheus, or delivery.mode: dual (default bind 127.0.0.1:2112, internal/app/metrics.go) - serves a single GET /metrics in the standard Prometheus exposition format. The delivery modes enable it even when the legacy boolean is false. It is separate from the admin server so pull scraping works even with the status page/pprof disabled, and an active instance gathers from every provider in the telemetry.ProviderSet (process + each tailnet). A coordinated standby serves only the process gatherer, selected on each scrape. Optionally gated by prometheus.auth.token (same Basic/Bearer constant-time check as the admin token). With no token, a loopback bind is open; a network-reachable bind returns HTTP 403 unless prometheus.auth.allow_unauthenticated: true explicitly acknowledges the exposure.

Key package boundaries¶

See Configuration for the full config reference and Metrics for every emitted signal.

Read the source¶

Every package above is browsable on GitHub - the composition root in internal/app is the best place to start, since it wires all of the rest together:

• internal/collector - scheduler, registry, and one package per source
• internal/telemetry - the OTEL facade
• internal/tsapi - Tailscale API client
• internal/provider - Tailscale/Headscale control-plane abstraction

Corrections and questions are welcome via GitHub issues.