Troubleshooting¶

This page covers the most common tailscale2otel problems, their root causes, and concrete fixes. All config keys reference the full key path; see Configuration for defaults and env-var equivalents. For release changes, start with the upgrade and rollback checklist, then use this page for symptoms observed after the restart.

Authentication failures¶

API key stopped working¶

Cause. A personal API key (tailscale.auth.method: apikey) expires in at most 90 days and is bound to the creating user. If that user is suspended or removed from the tailnet, the key is immediately revoked.

Fix. Switch to OAuth, which issues short-lived, auto-refreshing tokens that are not tied to any user and never expire on a fixed schedule:

tailscale:
  auth:
    method: oauth
    oauth:
      client_id: ""      # set via TS2OTEL_TAILSCALE__AUTH__OAUTH__CLIENT_ID
      client_secret: ""  # set via TS2OTEL_TAILSCALE__AUTH__OAUTH__CLIENT_SECRET
      scopes:
        - all:read

If you must keep method: apikey, the startup log will always contain a WARN advisory - that is expected and intentional.

401 responses logged at ERROR with OAuth¶

Cause. A 401 returned while OAuth is active (or an OAuth token-exchange failure with 401/403) is logged at ERROR by the API transport. This means the OAuth client credentials are wrong, the client has been deleted, or it lacks the required scopes.

Fix. Verify the client_id and client_secret match an active Tailscale OAuth client, and check that the client carries at least the all:read scope. If you use streaming.auto_configure, the log_streaming scope is also required:

tailscale:
  auth:
    oauth:
      scopes:
        - all:read
        - log_streaming   # only needed for streaming.auto_configure

No data arriving¶

Prometheus endpoint is empty, refused, or never appears in the scraper¶

Start with the pull-only starter and run the checks in order:

tailscale2otel -config examples/config/prometheus-only.yaml
curl --fail http://127.0.0.1:2112/metrics
tailscale2otel -prometheus-check -config examples/config/prometheus-only.yaml

The first command must remain running while the other two execute. A successful curl proves the listener; a successful -prometheus-check proves a bounded first exposition. Finally inspect the scraper's target status - that is the proof its network path and scrape configuration work. For a remote listener, configure prometheus.auth.token (or token_file) and use the same Bearer token in the scraper. An unauthenticated non-loopback bind requires the explicit allow_unauthenticated: true acknowledgement.

If the same backend also receives this exporter over OTLP, switch to delivery.mode: prometheus or stop the duplicate scrape. See Getting Started and Delivery modes.

Bare gateway URL returns 404 silently¶

Cause. When otlp.protocol: http, tailscale2otel calls <endpoint>/v1/metrics, <endpoint>/v1/logs and - when tracing is enabled - <endpoint>/v1/traces. It appends the per-signal paths for you. If you set otlp.endpoint to a bare gateway URL that does not end with /otlp (e.g. https://otlp-gateway-prod-us-central-0.grafana.net instead of …/otlp), those paths land at the wrong base and the gateway returns 404.

What a 404 actually does. A 404 is a delivery FAILURE, not a silently-accepted exchange - an earlier version of this page claimed otherwise and was wrong (corrected 2026-07-29 under #383, and now pinned by the wire-contract suite in internal/telemetry, which drives a real 404 and asserts the failure is recorded). It increments tailscale2otel_export_failures_total and shows on the admin status page's delivery panel with error class invalid. What makes it feel silent is that the failure is only visible in the exporter's OWN telemetry: nothing arrives at the backend, so a dashboard built on the exported data looks blank rather than broken. Check the admin status page or the process logs, not the absence of data.

Fix. Set otlp.endpoint to the base URL ending in /otlp:

otlp:
  endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp

The per-signal suffixes (/v1/metrics, /v1/logs, /v1/traces) are appended automatically. See Configuration for the Grafana Cloud default.

Wrong otlp.protocol¶

Cause. Setting otlp.protocol: stdout prints all signals to the console instead of sending them to a backend. This is correct for local debugging but will leave your metrics store empty.

Fix. Set the protocol to match your backend transport:

otlp:
  protocol: http   # or grpc

Flow-log / audit-log double-counting¶

Cause. flowlogs and auditlogs each have a source field that controls whether records come from the API poller, the Splunk-HEC stream receiver, or both. Setting source: both - or running the streaming receiver while a collector still polls the same log type - feeds the same records through the same processor twice. Cross-source de-duplication is a best-effort failsafe and does not guarantee exact-once delivery. The exporter logs a startup WARN when this condition is detected.

Fix. Pick exactly one ingestion path per log type:

collectors:
  flowlogs:
    source: poll     # or stream — not both
  auditlogs:
    source: poll     # or stream — not both

See Streaming & Webhooks for when to prefer stream over poll.

Running more than one instance against the same tailnet double-counts¶

With the default coordination.mode: none, two processes observing the same tailnet can emit duplicate records even if each uses only one ingestion source. Their dedup sets are process-local. Check for an old container, pod or service still running, and keep file checkpoints single-writer.

Run one active process per tailnet, or use Kubernetes coordination. In coordinated mode, check that all replicas use the same Lease and namespace and that listener Services select the leader. A separate uncoordinated process is outside that election and can still duplicate its traffic.

Flow/audit enrichment shows unknown or external¶

Cause. IP-to-device-name resolution for flow logs and audit records depends on the in-memory device-enrichment cache, which is populated by the devices collector. If devices is disabled, no cache is ever built and every address falls back to unknown (tailnet nodes) or external (off-tailnet addresses).

Fix. Ensure the devices collector is enabled (it is on by default):

collectors:
  devices:
    enabled: true

The tailscale2otel.enrich.cache_size gauge (→ tailscale2otel_enrich_cache_size_ratio) shows how many devices are currently in the cache; tailscale2otel.enrich.cache_age (→ tailscale2otel_enrich_cache_age_seconds) shows how stale it is.

/flows 404s after upgrading to 4.0.0¶

Symptom: the flow view was working, and after the upgrade /flows returns 404. The status page shows a "Flow view disabled" block, overall health reads degraded, and the log carries one line:

flow view disabled: persistent flow store failed to open ... cannot prove its tailnet identity

Cause: databases written before 4.0.0 are named flows-<tailnet>.db and carry no tailnet identity row. The rows hold user and device identities, and that filename is influenceable and cannot prove which tailnet they came from, so the service will not adopt one silently. Nothing has been deleted - the file is untouched.

Fix: stop the service and tell the binary which tailnet the database belongs to. Verify that is right before running it; naming the tailnet is the ownership assertion the filename cannot make.

This is the forward-only migration in the upgrade and rollback checklist.

tailscale2otel -config config.yaml -adopt-flow-db your-tailnet.example

It stamps the identity row, moves the file to the digest-qualified name, and reports how many rows came across. Run it once per tailnet. It is safe to re-run and safe to interrupt. If you would rather start clean, move the old file out of flows.store.directory instead.

See Adopting a database written before 4.0.0.

Cardinality overflow - series silently dropped¶

Cause. Every metric instrument is bounded by cardinality.metric_limit (default 10000). When the number of distinct active series for a single instrument reaches this cap, the OTLP SDK collapses all further series into a single {otel_metric_overflow="true"} series. Per-series detail is silently lost; only the overflow sentinel remains. The most common trigger is enabling per-port dimensions (cardinality.flow.source_port or cardinality.flow.destination_port) on a busy tailnet.

Diagnosis. Watch two self-observability signals:

• tailscale2otel_series_overflowing_ratio{metric_name="..."} - 1 when the named metric hit the cap during the last export interval.
• tailscale2otel_series_active{metric_name="..."} - the active series count, which pins at the cap when exceeded.
• A series with label otel_metric_overflow="true" appearing in your metrics store (e.g. tailscale_network_io_bytes_total{otel_metric_overflow="true"}) is the direct indicator.
• tailscale2otel_series_limit shows the configured cap (emitted only when a positive limit is set).

Fix. Either raise the cap or reduce cardinality:

cardinality:
  metric_limit: 50000        # raise the per-instrument series cap

  flow:
    source_port: false        # disable per-port dimensions (largest driver)
    destination_port: false
    metrics_mode: rollup      # use bounded top-N rollup instead of per-connection raw families
    rollup_top_n: 500         # keep only the busiest N src/dst pairs

Setting cardinality.metric_limit: 0 removes the cap entirely, at the cost of unbounded memory growth under high-cardinality conditions.

Node-metrics label collision (tailscale_node vs. instance)¶

Cause. The node-metrics scraper adds a tailscale_node label to every forwarded tailscaled series to identify which node the series came from. Deliberately, it does not use instance: on Grafana Cloud, the OTLP-to-Prometheus translation promotes the exporter's own service.instance.id resource attribute to the instance label. If the per-node label were also called instance, it would overwrite the collector-host value and collapse every scraped node's series onto the same instance, making per-node queries impossible.

If you see tailscale_node_up_ratio missing from your store, or all forwarded tailscaled_* series sharing the same instance label value rather than being distinguished by node name, check that your dashboards or recording rules query on tailscale_node, not instance.

Fix. No configuration change is required - the label is tailscale_node by design. Update any dashboard queries or alert rules that reference instance for these series to use tailscale_node instead.

Tracing enabled but no spans appear¶

Cause. With tracing.enabled: true but a *traceidratio sampler and tracing.sampler_arg: 0, the sampler records no spans. The startup log emits a WARN for this combination.

Fix. Set a non-zero tracing.sampler_arg (e.g. 1.0 to record everything, 0.1 for 10%), or use the always_on sampler. Also confirm the OTLP backend's access token carries traces:write - on Grafana Cloud, missing that scope drops trace export while metrics/logs still flow.

Suspected misconfiguration at runtime¶

Cause. Validate() errors and advisory Warnings() are logged at startup, but they are also surfaced as live gauges so you can alert without scraping logs.

Diagnosis. Query tailscale2otel_config_warnings_ratio (count of advisory warnings) and tailscale2otel_config_valid_ratio (0 when Validate() failed). Both are emitted each export cycle. The admin status page's Config tab lists every active advisory with the setting it concerns and its remediation, and /api/status.json carries the same list as advisories[] - so you do not have to find the startup log of a pod that may since have restarted.

Generating a support bundle¶

With admin.enabled: true, download GET /api/support-bundle.zip using admin authentication. The bounded archive includes the running version, configuration diagnostics, effective config with secrets reduced to {secret, set, source}, component/API/OTLP delivery state, and signal catalogs. manifest.json lists its contents and any truncation.

The archive also includes recent_logs.jsonl, a bounded process-log tail captured after structured log redaction. This is separate from raw flow, audit and webhook payloads.

The device inventory (device names, hostnames, users, IP addresses) is PII-heavy and is excluded by default; add ?include_devices=1 to opt in only when you intend to share it. Flow-log records and raw audit/webhook log content are never included by this bundle - there is no opt-in for either. Only their aggregate counts and store-health information are included.

Confirming the effective config without exposing secrets¶

Cause. Layered config (defaults < YAML file < TS2OTEL_* env) makes it easy to lose track of which value actually took effect, or which layer set it - especially across a multi-tailnet tailnets: list.

Fix. tailscale2otel -print-effective-config -config <file> loads the config exactly like a normal run and prints every effective key, redacted the same way the admin status page and support bundle are ({secret, set, source} for any secret, never a raw value), as a deterministic JSON array on stdout. Add -print-effective-config-format yaml for YAML instead, or -print-effective-config-provenance to also see which layer (default, file, or env) produced each key - still never the secret's content, just its origin. There is no flag that disables redaction; this command never prints a raw secret.

Still stuck?¶

Nothing here matching your symptom is worth reporting - undiagnosable failure modes are bugs in this page as much as in the code.

• Search existing issues - someone may have hit it already.
• Open a new issue - attach the support bundle above (or, without the admin server, the startup log, the output of tailscale2otel -validate -config <file>, and the version from tailscale2otel -version).
• Check the latest release notes - the behaviour may have changed since your build.

Please redact tailnet names, device names, and IP addresses from anything you paste that did not come from the support bundle above.