How to expose tailscaled metrics¶

This is the operator how-to for getting per-node Tailscale metrics out of tailscaled and into the optional node_metrics scraper. It is the companion to the Metrics & Logs Reference - that page documents every signal tailscale2otel emits (including the Node metrics scraper section that describes how forwarded series look once they land in Grafana Cloud); this page covers the upstream side: how a node publishes its metrics, what the metrics are, the access control you need, and how to point the scraper at them.

These per-node metrics are separate from the tailnet-wide signals tailscale2otel derives from the Tailscale API/log stream. They come straight off each host's tailscaled daemon and describe that node's own data plane (throughput, dropped packets, routes, DERP/peer-relay state). The scraper forwards them verbatim - they are not renamed into the curated tailscale.* namespace.

Setup summary¶

1. On each node you want to monitor, enable the client metrics endpoint (Tailscale v1.78.0+):

tailscale set --webclient

1. Open TCP port 5252 to your monitoring host with a tailnet ACL grant (see Access control).

2. Point the scraper at http://<node-tailscale-ip>:5252/metrics in collectors.node_metrics. Native endpoints are plain HTTP with no token and no TLS, so you configure nothing else.

What endpoints tailscaled can expose¶

Client (host) metrics - the supported path¶

Since Tailscale v1.78.0, the client can serve its own metrics in Prometheus text format. Enable it once per node:

tailscale set --webclient

Once enabled, the same /metrics payload is reachable two ways:

Both are plain HTTP. There is no bearer token and no TLS on the native endpoint - see Access control for how reachability is actually gated.

For the scraper you want the tailnet form (:5252), because tailscale2otel runs on a different host from the nodes it scrapes. The quad-100 local form is handy for a quick curl http://100.100.100.100/metrics sanity check while you are logged into the node.

CLI alternatives (no listener)¶

If you cannot or do not want to open a port - for example you already run a Prometheus node_exporter textfile collector on the host - tailscale can print or dump the same metrics locally:

tailscale metrics print          # write the Prometheus-text metrics to stdout
tailscale metrics write <file>   # write them to a file (e.g. a textfile-collector .prom)

These feed an existing scrape pipeline on the node rather than being scraped by tailscale2otel directly.

The tailscaled debug server - avoid for monitoring¶

tailscaled --debug=localhost:8080 exposes /debug/metrics. This is localhost-only, unauthenticated, and an UNSTABLE superset of the client metrics whose names and contents can change between releases. Prefer the documented client metrics (--webclient → :5252) for anything you intend to alert or build dashboards on; treat /debug/metrics as an interactive debugging aid only.

Documented metric families¶

These are the stable client-metric families exposed on :5252/metrics. Names are emitted as-is by tailscaled; the scraper forwards them verbatim (Grafana Cloud's standard OTLP→Prometheus normalization still applies on ingest - see the reference).

Counters¶

Gauges¶

The path label and cardinality¶

Cardinality warning. The throughput counters (tailscaled_{inbound,outbound}_{packets,bytes}_total) carry a path label whose values are direct_ipv4, direct_ipv6, derp, peer_relay_ipv4, and peer_relay_ipv6. Series count therefore scales as nodes × path × metric. Scraping a large fleet multiplies quickly: budget for it, and prefer scraping a representative subset (e.g. relays and exit nodes) over every node if you only need fleet-level throughput. Use the cardinality controls in Configuration and monitor the scraper and active-series signals in the metrics catalog when sizing DPM and ingest cost.

Version note. Nodes older than v1.78.0 do not serve :5252/metrics. The scraper marks such a target as down (tailscale.node.up → 0) rather than emitting node series for it.

Access control - the only auth you need¶

The native client-metrics endpoint has no application-layer auth. Access is gated entirely at the tailnet layer: a peer can only reach :5252 on a node if your tailnet policy grants it. This is the standard Tailscale model - reachability is the authorization.

Open TCP port 5252 from your monitoring identity to the nodes you want scraped with a grant in your tailnet policy file. For example, allow hosts tagged tag:monitoring (where tailscale2otel runs) to reach port 5252 on hosts tagged tag:server:

{
  "grants": [
    {
      "src": ["tag:monitoring"],
      "dst": ["tag:server"],
      "ip": ["5252"]
    }
  ]
}

Tighten src/dst to match your tagging scheme. Because there is no token to manage and no TLS to terminate, the grant is the access-control boundary - keep it as narrow as possible.

Wiring up the scraper¶

The scraper lives under collectors.node_metrics and is off by default.

Static targets and per-target labels/headers must be set via a config file, not environment variables. The targets field is a list of structs; flat TS2OTEL_* env vars can only override scalar fields. Scalar scraper settings (enabled, interval, timeout, metric_allow, etc.) can be overridden via env as usual - e.g. TS2OTEL_COLLECTORS__NODE_METRICS__INTERVAL=30s.

A minimal static-target config for native endpoints needs nothing more than the URL:

collectors:
  node_metrics:
    enabled: true
    interval: 60s
    timeout: 10s
    max_response_bytes: 4194304  # per-target response cap (4 MiB)
    max_samples: 50000           # per-target sample cap per scrape
    targets:
      - url: "http://100.64.0.10:5252/metrics"
        instance: "relay-1"          # defaults to the URL host:port if omitted
        labels: { role: "relay" }    # extra static labels merged onto every series

Native tailscaled endpoints are plain HTTP - leave all the auth/TLS fields unset and the scrape is a plain GET with no added headers.

Dynamic discovery first run¶

For a fleet that changes over time, the scraper can discover targets from the Tailscale devices API instead of requiring one static entry per node. Static targets remain supported; discovery is an additional path and can be used with them or on its own. This is a complete discovery-only configuration (the explicit empty list makes it clear that no static endpoint is required):

collectors:
  node_metrics:
    enabled: true
    targets: []                 # discovery-only; omitting this has the same empty default
    discovery:
      enabled: true

collectors.node_metrics.enabled is the master switch and discovery.enabled turns on the device-inventory refresh. The discovery block's defaults are deliberately conservative:

Discovery selects only usable Tailscale addresses (IPv4 in 100.64.0.0/10 or IPv6 in fd7a:115c:a1e0::/48); a device with no such address is skipped. Discovered targets also carry host.name/host.id and tailscale.tags by default, which makes them joinable with device metrics. Set include_tags: ["tag:server"] to limit discovery to tagged nodes, for example.

Before the first refresh and scrape, make sure both sides are ready:

1. On every candidate node, run tailscale set --webclient on Tailscale v1.78.0 or newer. The scraper then requests http://<node-tailscale-ip>:5252/metrics, the client-metrics endpoint described above.
2. Grant the exporter host/peer access to TCP port 5252 on those nodes in the tailnet ACL. The endpoint has no bearer-token or TLS authentication; the ACL grant is its access boundary. See Access control for the grant shape.
3. Give the exporter’s Tailscale API credential devices:core:read for the devices inventory used by discovery (GET /devices). Use the broader read-only all:read only when the same credential also serves other enabled collectors that require additional API surfaces. Discovery does not evaluate ACLs or pre-probe reachability: an API-visible but blocked node becomes a target and reports tailscale.node.up=0 when its scrape fails.

The first collection tick performs a discovery refresh immediately; later refreshes use discovery.interval (default 5m), independently of the scrape interval. The active set is the union of static and discovered targets. Static entries are kept first, and a discovered URL that is already present in targets is dropped, so the static target's labels, credentials, and TLS settings win and the endpoint is not scraped twice. If a refresh fails, the previous active set is retained and discovery retries on the next collection tick. With the discovery-only YAML above, tailscale2otel.nodemetrics.discovery.targets is therefore the number of discovered targets after a successful refresh.

After the first successful refresh and scrape, use this Grafana Explore query to require a successful discovery, a non-empty discovered target set, target health, and at least one forwarded raw tailscaled_* series per discovered node:

(
  (tailscale_node_up_ratio == 1)
  and on (tailscale_tailnet, tailscale_node)
  (count by (tailscale_tailnet, tailscale_node) ({__name__=~"tailscaled_.+"}) > 0)
)
* scalar(min(tailscale2otel_nodemetrics_discovery_success_ratio))
* scalar(sum(tailscale2otel_nodemetrics_discovery_targets))
> 0

Each returned tailscale_node is an active target that answered and forwarded at least one raw family; when static targets are also configured, the result can include them. The min(...) term requires the last discovery refresh to have succeeded for every tailnet in a multi-tailnet deployment, while the sum(...) term separately proves a non-zero discovered target set; both are reduced to one scalar before filtering the per-node results. If a node exposes only cumulative counters, run the query after the second scrape: the first counter observation establishes its delta baseline, while gauges and tailscale.node.up are available on the first scrape.

Tailscale Kubernetes operator¶

1. Operator proxy pods are ordinary discovered tailnet devices, but their metrics listener is on 9002, not standalone tailscaled's 5252.

2. Enable metrics with ProxyClass.spec.metrics.enable, then set ProxyClass.spec.statefulSet.pod.tailscaleContainer.env to bind the metrics listener on the tailnet address:

spec:
  metrics:
    enable: true
  statefulSet:
    pod:
      tailscaleContainer:
        env:
          TS_LOCAL_ADDR_PORT: "[::]:9002"

The default cluster-IP bind is invisible from the tailnet. Keep 9002 because the operator hardcodes it in the metrics Service.

1. The tailnet ACL must grant the scraping host's tag to the proxy tags on tcp:9002.

2. For diagnosis, a missing ACL is a timeout (curl exit 28); a missing listener is connection refused (curl exit 7). Tailscale silently filters traffic that the ACL does not grant.

3. kube-apiserver ProxyGroups cannot be scraped via the tailnet: cmd/k8s-proxy ignores TS_LOCAL_ADDR_PORT and binds POD_IP unconditionally. An in-cluster ServiceMonitor is supported for those pods; their tailnet unreachability is not a defect in this scraper.

4. Use port_overrides for the operator ports while leaving devices without a matching override on discovery.port (5252). For example:

collectors:
  node_metrics:
    enabled: true
    discovery:
      enabled: true
      port: 5252
      port_overrides:
        "tag:k8s": [9002]
        "tag:k8s-egress": [9002]
        "tag:k8s-apiserver": [9002]

A device carrying more than one listed tag gets the sorted, deduplicated union of those port lists. The tag:k8s-apiserver targets remain down when reached through the tailnet, as described above.

What the scraper emits¶

• Verbatim series. Each scraped tailscaled metric is re-emitted with its original name and labels preserved (not renamed into tailscale.*).
• Counters → deltas, gauges → gauges. Cumulative counters (and histogram/summary components) are converted to monotonic deltas across scrapes; gauges are forwarded as gauges.
• Per-target tailscale.node.up. A health gauge (→ tailscale_node_up_ratio) is emitted per target reporting whether the last scrape of that node succeeded.
• Node identity = the tailscale_node label. Every forwarded series carries a tailscale.node attribute (Prometheus-normalized to tailscale_node) set to your instance: value, or the URL host:port if omitted - identity is a label, not an OTEL Resource, and deliberately not called instance: Grafana Cloud's OTLP→Prometheus translation promotes the resource attribute service.instance.id to the instance label, and that resource value would win, collapsing every node onto a single series. instance: in config only names the operator-facing field you set per target - it does not name the resulting attribute key. See the reference for how these query in Grafana.

Curated metrics¶

On top of the verbatim forward, the scraper emits a small set of curated, first-class metrics derived from specific tailscaled families. These are the catalog-documented tailscale.node.* series that dashboards and alerts build on, with stable OTel-native attributes instead of the raw per-node label spellings. Curation is purely additive: the raw tailscaled_* series is still forwarded byte-for-byte exactly as above - the curated series are emitted in addition, never instead.

Every curated series also carries the tailscale.node identity label, exactly like tailscale.node.up.

Attribute folding (deliberate cardinality reduction). tailscale.path collapses the raw path label's IP-version split - direct_ipv4/direct_ipv6 → direct, peer_relay_ipv4/peer_relay_ipv6 → peer_relay, derp stays derp - halving the per-node path fan-out; any unrecognized value folds to other. tailscale.drop.reason is a bounded admit-set (acl, error) with any other value folded to other, since scraped labels come from semi-trusted tailnet-member nodes. tailscale.health.type is passed through as-is (the value set is code-defined in tailscaled, not free text). The raw forwarded series keep their original, unfolded path/reason values - folding applies only to the curated copy.

Counters share the delta. A curated counter (tailscale.node.io, .packets, .packets.dropped, .peer_relay.*) consumes the same per-scrape delta the raw forward computes for its source series - including counter-reset detection and stale-baseline eviction - so there is no second baseline and no double-counting.

Filters bypass curation. The metric_allow / metric_deny / drop_labels passthrough filters apply only to the raw forward. Curated metrics are catalog metrics, not passthrough, so they are emitted regardless of those filters - denying tailscaled_.* still yields the curated tailscale.node.* series.

Scrape and discovery safety limits¶

The scraper enforces bounded work per node. max_response_bytes caps the number of bytes read from one target response (default 4194304, 4 MiB), and max_samples caps the number of valid samples accepted from one scrape (default 50000). A target that exceeds either limit is treated as a failed scrape and reports tailscale.node.up=0 for that collection.

When dynamic discovery is enabled, discovery.max_targets caps the number of discovered targets emitted in one refresh (default 1000), not the number of devices. Use include_tags/exclude_tags together with this cap to keep automatic discovery scoped to the nodes you intend to scrape. Both accept comma-separated values as env vars:

TS2OTEL_COLLECTORS__NODE_METRICS__DISCOVERY__INCLUDE_TAGS=tag:server,tag:relay
TS2OTEL_COLLECTORS__NODE_METRICS__DISCOVERY__EXCLUDE_TAGS=tag:exit-node

Optional per-target auth & TLS (proxied / HTTPS targets only)¶

You only need these fields if you front a node's metrics behind an HTTPS reverse proxy or an authenticating gateway - for example a Prometheus-compatible proxy that fans several nodes out behind one TLS endpoint. Native tailscaled endpoints need none of them.

targets:
  - url: "https://relay-2.ts.net:5252/metrics"
    bearer_token_file: "/run/secrets/scrape-token"   # read fresh each scrape (rotation-safe)
    # bearer_token: "..."                             # inline alternative; bearer_token_file wins
    headers: { X-Scope-OrgID: "1" }                   # extra request headers (e.g. tenant routing)
    tls:
      ca_file: "/etc/ssl/ca.pem"                      # custom CA to trust
      cert_file: "/etc/ssl/client.pem"                # client cert (mTLS)
      key_file:  "/etc/ssl/client-key.pem"            # client key  (mTLS)
      server_name: "relay-2.ts.net"                   # SNI / cert name override
      insecure_skip_verify: false                     # DEFAULT false — leave it false

Field notes:

A target with a non-empty tls block gets its own dedicated HTTP client/transport; targets without one share the default client.

Quick verification¶

From the node itself (local quad-100 address):

curl -s http://100.100.100.100/metrics | head

From the monitoring host (over the tailnet, after the ACL grant is in place):

curl -s http://<node-tailscale-ip>:5252/metrics | head

You should see Prometheus-text output beginning with # HELP / # TYPE lines and the tailscaled_* families above. If the tailnet curl hangs or is refused, the grant for port 5252 is missing or too narrow. If it returns 404 / empty, --webclient is not enabled on that node, or the node predates v1.78.0.

Once the scraper is running, confirm each target is healthy in Grafana:

tailscale_node_up_ratio

A value of 1 per tailscale_node means the last scrape of that node succeeded.

Sources¶

• Tailscale client metrics reference - https://tailscale.com/kb/1482/client-metrics
• Tailscale KB 1192 - ACL syntax / grants - https://tailscale.com/kb/1192/acl-tags
• Tailscale KB 1278 - tailnet policy file (grants) - https://tailscale.com/kb/1278/tailnet-policy-syntax
• Tailscale CLI - tailscale set, tailscale metrics - https://tailscale.com/kb/1080/cli

See also the Metrics & Logs Reference for the full catalog of signals tailscale2otel emits and how the forwarded node series appear in Grafana Cloud.