Collector gateway¶

tailscale2otel can export OTLP straight to your backend, or through a gateway — a Grafana Alloy or OpenTelemetry Collector instance that receives OTLP from the exporter, buffers it, and forwards it on. This page covers when the gateway is worth it, and ships a validated recipe for both Docker and Kubernetes.

The Docker recipe lives in deploy/alloy/ and is validated against a pinned Alloy release. Its README carries the tuning notes, the version-pinning traps and the full smoke-test transcript.

Direct export or gateway?¶

Both modes are supported. Direct export is the default and stays the right answer for plenty of deployments.

Grafana's own guidance is that the recommended architecture for production observability uses Grafana Alloy, for exactly these reasons — reliability, metadata enrichment, sampling and multi-backend routing — and that direct SDK export is the quickstart shape rather than the durable one.

Take the gateway when you cannot afford to lose telemetry across a backend outage or maintenance window, you already run one for other services, you want a single egress point, you need to enrich or route, or you would rather the backend token lived in one place than in every exporter.

Stay direct when a gap during an outage is acceptable, or a second container is not worth the operational surface. Point otlp.endpoint at your backend and you are done — see Getting Started.

What a gateway does not give you¶

The queue and retry loop mean an outage costs you far less. They do not make delivery perfect.

This reduces telemetry loss. It does not eliminate it, and it is not exactly-once delivery.

• Retries have a deadline. A batch not accepted within retry_on_failure.max_elapsed_time (30m in the shipped config) is dropped, persistent queue or not.
• The queue is finite. Past queue_size the gateway starts returning retryable errors and the exporter sheds data.
• Retries can duplicate. The gateway acknowledges on enqueue, not on backend acceptance, so a batch the backend committed but failed to acknowledge is sent again. That is the direct cost of the buffering.

Tolerating an outage of length T needs both a max_elapsed_time of at least T and a queue big enough for the batches produced during T. Raising one alone achieves nothing.

The pipeline¶

otelcol.receiver.otlp          :4317 gRPC / :4318 HTTP, on a private network
  -> otelcol.processor.memory_limiter    soft limit, first in the chain
  -> otelcol.processor.batch             coalesce a poll cycle
  -> otelcol.exporter.otlphttp           retry + sending queue
       -> otelcol.storage.file           optional, disk-backed queue (public preview)

Two ordering rules are load-bearing. The memory limiter goes first — anything ahead of it buffers data it can no longer protect — and the batch processor goes after it, which is Grafana's documented recommendation.

Docker Compose¶

docker compose --env-file deploy/.env -f deploy/alloy/docker-compose.yaml up -d

That compose file replaces deploy/docker-compose.yaml rather than overlaying it — do not pass both. It runs Alloy pinned to a specific version tag (never latest, because Alloy component arguments change between minor versions) alongside the exporter, and points the exporter at the sidecar:

TS2OTEL_OTLP__PROTOCOL: grpc
TS2OTEL_OTLP__ENDPOINT: http://alloy:4317
TS2OTEL_OTLP__TLS__INSECURE: "true"

tls.insecure disables transport security entirely, so it is only ever acceptable on a trusted private hop. It is safe here specifically because there is no credential on that hop — in gateway mode the backend token belongs to Alloy alone, and no otlp.grafana_cloud.* is set on the exporter at all.

Credentials¶

Every secret is supplied externally. The committed config.alloy reads its three backend settings through sys.env() and the compose file requires them with ${VAR:?}, so a missing value fails up with a named error instead of starting an unauthenticated gateway. Put them in the git-ignored deploy/.env:

GATEWAY_OTLP_ENDPOINT=https://otlp-gateway-prod-us-central-0.grafana.net/otlp
GATEWAY_OTLP_USERNAME=<your-numeric-instance-id>
GATEWAY_OTLP_PASSWORD=<your-access-policy-token>

For Grafana Cloud the username is the numeric instance/stack ID and the password is an access-policy token with the OTLP write scopes; both come from the Cloud Portal (organization Overview → Launch stack → Configure on the OpenTelemetry tile). Use your own region's otlp-gateway-<zone>.grafana.net host. The exporter appends /v1/metrics, /v1/logs and /v1/traces to the base URL itself.

Kubernetes¶

The tailscale2otel chart does not bundle Alloy, and deliberately so — a gateway is shared infrastructure with its own lifecycle, and most clusters that want one already have one. Deploy Alloy with its own chart (or k8s-monitoring), give it the pipeline from deploy/alloy/config.alloy, then point this chart at its Service.

Assuming Alloy is running as Service alloy in namespace monitoring with the OTLP receiver on 4317:

# values.yaml for the tailscale2otel chart, gateway mode.
config:
  otlp:
    protocol: grpc
    endpoint: http://alloy.monitoring.svc.cluster.local:4317
    tls:
      # Plaintext, in-cluster only. Safe here because gateway mode puts NO
      # credential on this hop - the backend token lives in Alloy. Use a real
      # TLS endpoint (and drop this) if the hop leaves the cluster.
      insecure: true
    # Deliberately absent: grafana_cloud.instance_id / grafana_cloud.token.
    # In gateway mode the backend credential belongs to Alloy, not here.

# The Tailscale credentials are still this chart's business. Supply them from a
# secret you manage rather than inline values.
existingSecret: tailscale2otel-credentials

That existingSecret must carry the usual TS2OTEL_* keys (TS2OTEL_TAILSCALE__AUTH__OAUTH__CLIENT_ID, TS2OTEL_TAILSCALE__AUTH__OAUTH__CLIENT_SECRET); see Installation for the full key list. Rotating an externally managed secret needs the pod replaced — Kubernetes never refreshes env in a running container — which is what the chart's rolloutTrigger value is for.

On the Alloy side, three things carry over from the Docker recipe and are easy to miss:

• Pass --stability.level=public-preview in the Alloy container args if you keep the otelcol.storage.file block. Without it Alloy will not start.
• Give the queue a PVC, not an emptyDir. A persistent queue on an emptyDir is discarded exactly when you needed it, which is worse than not enabling it — the config looks durable and is not.
• Keep memory_limiter.limit at roughly 80% of the container memory limit. A limiter budget above the pod's limit means the pod is OOM-killed before the limiter ever engages.

Verify it works¶

The gateway is an extra hop that can fail silently, so confirm both ends rather than assuming.

End to end: the exporter's own tailscale2otel.up gauge (tailscale2otel_up once normalized for Prometheus) appearing in your backend proves the whole chain. The exporter's admin status page shows OTLP delivery state per signal on its side of the hop — which tells you whether the exporter is reaching the gateway, not whether the gateway is reaching the backend.

The gateway's own health: Alloy serves a readiness endpoint and its own metrics on port 12345. Keep that listener on loopback or behind your own auth — it is unauthenticated and exposes pipeline internals.

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:12345/-/ready
curl -s http://127.0.0.1:12345/metrics | grep '^otelcol_exporter_queue_size'

otelcol_exporter_queue_size is the number you actually want on a dashboard: sustained growth means the backend is rejecting or throttling, and hitting otelcol_exporter_queue_capacity means you are dropping.

The outage and restart drill — stop the backend, watch the queue fill, restart the gateway, watch the backlog survive, restore the backend, watch it drain — is written up step by step with observed values in the deploy/alloy/ README. Run it once before you rely on the queue.

Next steps¶

• Installation — the direct-export Docker, Helm and binary paths.
• Configuration — every otlp.* key, including headers and TLS.
• Troubleshooting — what to check when nothing arrives.