Observability Guide¶

Observability in Paperless NGX Dedupe is comprehensive OpenTelemetry support: traces, metrics, and logs from the backend; Grafana Faro instrumentation from the frontend; and optional Pyroscope continuous profiling. This guide covers configuring all three for Grafana Cloud or a self-hosted OTLP collector.

Quick Start — Grafana Cloud¶

Set these environment variables to send all telemetry to Grafana Cloud:

# Backend telemetry (traces, metrics, logs)
OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-<region>.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64(instanceId:apiKey)>
OTEL_EXPORTER_OTLP_COMPRESSION=gzip
OTEL_SERVICE_NAMESPACE=paperless-dedupe

# Frontend telemetry (errors, Web Vitals, distributed traces)
FARO_ENABLED=true
FARO_COLLECTOR_URL=https://faro-collector-prod-<region>.grafana.net/collect/<app-key>

# Continuous profiling (optional)
PYROSCOPE_ENABLED=true
PYROSCOPE_SERVER_ADDRESS=https://profiles-prod-<region>.grafana.net
PYROSCOPE_BASIC_AUTH_USER=<instance-id>
PYROSCOPE_BASIC_AUTH_PASSWORD=<api-key>

# Stable semantic conventions
OTEL_SEMCONV_STABILITY_OPT_IN=database

End-to-End Trace Correlation¶

Frontend and backend traces are automatically correlated when both OTEL_ENABLED and FARO_ENABLED are set. Faro injects traceparent and tracestate headers into API requests, which the backend OTel instrumentation picks up to create connected traces.

Both services share the same service.namespace (paperless-dedupe by default), which groups them as one logical application in Grafana Cloud Application Observability.

Grafana Alloy (Optional)¶

For production deployments with multiple instances, a Grafana Alloy collector provides retry buffering, metadata enrichment, and exemplar relay. The app can send directly to Grafana Cloud for simpler setups.

Alloy Configuration¶

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http  { endpoint = "0.0.0.0:4318" }
  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.otlphttp.grafana_cloud.input]
    logs    = [otelcol.exporter.otlphttp.grafana_cloud.input]
    traces  = [otelcol.exporter.otlphttp.grafana_cloud.input]
  }
}

otelcol.exporter.otlphttp "grafana_cloud" {
  client {
    endpoint = "https://otlp-gateway-prod-<region>.grafana.net/otlp"
    auth     = otelcol.auth.basic.grafana_cloud.handler
  }
}

otelcol.auth.basic "grafana_cloud" {
  username = "<instance-id>"
  password = sys.env("GRAFANA_CLOUD_API_KEY")
}

When using Alloy, point the app at the Alloy receiver:

OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4318

Exemplars¶

OTLP metrics include exemplars (trace IDs) automatically when metrics are recorded during an active span. To use exemplars in Grafana:

1. In your dashboard panel, enable the Exemplars toggle
2. Configure the Prometheus data source to link its exemplar data source to your Tempo instance
3. Click exemplar diamonds on time series graphs to jump to the associated trace

When routing metrics through Alloy with Prometheus remote write, set send_exemplars = true:

prometheus.remote_write "default" {
  endpoint {
    url = "https://prometheus-<region>.grafana.net/api/prom/push"
    send_exemplars = true
    basic_auth {
      username = "<instance-id>"
      password = sys.env("GRAFANA_CLOUD_API_KEY")
    }
  }
}

Pyroscope Flame Graphs¶

When Pyroscope is enabled, profiles are labeled by operation type:

Filter flame graphs in Grafana Pyroscope by these labels to isolate CPU/memory hotspots per operation.

Semantic Conventions¶

This app follows OpenTelemetry semantic conventions:

• Database: Stable conventions (db.system.name, db.operation.name, db.query.text). Set OTEL_SEMCONV_STABILITY_OPT_IN=database.
• GenAI: Experimental conventions (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens).
• App-specific: Prefixed with app.* (app.job.id, app.ai.total_documents, app.batch.*).