Workloads¶

A workload is the request-generating layer of a blueprint. Where constructs emit infrastructure telemetry on a fixed tick, a workload mints correlated request samples per master tick and projects them across traces, logs, and optionally RUM — threading one correlation key-set through every signal class so a trace ID in a span matches the same ID in the application log and the browser beacon.

Two workload kinds exist:

• web_service — a single service: a browser→backend→DB hop tree with optional gen_ai/LLM hops, RUM, Beyla, and profiling. Simple and common.
• app — a blueprint-declared service graph of typed nodes, each with its own custom metrics/logs/spans via the telemetry DSL, and one end-to-end correlated trace across the whole graph. Use when you need multiple backend services each with distinct telemetry, per-service incident targeting, or per-service scaling.

The two kinds may coexist in one blueprint: model the core correlated flow as an app, and simpler peripheral services as standalone web_service workloads.

web_service¶

web_service models a single backend service. One minted request produces:

• A trace: optional browser CLIENT root span → backend SERVER span → one CLIENT span per downstream call (database or cache hops, with db.* semantic conventions). Traces are backdated so they end at approximately now, matching how real spans export on completion.
• Application logs: one structured log line per request, correlated by trace/span ID.
• APM span-metrics: traces_spanmetrics_* histograms + service-graph series on the 60-second metric tick.
• Optional Faro/RUM beacons when rum: true and GC_FARO_* credentials are present.
• Optional Beyla eBPF observation lane when observability.beyla is set.
• Optional Pyroscope SDK-push profiles when pyroscope: is set with mode: sdk.
• Optional native OTLP application metrics (http.server.*) when otel.metrics: true.

Key fields¶

workloads:
  - type: web_service
    name: mine-api
    runs_on: mine-prod-use1      # binds to a cluster by its declared name
    tracing: true                # default true; false omits the OTLP lane
    rum: true                    # requires GC_FARO_* creds; omit or false = disabled
    traffic:
      off_peak_rps: 10           # trough request rate (default 5)
      peak_rps: 80               # plateau request rate (default 50)
    endpoints:
      - { route: "GET /v1/search",  error_rate: 0.01, p95_ms: 140 }
      - { route: "POST /v1/items",  error_rate: 0.02, p95_ms: 220 }
    calls:
      - { db: mine-app-db }      # resolved to the named database fixture
      - { cache: mine-sessions } # resolved to the named cache fixture

traffic drives the metric lane volume (span-metrics RPS). The correlation narrative (one request per master tick) is separate and smaller by design — it seeds realistic trace/log samples without inflating span-metric series cardinality.

endpoints are drawn uniformly across each minted request; error_rate and p95_ms shape the per-route latency distribution and error fraction in both spans and span-metrics.

gen_ai / LLM hops¶

When a blueprint wires an AI infrastructure construct (AgentCore, Bedrock) to the same cluster, web_service emits in-process gen_ai.* span attributes and correlated AI logs on the backend span. No additional YAML is needed on the workload; the gen_ai trace vocabulary comes from internal/genai (the same seam used by app).

app¶

app declares a multi-service graph where each node is a first-class service with its own telemetry. One request minted at the entry node propagates a single correlated trace across the whole graph: every node adds its own SERVER span (and optional CLIENT spans to its downstream calls), so the resulting trace shows the full request journey across all services.

Service nodes¶

Each node in services: is a ServiceNode:

Service graph example¶

workloads:
  - type: app
    name: mine-app
    runs_on: mine-prod-use1
    traffic:
      off_peak_rps: 5
      peak_rps: 40
      request_latency_p95_ms: 9000   # LLM-call budget; default 200ms suits plain HTTP
    models:
      - { model: gpt-4o,             provider: azure-openai }
      - { model: claude-3-5-sonnet,  provider: bedrock }
    services:
      - name: mine-frontend
        type: frontend
        entry: true
        runtime: node
        replicas: 2
        routes: ["GET /", "GET /search"]
        profiles: [rum_faro, runtime_node]
        calls: [mine-api]

      - name: mine-api
        type: web
        runtime: go
        replicas: 3
        profiles: [scraped_http_server, runtime_go, gen_ai_client]
        calls: [mine-db, mine-gateway]

      - name: mine-gateway
        type: gateway
        runtime: go
        replicas: 2
        profiles: [gateway_export_log]
        calls: [mine-llm]
        external: false

      - name: mine-llm
        type: llm
        external: true              # managed endpoint: trace hop only, no k8s pod
        calls: []

      - name: mine-db
        type: db
        runtime: go
        db_instance: mine-app-db   # links to the RDS fixture for db.* CLIENT span attrs
        calls: []

The resulting trace shape:

graph LR
    browser["browser CLIENT"]
    fe["mine-frontend SERVER"]
    api["mine-api SERVER"]
    gw["mine-gateway SERVER"]
    llm["mine-llm CLIENT (external)"]
    db["mine-db SERVER (db hop)"]

    browser --> fe
    fe --> api
    api --> gw
    api --> db
    gw --> llm

Models and gen_ai hops¶

The top-level models: list declares valid (model, provider) pairings. The app minter draws one pair per request and stamps it into the correlation — so gen_ai.* span attributes, gateway export logs, and eval log entries all carry the same model and provider for that request. Pairing prevents impossible combinations (e.g. a Claude model on the Azure-OpenAI provider).

An agentic_flow on a node adds a nested gen_ai span subtree inside that node's SERVER span:

- name: mine-api
  type: web
  runtime: go
  agentic_flow:
    workflow: mine-search-workflow
    agents:
      - name: search-agent
        tools: [vector_search, rerank, summarise]
    omit_chat: false    # false = include the chat <model> leaf (the LLM call)

Set omit_chat: true when a connected gateway or llm node already models the LLM call so it is not double-counted.

Telemetry DSL¶

Each node can declare custom metrics, log streams, and extra span attributes via inline DSL specs. The DSL is the profiles: escape hatch — use catalog profiles first, inline specs for anything not covered.

Value models (exactly one per field):

Capability matrix (enforced at load time):

• Metric labels and Loki stream labels: only const / const_str / enum. These must enumerate a stable, total domain on every run.
• Metric values: only numeric models — const, int_range, float_range, normal, bool, shape.
• Log body fields and span attributes: any model, including ref. High-cardinality correlation keys (trace_id, portkey_trace_id, run_id, etc.) ride here — never as labels or stream labels.

Example inline metric:

- name: mine-api
  type: web
  metrics:
    - name: mine_requests_total
      instrument: counter
      labels:
        route:  { enum: [{value: "/search", weight: 3}, {value: "/items", weight: 1}] }
        status: { enum: [{value: "200", weight: 9}, {value: "500", weight: 1}] }
      value:
        shape: { base: 1.0, mode: throughput_drop }

Catalog profiles¶

The catalog ships reusable profile templates that any node can apply by name in profiles::

Failure modes¶

Workloads register failure modes that incidents can target. See Incidents for how to activate them.

web_service modes (axis: workload):

app service-node modes (axis: service — each node is individually addressable):