sf2loki configuration reference¶

Config¶

Field	Type	Required	Description
`salesforce`	`SalesforceConfig`	no	Single-org Salesforce connection and authentication. Set this OR `orgs` (exactly one). Omit when using the multi-org `orgs` list.
`orgs`	`list[OrgConfig]`	no	Multi-org list: ingest several Salesforce orgs from one process into one shared sink. Set this OR top-level `salesforce` (exactly one). Each entry carries its own salesforce + sources; the sink/state/service stay shared.
`sources`	`SourcesConfig`	no	Single-org event source selection and settings. Ignored (and rejected if customized) when `orgs` is used — put per-org sources under each org.
`sink`	`SinkConfig`	yes	Log sink settings.
`state`	`StateConfig`	no	Checkpoint/state store settings.
`coordinate`	`CoordinateConfig`	no	Leadership coordination for active-passive HA.
`service`	`ServiceConfig`	no	Application-level service settings.

SalesforceConfig¶

Field	Type	Default	Required	Description
`login_url`	`str`	""	no	Optional: derived from `environment` when omitted; set to a custom My Domain URL to override. An explicit value always takes precedence over the environment-derived default.
`environment`	`Literal['production', 'sandbox']`	production	no	production \| sandbox — derives login_url when login_url is unset; an explicit login_url takes precedence.
`auth_mode`	`Literal['jwt_bearer', 'client_credentials']`	jwt_bearer	no	jwt_bearer (private key + cert) \| client_credentials (consumer key + secret).
`client_id`	`str`	${SF_CLIENT_ID}	yes	External Client App consumer key.
`client_secret`	`SecretStr`	(secret)	no	client_credentials flow secret (injectable from file/env like the key). Required when auth_mode is client_credentials; unused for jwt_bearer.
`client_secret_file`	`Path`	(secret)	no	File path to the client_credentials secret; required when auth_mode: client_credentials.
`username`	`str`	svc@example.com	no	Integration user (pre-authorised on the app's Policies tab). Required for jwt_bearer (the JWT `sub` claim); not needed when auth_mode: client_credentials.
`private_key_file`	`Path`	(secret)	no	File path to the jwt_bearer private key; not needed when auth_mode: client_credentials.
`private_key`	`SecretStr`	(secret)	no	jwt_bearer private key, inline (alternative to private_key_file).
`api_version`	`str`	"60.0"	no	Salesforce REST/SOAP API version to target.
`org_id`	`str`	null	no	Set this to keep the app on the `api` scope alone; left null it is auto-resolved via /services/oauth2/userinfo (which then needs the `openid` scope).
`token_ttl`	`Duration`	1h	no	Assumed access-token lifetime (Salesforce returns no expires_in for these flows; the real lifetime is the org's session timeout, which can be as short as 15m). Refresh is also reactive on 401, so this only tunes proactive re-mint cadence.
`limits`	`SalesforceLimitsConfig`		no	Org-limits metric poller settings.

SalesforceLimitsConfig¶

Field	Type	Default	Required	Description
`enabled`	`bool`	false	no	Poll /services/data/vXX.0/limits for org-limit gauges (API usage, storage, streaming events, ...).
`poll_interval`	`Duration`	5m	no	How often to poll the limits endpoint.

OrgConfig¶

Field	Type	Default	Required	Description
`name`	`str`	prod	yes	Org identifier: becomes the `org` stream label and the checkpoint key prefix. Letters, digits, underscore, hyphen only; must be unique.
`salesforce`	`SalesforceConfig`		yes	This org's Salesforce connection and authentication.
`sources`	`SourcesConfig`		no	This org's event source selection and settings.

SourcesConfig¶

Field	Type	Default	Required	Description
`pubsub`	`PubSubConfig`		no	Pub/Sub API streaming source.
`eventlog_objects`	`EventLogObjectsConfig`		no	Event-object SOQL polling source.
`eventlogfile`	`EventLogFileConfig`		no	EventLogFile (CSV) ingestion source.
`apexlog`	`ApexLogConfig`		no	ApexLog (Tooling API debug log) polling source.
`allow_overlap`	`bool`	false	no	Bypass the fail-fast overlap guard that refuses to start when one event category is enabled on more than one source.
`transform_salt`	`SecretStr`	(secret)	no	Deployment-wide salt for `hash` transform rules (stable pseudonyms that still correlate within this deployment). Strongly recommended whenever a hash rule is configured — unsalted hashes of low-entropy values (IPs, usernames) are trivially reversible by table lookup.
`transform_salt_file`	`Path`	/etc/sf2loki/secrets/transform-salt	no	File path to the transform hash salt (alternative to transform_salt).

PubSubConfig¶

Field	Type	Default	Required	Description
`enabled`	`bool`	true	no	Enable the Pub/Sub streaming source.
`endpoint`	`str`	api.pubsub.salesforce.com:7443	no	Pub/Sub API gRPC endpoint.
`default_num_requested`	`int`	100	no	Flow-control batch size (1-100; Salesforce clamps at 100 and returns INVALID_ARGUMENT when over-asked).
`replay_preset`	`Literal['LATEST', 'EARLIEST', 'CUSTOM']`	CUSTOM	no	Replay position; falls back to LATEST when no stored replay_id.
`topics`	`list[str]`	[/event/LoginEventStream, /event/ApiAnomalyEvent]	no	Explicit topics, or "" to DISCOVER and subscribe to every RTEM stream the org exposes (the EventStream channels), re-filtered by include/exclude.
`include`	`list[str]`	["*"]	no	Operator inclusion globs applied to discovered/explicit topics.
`exclude`	`list[str]`	[]	no	Operator exclusion globs applied to discovered/explicit topics.
`rediscovery_interval`	`Duration`	1h	no	How often to re-run wildcard ("*") topic discovery while running, so channels enabled after startup are picked up without a restart. 0 disables (discovery then runs only at startup).
`sample`	`dict[str, typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Gt(gt=0.0), Le(le=1.0)])]]`	{/event/ApiEventStream: 0.25}	no	Opt-in lossy volume control: topic glob -> keep fraction (0-1], first matching glob wins. Sampling is deterministic by replay_id hash, so a replay keeps exactly the same rows (Loki dedup stays intact). Sampled-out events still advance checkpoints.
`transforms`	`list[TransformRule]`		no	Redaction/filter rules applied to each decoded event payload before shaping (see TransformRule).
`bridge_max_bytes`	`int`	134217728	no	Approximate byte budget for the Pub/Sub source's internal topic->events bridge queue, which sits UPSTREAM of the pipeline's sink.loki.batch.queue_max_bytes. Topic tasks block (structural backpressure, propagated to flow-control credits) once the bridged-but-undrained bytes exceed this, so a sink outage can't balloon per-org buffering past the count bound. Default 128 MiB; 0 disables byte accounting on the bridge.

TransformRule¶

Field	Type	Default	Required	Description
`action`	`Literal['hash', 'mask', 'drop_field', 'drop_row', 'regex_replace']`	null	yes	hash (salted SHA-256 -> stable pseudonym) \| mask (format-aware: emails keep the domain, IPv4 truncates to /24, else '***') \| drop_field \| drop_row (row filter via `match`) \| regex_replace (pattern -> replacement).
`fields`	`list[str]`	[SOURCE_IP, CLIENT_IP]	no	Payload field names the action applies to (required for hash/mask/drop_field/regex_replace; not used by drop_row).
`match`	`dict[str, str]`	{}	no	drop_row only: drop rows where EVERY field equals (or glob-matches) its value, e.g. {EVENT_TYPE: Sites}.
`pattern`	`str`	null	no	regex_replace only: the regular expression to replace (must compile).
`replacement`	`str`	""	no	regex_replace only: the replacement text (backrefs allowed).
`name`	`str`	""	no	Optional stable rule name used as the `rule` metric label for drop_row counts; defaults to "-".

EventLogObjectsConfig¶

Field	Type	Default	Required	Description
`enabled`	`bool`	false	no	Enable the event-object polling source.
`objects`	`list[EventLogObjectConfig]`		no	Event objects to poll.
`transforms`	`list[TransformRule]`		no	Redaction/filter rules applied to each polled record before shaping (see TransformRule).

EventLogObjectConfig¶

Field	Type	Default	Required	Description
`name`	`str`	""	yes	The event object API name to poll (e.g. LoginEvent).
`timestamp_field`	`str`	EventDate	no	The field used as the per-row event time for polling/checkpointing.
`poll_interval`	`Duration`	5m	no	How often to poll this object.
`lookback`	`Duration`	1h	no	Initial window to fetch on first run (no checkpoint).
`sample`	`float`	1.0	no	Opt-in lossy volume control: keep fraction (0-1] of rows, deterministic by record Id hash (replay-stable). 1.0 keeps everything.
`big_object`	`bool`	false	no	Set true for Salesforce Big Objects (the stored RTEM event family: LoginEvent, ApiEvent, FileEventStore, *EventStore, ...). Big Objects reject ORDER BY ASC, so the source drains them newest-first (ORDER BY timestamp_field DESC) with a ratcheting upper bound and re-sorts each cycle's window ascending before emitting. Leave false for standard and custom objects (LoginHistory, MyAudit__c), which use the ASC path.
`max_catchup_records`	`int`	200000	no	Cap on records collected into memory per big_object DESC drain cycle. The drain buffers a cycle's window in memory to re-sort it ascending; a post-outage catch-up over a large gap would otherwise be unbounded and can OOM. When the cap is hit the drain emits that (internally sorted) segment and ratchets its upper bound down so catch-up proceeds in bounded chunks across cycles. 0 = unbounded (pre-cap behaviour). Ignored on the ASC path (streamed page-by-page, already bounded).

EventLogFileConfig¶

Field	Type	Default	Required	Description
`enabled`	`bool`	false	no	Enable the EventLogFile ingestion source.
`interval`	`Literal['Hourly', 'Daily']`	Hourly	no	Daily (settled, ~1d lag) \| Hourly (fresher, needs the Event Monitoring hourly opt-in in Setup) — pick ONE. Many orgs generate ONLY Daily files; check before Hourly.
`event_types`	`list[EventLogFileTypeConfig]`		no	ELF EventTypes to ingest (required when enabled — there is no sensible "all" default given ~70 types and the either/or-per-category model). Each item is a bare string (e.g. "Login") or a per-type object (name + optional structured_metadata_fields/labels). Use "*" to discover and ingest every EventType the org produces for this interval.
`exclude`	`list[str]`	[]	no	EventTypes to skip when the wildcard "*" is used (e.g. a category served by another source, or a high-volume type you don't want). Ignored otherwise.
`poll_interval`	`Duration`	1h	no	How often to list new files (Daily files land ~once/day).
`lookback`	`Duration`	1d	no	Initial window on first run (no checkpoint); reach back far enough to catch the last few settled Daily files.
`timestamp_column`	`str`	TIMESTAMP_DERIVED	no	Per-row event time column.
`page_size`	`int`	1000	no	SOQL LIMIT for the file-listing query.
`settle_window`	`Duration`	0s	no	Skip files whose CreatedDate is newer than now-settle_window, so we don't pull a half-written hourly CSV whose tail rows would then be skipped when the watermark passes it. Left unset it defaults to 5m for interval: Hourly (Hourly blobs can be listed while server-side incomplete) and 0 for Daily (files land long after the day closes). Set explicitly to override either.
`download_max_age`	`Duration`	1d	no	A file whose body keeps failing to download and is older than this is abandoned (checkpoint advances past it) so a permanently-missing file can't wedge the watermark forever. Files younger than this are retried.
`transforms`	`list[TransformRule]`		no	Redaction/filter rules applied to each CSV row before shaping (see TransformRule).
`concurrency`	`int`	4	no	Event types processed concurrently per poll cycle (per-type ordering and checkpoints are unaffected — types are independent). Peak memory is roughly concurrency x 8 MiB of download spool.

EventLogFileTypeConfig¶

Field	Type	Default	Required	Description
`name`	`str`	""	yes	The ELF EventType this override applies to (e.g. ReportExport), or "*" to discover all types.
`structured_metadata_fields`	`list[str]`	[REPORT_ID, OWNER_ID]	no	Per-type override of the global sink.loki.structured_metadata_fields; omit (None) to inherit the global list, or set to [] to suppress it.
`labels`	`list[str]`	[DELEGATED_USER]	no	Columns promoted to Loki stream labels for this event type. Keep these LOW cardinality — each distinct value is a new Loki stream.
`sample`	`float`	1.0	no	Opt-in lossy volume control: keep fraction (0-1] of this type's rows, deterministic by row key hash (replay-stable, Loki-dedup-safe). 1.0 keeps everything.

ApexLogConfig¶

Field	Type	Default	Required	Description
`enabled`	`bool`	false	no	Enable the ApexLog polling source.
`poll_interval`	`Duration`	1m	no	How often to poll for new ApexLog rows.
`lookback`	`Duration`	1h	no	Initial window to fetch on first run (no checkpoint).
`users`	`list[str]`	[integration@example.com]	no	Salesforce usernames whose logs to ingest (matched via LogUser.Username). Empty = every ApexLog visible to the integration user.
`max_body_bytes`	`int`	5242880	no	Skip the body download for logs whose LogLength exceeds this (the metadata line is still shipped, flagged body_skipped); the per-line cap (sink.loki.batch.max_line_bytes) truncates whatever is shipped.
`sample`	`float`	1.0	no	Opt-in lossy volume control: keep fraction (0-1] of logs, deterministic by log Id hash (replay-stable). 1.0 keeps everything.
`transforms`	`list[TransformRule]`		no	Redaction/filter rules applied to each log's metadata before shaping.

SinkConfig¶

Field	Type	Default	Required	Description
`loki`	`LokiConfig`		yes	Loki sink settings.

LokiConfig¶

Field	Type	Default	Required	Description
`url`	`str`	${GC_LOKI}	yes	Loki push API URL.
`tenant_id`	`str`	${GC_TENANT_ID}	no	Loki tenant (X-Scope-OrgID); required for Grafana Cloud and multi-tenant Loki.
`auth_token_file`	`Path`	(secret)	no	File path to the Loki auth token.
`auth_token`	`SecretStr`	(secret)	no	Loki auth token, inline (alternative to auth_token_file).
`encoding`	`Literal['protobuf', 'json']`	protobuf	no	Wire encoding for the push request.
`compression`	`Literal['snappy', 'gzip', 'none']`	snappy	no	Compression: snappy (protobuf) \| gzip (json) \| none.
`batch`	`LokiBatchConfig`		no	Push batching settings.
`egress`	`EgressConfig`		no	Egress guardrails: rate caps + daily byte budget (all off by default).
`labels`	`dict[str, str]`	{environment: prod}	no	Static stream labels merged onto every push (job + sf_org_id are added automatically).
`structured_metadata_fields`	`list[str]`	[replay_id, schema_id, event_uuid, user_id, username, source_ip, session_key]	no	Event fields promoted to Loki structured metadata (not stream labels).

LokiBatchConfig¶

Field	Type	Default	Required	Description
`max_entries`	`int`	1000	no	Flush the batch after this many log entries.
`max_bytes`	`int`	1048576	no	Flush the batch after this many bytes.
`flush_interval`	`Duration`	1s	no	Flush the batch after this much time, regardless of size.
`max_line_bytes`	`int`	262144	no	Per-line UTF-8 byte cap; lines longer than this are truncated (with a marker) before push so one oversized row can't 400 its whole batch. Mirrors Loki's server-side `max_line_size` default (256 KiB). 0 disables.
`queue_maxsize`	`int`	10000	no	Entry-count bound of each internal source->sink lane queue. Producers block (structural backpressure) when full. Applied PER LANE (streaming vs bulk), so the worst-case entry count is queue_maxsize x number-of-lanes (<= 2).
`queue_max_bytes`	`int`	268435456	no	Approximate byte budget for queued entries, applied PER LANE (streaming vs bulk). Producers on a lane block when its budget is exceeded, even if the entry-count bound is not reached. Worst-case buffered memory during a sink outage is therefore queue_max_bytes x number-of-lanes (<= 2x). Default 256 MiB; 0 disables accounting.

EgressConfig¶

Field	Type	Default	Required	Description
`max_lines_per_second`	`float`	0	no	Token-bucket cap on pushed lines/second; 0 disables.
`max_bytes_per_second`	`float`	0	no	Token-bucket cap on pushed (pre-compression) bytes/second; 0 disables.
`daily_byte_budget`	`int`	0	no	Maximum pre-compression bytes pushed per UTC day; 0 disables. The used counter persists in the state store, so restarts don't reset it. WARN at 80%, ERROR + budget_action at 100%.
`budget_action`	`Literal['pause', 'drop']`	pause	no	What to do when the daily budget is exhausted: pause (hold pushes + checkpoints until the next UTC day; data delayed, never lost, readiness reports degraded) \| drop (keep running, discard over-budget entries).

StateConfig¶

Field	Type	Default	Required	Description
`store`	`Literal['file', 's3', 'gcs']`	file	no	State backend: file (local JSON, needs a persistent volume) \| s3 (S3-compatible object storage, for stateless deployments; needs the sf2loki[s3] extra) \| gcs (Google Cloud Storage, for stateless deployments; needs the sf2loki[gcs] extra).
`file`	`FileStateConfig`		no	File-backed state store settings.
`s3`	`S3StateConfig`		no	S3-backed state store settings.
`gcs`	`GcsStateConfig`		no	GCS-backed state store settings.

FileStateConfig¶

Field	Type	Default	Required	Description
`path`	`Path`	/var/lib/sf2loki/state.json	no	Checkpoint file path; persist on a mounted volume for durable resume.

S3StateConfig¶

Field	Type	Default	Required	Description
`bucket`	`str`	""	no	Bucket name (required when state.store is s3).
`key`	`str`	sf2loki/state.json	no	Object key holding the checkpoint document.
`region`	`str`	null	no	AWS region; omit to use the default-chain region.
`endpoint_url`	`str`	http://minio:9000	no	Custom S3 endpoint for MinIO/R2/Ceph; omit for AWS S3.

GcsStateConfig¶

Field	Type	Default	Required	Description
`bucket`	`str`	""	no	Bucket name (required when state.store is gcs).
`object_name`	`str`	sf2loki/state.json	no	Object name holding the checkpoint document.
`service_file`	`Path`	null	no	Path to a service-account JSON key; omit to use Application Default Credentials.

CoordinateConfig¶

Field	Type	Default	Required	Description
`type`	`Literal['noop', 'file_lease', 'k8s_lease']`	noop	no	noop (single instance) \| file_lease (active-passive via a shared lease file) \| k8s_lease (active-passive via a Kubernetes Lease; needs the sf2loki[k8s] extra).
`file_lease`	`FileLeaseConfig`		no	File-lease coordinator settings.
`k8s_lease`	`K8sLeaseConfig`		no	Kubernetes-Lease coordinator settings.

FileLeaseConfig¶

Field	Type	Default	Required	Description
`path`	`Path`	/var/lib/sf2loki/leader.lease	no	Lease file path on storage shared by all replicas.
`ttl`	`Duration`	30s	no	Lease lifetime: a standby takes over once the lease is this stale. Failover time is bounded by ttl; must exceed inter-host clock skew.
`renew_interval`	`Duration`	10s	no	How often the leader re-writes the lease (must be < ttl/2).
`holder_id`	`str`	""	no	Stable identity written into the lease; defaults to hostname-pid at startup. Set explicitly when hostnames aren't unique.

K8sLeaseConfig¶

Field	Type	Default	Required	Description
`namespace`	`str`	default	no	Namespace holding the Lease object.
`name`	`str`	sf2loki-leader	no	Lease object name (shared by all replicas).
`identity`	`str`	""	no	holderIdentity written into the Lease; defaults to the pod name ($HOSTNAME) at startup.
`lease_duration`	`Duration`	30s	no	Lease lifetime: a standby takes over once the lease is this stale. Failover time is bounded by this.
`renew_interval`	`Duration`	10s	no	How often the leader renews the Lease (must be < lease_duration/2).
`kubeconfig`	`Path`	null	no	Path to a kubeconfig for out-of-cluster dev; omit to use in-cluster config.

ServiceConfig¶

Field	Type	Default	Required	Description
`log_level`	`Literal['debug', 'info', 'warning', 'warn', 'error', 'critical']`	info	no	Application log level: debug \| info \| warning \| error \| critical (case-insensitive).
`log_format`	`Literal['json', 'logfmt']`	json	no	Application log output format.
`health_addr`	`str`	":8080"	no	Address to bind the health-check HTTP server.
`shutdown_grace`	`Duration`	25s	no	Grace period allowed for in-flight work to finish on shutdown.
`unready_after_sink_failing`	`Duration`	15m	no	/readyz reports 503 when Loki pushes have been failing continuously for this long (data is retried and safe, but the instance is degraded and an orchestrator should surface it). 0 disables the readiness degradation.
`telemetry`	`TelemetryConfig`		no	OTLP metrics egress settings.

TelemetryConfig¶

Field	Type	Default	Required	Description
`enabled`	`bool`	false	no	Push all metrics via OTLP/HTTP.
`endpoint`	`str`	https://otlp-gateway-.grafana.net/otlp/v1/metrics	no	Full OTLP/HTTP metrics URL. For Grafana Cloud this is the stack OTLP gateway, e.g. https://otlp-gateway-.grafana.net/otlp/v1/metrics; for a local Alloy otelcol.receiver.otlp, e.g. http://alloy:4318/v1/metrics.
`auth`	`Literal['basic', 'none']`	basic	no	Auth for the OTLP endpoint. "basic" sends Authorization: Basic base64(user:token); "none" sends none (e.g. in-cluster Alloy). For basic, the credentials default to the Loki sink's tenant_id/auth_token when left blank — Grafana Cloud uses one stack credential for both Loki and OTLP.
`basic_auth_user`	`str`	""	no	Basic-auth username; defaults to the Loki sink's tenant_id when blank.
`basic_auth_token`	`SecretStr`	(secret)	no	Basic-auth token, inline; defaults to the Loki sink's auth_token when blank.
`basic_auth_token_file`	`Path`	(secret)	no	File path to the basic-auth token (alternative to basic_auth_token).
`headers`	`dict[str, str]`	{}	no	Explicit headers, merged on top of any computed Authorization header. Values support ${ENV} interpolation at config load.
`export_interval`	`Duration`	1m	no	How often to export accumulated metrics via OTLP.
`resource_attributes`	`dict[str, str]`	{}	no	Extra OTel resource attributes merged onto the defaults (service.name, etc.).