Observability and Tracing

flAPI emits OpenTelemetry traces for every HTTP request on every route, and correlates them with its audit log and application log through one shared request identity.

Tracing is off by default. Turning it on is an explicit operator action, and even then flAPI exports no customer data unless you opt a specific endpoint into payload capture.

Not the same thing as product telemetry

This is a different subsystem from the anonymous product analytics flAPI sends at startup. Different purpose, different consent model, different destination. Neither implies the other, and disabling one does not affect the other.

What you get without configuring anything

Every response carries X-Request-Id, and that id appears in the audit log and in every application log line emitted while serving that request.

curl -i https://flapi.example/customers/42

HTTP/1.1 500 Internal Server Error
X-Request-Id: req-9f2c1a7b8e4d5063

A support report quoting that header is enough to find the request. It is always server-minted — an inbound X-Request-Id is never honoured, so a caller cannot choose their own id or collide with someone else's.

Turning tracing on

tracing:
  enabled: true                 # off by default; nothing else turns it on
  exporter: otlp_http           # otlp_http | otlp_file | none
  endpoint: http://localhost:4318
  capture: metadata             # off | metadata | payload

That is the minimum. Everything else has a working default.

Key	Default	Meaning
enabled	false	Master switch. Nothing is exported until this is true.
service_name	flapi	service.name resource attribute.
service_namespace	—	service.namespace.
exporter	otlp_http	otlp_http, otlp_file, or none. Anything unrecognised behaves as none.
endpoint	SDK default	OTLP/HTTP base URL. otlp_http only.
protocol	unset	Only the exact string http/json changes anything.
headers	{}	Extra headers, e.g. auth for a SaaS backend. otlp_http only.
timeout_ms	10000	Export timeout.
capture	metadata	See Data protection.
openinference	false	OpenInference attribute overlay, for LLM trace tooling.
db_profiling	off	off, summary, detailed. See Inside a slow query.
exclude_routes	probes + docs	Routes that produce no span.
sample.type	parentbased_traceidratio	always_on, always_off, parentbased_traceidratio.
sample.ratio	1.0	Sampling ratio.
flush.mode	batch	batch, or on_response with exporter: otlp_file only.
flush.timeout_ms	2000	Batch interval.
flush.max_queue_size	2048	Queue bound, in spans — not requests.
file.path	traces.jsonl	For exporter: otlp_file, relative to the working directory.
resource_attributes	{}	e.g. deployment.environment.
payload.max_value_bytes	8192	Per-value clamp at the payload tier.

The tracing: block is read once, at startup

There is no hot reload for it. Changing tracing configuration means restarting flAPI.

An explicit exclude_routes list replaces the defaults (/health, /health/live, /mcp/health, /doc, /doc.yaml) rather than extending them, and each entry must match a request path exactly — trailing slash included.

What is traced

One SERVER span per HTTP request, on every route — including requests rejected in middleware (401, 403, 429), CORS preflights, static routes, the config service, and unmatched 404s. Instrumenting only the request handler would miss all of those, which is why the span is created in the first middleware.

Children of that span:

Span	Kind	What it tells you
flapi.render_template	INTERNAL	Mustache rendering, template size
duckdb.query	CLIENT	DuckDB execution, rows returned, SQL verb
outbound GET/POST	CLIENT	flAPI's own OIDC / JWKS calls

Both query paths are covered — the plain one and the prepared path any endpoint with typed request fields takes. A paginated endpoint produces a separate span for its count query, so the two costs are visible apart.

An MCP tools/call is one span carrying both the http.* and the mcp.*/gen_ai.* attribute sets, named tools/call <tool> — not a generic HTTP span with a child.

Route labels are always bounded

http.route is always a template or a fixed literal, never a filled path, and every unmatched path collapses to a single <unmatched> bucket. A vulnerability scanner hitting a thousand random URLs produces one route label, not a thousand — that is both a cardinality and a cost concern. Caller-supplied MCP method and tool names are checked against the configured set before they can become a span name, for the same reason.

Joining an agent's trace

MCP revision 2026-07-28 reserves the unprefixed keys traceparent, tracestate and baggage inside params._meta (SEP-414). flAPI honours them, so one trace can span the agent, flAPI, and the database query underneath.

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "customer_lookup",
    "arguments": { "id": 42 },
    "_meta": {
      "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
    }
  }
}

_meta beats the HTTP traceparent header when both are present: behind a gateway the HTTP hop carries the gateway's span, while _meta carries the agent's — and the agent's is the one you are trying to follow.

This is parsed even when tracing is disabled, because correlating an MCP call into the audit and application logs is useful on its own.

Seeing inside a slow query

db_profiling attaches DuckDB's own execution metrics — the numbers behind EXPLAIN ANALYZE — to the database span.

tracing:
  db_profiling: summary     # off (default) | summary | detailed

Attribute	Tier	Meaning
flapi.db.latency_ms	summary	Query execution time as DuckDB measures it
flapi.db.blocked_thread_time_ms	summary	Time blocked rather than working
flapi.db.result_set_bytes	summary	Materialised result size
flapi.db.bytes_read	summary	Bytes read from storage
flapi.db.cpu_time_ms	detailed	CPU across all threads
flapi.db.rows_scanned	detailed	Rows scanned, cumulative

A metric DuckDB does not report is omitted, never exported as 0 — a fabricated zero reads as "instant" on every dashboard.

The two tiers cost the same. Measured on a trivial query, interleaved across configurations: off 253 µs, summary 359 µs, detailed 336 µs for the database span. detailed measuring lower is not a saving, it is the difference being below the measurement's own noise. Choose a tier by what you want to see, not to save time.

Nearly all of that ~100 µs is switching profiling on rather than measuring: DuckDB's profiling settings are per-connection, and flAPI opens a connection per query. The overhead is fixed per query, so on a request doing real work it does not show; a paginated endpoint runs two queries and pays it twice. It applies only to requests that are actually sampled.

Data protection

Tier	Exports	Never exports
off	nothing; no provider is constructed	—
metadata (default)	span structure and timings, route templates, method and status, User-Agent, auth kind, tool and MCP method names, the SQL verb, rows returned, byte counts, template basename, bound-parameter count, enumerated error kinds	any argument value, any result row, any filled path or query string, any other header value, any credential
payload	metadata plus the values of declared request fields	credentials, headers, filled paths and query strings — excluded at every tier

Four properties are enforced by tests rather than asserted in prose:

1. A global capture: off beats any per-endpoint opt-in. One lever is guaranteed to stop export.
2. Credential-shaped keys are redacted at every tier, regardless of your configuration.
3. Filled paths and query strings are never exported, at any tier. A query string on a data API is by definition a filter over customer data.
4. The payload tier genuinely captures — without a test asserting a declared parameter value does appear, the three exclusions above would pass simply because nothing was captured at all.

Error status is always an enumerated error.type, never an exception message — that is the most reliable way to leak a row value into a trace.

Redaction

Two lists, matched differently on purpose:

List	Match	Effect
Built-in credential stems	Substring of the normalised key	api_key, x-api-key, user_api_key, auth_token are all redacted. You cannot forget one.
Your audit.redact entries	Whole normalised key	tax_id redacts tax_id and Tax-Id, but not customer_tax_id.

"Normalised" means lower-cased with - and _ removed, so case and separator style never matter. Redaction happens before clamping, so a truncated value cannot leave a partial secret behind.

The same list serves the audit log and the trace, so you configure it once and the two cannot drift apart.

Per-endpoint opt-in

mcp-tool:
  name: customer_lookup
  response:
    redact-columns: [email, tax_id]
  tracing:
    capture: payload        # or `off`, to exclude a sensitive endpoint

Payload capture has legal implications

At the payload tier flAPI becomes a processor exporting personal data to a third destination, with DPA/AVV consequences you need to have considered. flAPI logs a warning at startup saying exactly that.

Deployment

Kubernetes / on-prem

flush.mode: batch, exporter pointed at a collector. Set service.instance.id from the pod name via resource_attributes:

tracing:
  enabled: true
  exporter: otlp_http
  endpoint: http://otel-collector.observability:4318
  resource_attributes:
    deployment.environment: production

Serverless (Cloud Run, App Runner, Lambda)

With CPU allocated only during request processing, the instance is throttled after the response is sent, so a background export thread may never be scheduled and spans are lost.

Allocate CPU always. That is the only remedy that works with an OTLP/HTTP collector. flush.mode: on_response is not an alternative here: it is honoured only with exporter: otlp_file, and with otlp_http flAPI logs a warning and falls back to batch. Flushing a network export on the request thread would couple your p99 to the collector's availability, so it is refused rather than supported.

SIGTERM force-flushes on shutdown.

Air-gapped — the zero-egress option

tracing:
  enabled: true
  exporter: otlp_file
  file:
    path: /var/lib/flapi/traces.jsonl
  flush:
    mode: on_response

With exporter: otlp_file, and the product telemetry disabled, flAPI opens no outbound network connection for observability. (Authentication is separate: if you configure OIDC, JWKS fetches still dial out.) Log rotation is your existing tooling's job.

Checking that export is working

GET /api/v1/_config/metrics reports whether spans are reaching the collector. It requires the config-service token.

curl -s -H "Authorization: Bearer $FLAPI_CONFIG_SERVICE_TOKEN" \
     http://localhost:8080/api/v1/_config/metrics

{
  "tracing": { "enabled": true, "spans_exported": 10482, "spans_dropped": 0 },
  "arrow":   { "total_requests": 12, "successful_requests": 12, "failed_requests": 0,
               "total_rows": 48210, "active_streams": 0 },
  "endpoints": { "count": 18 }
}

spans_dropped counts failed export batches — an unreachable, erroring or timing-out collector.

A flat spans_dropped does not prove nothing was lost

Spans discarded because the batch queue was already full are dropped before export is attempted, and are not counted here. If spans_exported is lower than your request rate implies while spans_dropped stays at zero, suspect flush.max_queue_size rather than the collector.

Correlation

trace_id and span_id appear in the audit log and in application log lines emitted while serving a request, so a trace joins to an audit entry and to the log. With log-format: json the log is directly ingestible.

log-level: info
log-format: json     # request_id / trace_id / span_id on every line
audit:
  enabled: true
  sink: file
  path: /var/log/flapi/audit.jsonl

A traced response also carries X-Trace-Id, and it is the caller's trace id when the caller supplied one, so both sides join on the same value.

Environment variables

Configuration is YAML-first. Only two environment paths affect tracing:

Variable	Effect
OTEL_SDK_DISABLED	Exactly true disables tracing even when tracing.enabled: true. Always wins.
OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS	Used as the exporter's defaults only when the corresponding YAML keys are absent.

Everything else is YAML-only — OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER, OTEL_TRACES_EXPORTER and the OTEL_BSP_* batch knobs have no effect. If you rely on Kubernetes OTel Operator injection, set the equivalents in tracing: explicitly rather than assuming the injected environment is read.

One deliberate divergence: an OTEL_EXPORTER_OTLP_ENDPOINT in the environment does not by itself enable tracing. A platform-wide environment variable is not an operator's consent to ship data off the machine, so tracing.enabled is still required.

Building without tracing

cmake -DFLAPI_WITH_TRACING=OFF ...

The facade compiles to no-ops and no OpenTelemetry symbol is linked. Request ids, log correlation, the REST audit log and /api/v1/_config/metrics all still work — they are not tracing features.

Further reading

• docs/OBSERVABILITY.md — upstream operator reference
• docs/spec/components/observability.md — how it is implemented
• Configuration — the audit log block
• Deployment — platform-specific notes