Distributed Tracing

Pipelines-as-Code emits trace spans for webhook event processing and PipelineRun lifecycle timing.

Enabling tracing

Two configuration paths can enable tracing.

Via OpenTelemetry environment variables

Set on the controller and watcher pods:

• OTEL_EXPORTER_OTLP_ENDPOINT - OTLP collector endpoint URL. Required.
• OTEL_TRACES_SAMPLER - Sampler family. Required. Supported: always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, parentbased_traceidratio.
• OTEL_TRACES_SAMPLER_ARG - Numeric argument for ratio samplers. Example: 0.1 with parentbased_traceidratio samples 10% of root traces while keeping the chain coherent.
• OTEL_EXPORTER_OTLP_PROTOCOL (or traces-specific OTEL_EXPORTER_OTLP_TRACES_PROTOCOL) - OTLP transport: grpc or http/protobuf. Default: grpc.

Both OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_TRACES_SAMPLER must be set. Inbound traceparent headers on webhook requests are honored via the W3C TraceContext propagator. Changes take effect on the next pod restart.

Sampler choice and chain coherency

The parentbased_* sampler family honors the parent span’s sample decision carried in the W3C traceparent flag bit. When every service in the delivery chain uses parent-based samplers, the root span’s sampling decision propagates end to end: each service either keeps its spans or drops them based on what the root chose. Flat-rate samplers (traceidratio without parent-based) cause each service to roll independently, which at fractional sampling fragments the chain into orphaned spans whose parent_spanID references a span that was dropped. parentbased_always_on keeps everything; parentbased_traceidratio with a numeric argument samples a coherent fraction.

Via Knative observability ConfigMap

Set in pipelines-as-code-config-observability:

• tracing-protocol - grpc, http/protobuf, stdout, or none. Default: none.
• tracing-endpoint - Collector endpoint for grpc or http/protobuf.
• tracing-sampling-rate - Sample fraction. Per-component independent.

Changes to Knative’s tracing config require restarting the controller and watcher pods. The tracer is built once at startup.

When both are configured

OpenTelemetry takes precedence when set: all spans flow through the OpenTelemetry exporter. With OpenTelemetry unset, Knative’s tracer (from the ConfigMap above) remains as the runtime tracer.

To use only OpenTelemetry, set tracing-protocol: none in pipelines-as-code-config-observability.

To use only Knative, unset OTEL_EXPORTER_OTLP_ENDPOINT on the controller and watcher pods.

Emitted spans

The controller emits a PipelinesAsCode:ProcessEvent span for each webhook event. The watcher emits waitDuration and executeDuration spans for completed PipelineRuns. The OTel resource attribute service.name on all emitted spans is pipelines-as-code.

Webhook event span (PipelinesAsCode:ProcessEvent)

OTel VCS semantic conventions:

PaC-specific:

PipelineRun timing spans (waitDuration, executeDuration)

Tekton-compatible bare keys (match Tekton’s own reconciler spans for correlation):

Cross-service delivery attributes (delivery.tekton.dev.*):

Additional delivery.tekton.dev.* attributes are sourced from configurable PipelineRun labels.

OTel CI/CD semantic conventions (executeDuration only):

cicd.pipeline.result enum

Configuring label-sourced attributes

Some span attributes are read from PipelineRun labels. The label names are configurable via the main pipelines-as-code ConfigMap so deployments can point at their existing labels without rewriting producers:

Setting a ConfigMap key to the empty string disables emission of that label-sourced attribute. Only label-sourced attributes are affected; all other span attributes are always emitted. The emitted span attribute keys are fixed regardless of which labels are read, so cross-service queries work uniformly.

Unlike the observability ConfigMap above (which requires a pod restart), changes to these label mappings are picked up automatically without restarting pods.

Trace context propagation

When Pipelines-as-Code creates a PipelineRun, it sets the tekton.dev/pipelinerunSpanContext annotation with a JSON-encoded OTel TextMapCarrier containing the W3C traceparent. PaC tracing works independently - you get PaC spans regardless of whether Tekton Pipelines has tracing enabled.

If Tekton Pipelines is also configured with tracing pointing at the same collector, its reconciler spans appear as children of the PaC span, providing a single end-to-end trace from webhook receipt through task execution. See the Tekton Pipelines tracing documentation for Tekton’s independent tracing setup.

Deploying a trace collector

Pipelines-as-Code exports traces using the standard OpenTelemetry Protocol (OTLP). You need a running OTLP-compatible collector for OTEL_EXPORTER_OTLP_ENDPOINT to point to. Common options include:

• OpenTelemetry Collector – the vendor-neutral reference collector
• Jaeger – supports OTLP ingestion natively since v1.35

Deploying and operating a collector is outside the scope of Pipelines-as-Code. Refer to your organization’s observability infrastructure or the links above for setup instructions.