otel-ottl

---
name: otel-ottl
description: OpenTelemetry Transformation Language (OTTL) expert for writing and debugging telemetry transformations in the OpenTelemetry Collector. Use when authoring or reviewing `transform`, `filter`, `tail_sampling` processor configs or `routing` connector configs, debugging OTTL syntax or semantics, transforming traces, metrics, logs, or profiles, or converting data-processing requirements into OTTL statements.
---

# OpenTelemetry Transformation Language (OTTL)

OTTL is a domain-specific language for transforming telemetry inside the OpenTelemetry Collector. It is consumed by the `transform`, `filter`, and `tail_sampling` processors, the `routing` connector, and a few other components in [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl).

This skill targets `pkg/ottl` as of collector-contrib **v0.156.0**. Function and path availability differs across Collector releases; check the upstream `pkg/ottl/ottlfuncs/README.md` and `pkg/ottl/contexts/*/README.md` for the exact set in older or newer releases.

## Statement syntax

```ottl
function(arguments) [where condition]
```

Every statement has exactly one **editor** (lowercase: `set`, `delete_key`, `append`, …) optionally guarded by a `where` clause whose body is a boolean expression. Conditions can call **converters** (uppercase: `Concat`, `IsMatch`, `ParseJSON`, …) which return values but do not mutate telemetry.

```ottl
set(span.attributes["env"], "prod") where resource.attributes["env"] == nil
```

## Workflow

1. **Pick the component.** `transform` rewrites; `filter` drops; the `routing` connector fans out by pipeline; `tail_sampling` keeps/drops traces. The component decides which contexts and function set are usable.
2. **Pick the context.** `resource`, `scope`, `span`, `spanevent`, `metric`, `datapoint`, `exemplar`, `log`, `profile`, `profilesample`, or `otelcol` for Collector request/client metadata. Operate at the lowest level that gives you the data — using `datapoint` to set attributes is much cheaper than walking through `metric.data_points` from the metric context.
3. **Write statements.** Reach for `references/quick-reference.md` for common recipes; `references/contexts.md` for paths/enums; `references/functions.md` for the editor and converter catalog.
4. **Set `error_mode`.** `ignore` keeps the pipeline running and logs errors; `silent` does the same but quietly; `propagate` aborts on first failure. In v0.156, `transform` and `filter` default to `ignore` through enabled beta feature gates; `routing` defaults to `propagate` unless `connector.routing.defaultErrorModeIgnore` is enabled; lower-level OTTL sequences default to `propagate`.
5. **Verify.** OTTL gotchas are the kind that pass the eye test (see [Common gotchas](#common-gotchas)). Use the [telemetrygen verification recipe](../otel-telemetrygen/SKILL.md#verifying-a-collector-config) — `otelcol-contrib` + file exporter + telemetrygen — to confirm the snippet does what the prose claims before shipping.

## Contexts at a glance

OTTL paths are scoped by signal. Higher levels are reachable from lower ones (e.g., `resource.attributes` from a span statement); the reverse is not true.

| Context | Common paths |
|---------|--------------|
| Resource | `resource.attributes["service.name"]`, `resource.schema_url` |
| Scope | `scope.name`, `scope.version`, `scope.attributes["…"]` |
| Span | `span.name`, `span.kind`, `span.status.code`, `span.attributes["…"]`, `span.flags` |
| Span Event | `spanevent.name`, `spanevent.attributes["…"]`, `spanevent.event_index` |
| Metric | `metric.name`, `metric.unit`, `metric.type`, `metric.aggregation_temporality` |
| DataPoint | `datapoint.value_double`, `datapoint.value_int`, `datapoint.attributes["…"]` |
| Exemplar | `exemplar.double_value`, `exemplar.int_value`, `exemplar.filtered_attributes["…"]` (transform `metric_statements` only, v0.156+) |
| Log | `log.body`, `log.body.string`, `log.severity_number`, `log.attributes["…"]` |
| Profile | `profile.profile_id`, `profile.attributes["…"]` (Development) |
| OTelCol | `otelcol.client.metadata["x-tenant"][0]`, `otelcol.grpc.metadata["x-tenant"][0]` (read-only, enabled by default feature gate) |

Full path inventory plus enums in `references/contexts.md`.

## Essential functions

```ottl
# Editors (mutate telemetry)
set(target, value)
delete_key(target, key)               # delete by exact key
delete_matching_keys(target, regex)   # delete by regex
delete_index(target, start, end?)     # remove one slice item or range (v0.145+)
keep_keys(target, [k1, k2])           # keep only these keys
merge_maps(target, source, "upsert")  # "insert" | "update" | "upsert"
truncate_all(target, max_len)         # UTF-8 safe by default in v0.148+
replace_pattern(target, regex, replacement)
stringify_all(target)                 # map values -> strings (v0.155+)

# Converters (return values)
Concat([a, b], "-")
Split(s, ",")
ToLowerCase(s) / ToUpperCase(s)
IsMatch(s, "pattern")                 # bool
String(v) / Int(v) / Double(v) / Bool(v)
ParseJSON(s) / ParseKeyValue(s, "=", "&")
URL(s)                                # parse URL components (v0.127+)
ExtractPatterns(s, "(?P<name>…)")     # named captures → map
ExtractGrokPatterns(s, "%{IP:client}") # Grok (v0.130+)
IsInCIDR(ip, ["10.0.0.0/8"])          # CIDR membership (v0.146+)
SHA256(s) / Murmur3Hash(s) / XXH3(s)  # hexadecimal string hashes
UUID() / UUIDv7()
```

Converter results can be indexed by dynamic string or integer expressions in v0.155+:

```ottl
set(log.attributes["last"], Split(log.body.string, ",")[Len(Split(log.body.string, ",")) - 1])
set(log.attributes["selected"], ParseJSON(log.body.string)[log.attributes["field_name"]])
```

Full catalog with signatures in `references/functions.md`.

The `transform` processor also provides 17 signal-specific functions that are not
part of the common `pkg/ottl/ottlfuncs` catalog. See the
[transform-only function table](references/functions.md#transform-processor-only-functions)
for their contexts, signatures, and released-source links.

## Common patterns

```ottl
# Conditional set
set(span.attributes["sampled"], true)
    where (span.end_time_unix_nano - span.start_time_unix_nano) > 1000000000

# Normalize a value
set(span.attributes["http.method"],
    ToUpperCase(String(span.attributes["http.method"])))

# Parse a JSON log body into structured attributes
set(log.attributes, ParseJSON(log.body.string))
    where IsString(log.body) and IsMatch(log.body.string, "^\\s*\\{.*\\}\\s*$")

# Mark errored HTTP spans
set(span.status.code, STATUS_CODE_ERROR)
    where IsInt(span.attributes["http.status_code"])
      and Int(span.attributes["http.status_code"]) >= 400

# Redact secrets by key pattern
delete_matching_keys(span.attributes, "(?i).*(password|secret|token|apikey).*")

# Hash PII rather than dropping it (preserves cardinality for analytics)
set(span.attributes["user.email_hash"], SHA256(span.attributes["user.email"]))
delete_key(span.attributes, "user.email")
```

More recipes (sampling, redaction, time math, parsing) in `references/quick-reference.md`.

## Processor wiring

```yaml
processors:
  transform:
    error_mode: ignore        # ignore | silent | propagate
    trace_statements:
      - context: span
        statements:
          - set(span.attributes["processed"], true)
    log_statements:
      - context: log
        statements:
          - set(log.attributes["source"], "collector")
    metric_statements:
      - context: datapoint
        statements:
          - set(datapoint.attributes["env"], "prod")

  filter:
    error_mode: ignore
    trace_conditions:
      - 'IsMatch(span.name, "^/health.*")'
    log_conditions:
      - 'log.severity_number < SEVERITY_NUMBER_WARN'

  tail_sampling:
    policies:
      - name: errors
        type: ottl_condition
        ottl_condition:
          span:
            - 'span.status.code == STATUS_CODE_ERROR'

connectors:
  routing:
    error_mode: ignore
    default_pipelines: [traces/default]
    table:
      - condition: 'resource.attributes["env"] == "prod"'
        pipelines: [traces/prod]
      - condition: 'span.status.code == STATUS_CODE_ERROR'
        pipelines: [traces/errors]
```

## Common gotchas

These mistakes pass YAML validation but break OTTL semantics. Most have cost real time in production rollouts.

### `replace_pattern` backreferences need `$${1}` in YAML

OTTL uses `${1}`, `${2}`, … for regex backreferences. The collector's YAML loader treats `$` as an env-var marker, so YAML `$$` becomes OTTL `$`. To produce `${1}` at the OTTL level, write `$${1}` in YAML. Writing `"$1REDACTED"` produces literal `$1REDACTED` with no replacement — silent failure.

### Go RE2 has a repeat-count ceiling

Patterns like `(.{1024}).*` fail to compile with "invalid repeat count". For length-based truncation, prefer `Substring` + `Len`:

```ottl
set(attributes["db.statement"], Substring(attributes["db.statement"], 0, 1024))
    where attributes["db.statement"] != nil and Len(attributes["db.statement"]) > 1024
```

Or use `truncate_all` for whole maps (UTF-8 safe by default since v0.148):

```ottl
truncate_all(span.attributes, 1024)
```

### `attributes` processor ≠ `resource` processor

The OTel `attributes` processor only operates on span/log/metric attributes. To touch a *resource* attribute use the `resource` processor or a `transform` processor with `context: resource`. A config like `attributes/strip_resource: actions: [...delete os.description...]` runs without error but doesn't change resource attributes — silent no-op.

### Per-log resource/scope rewrites may need `flatten_data`

Log records can share resource and scope data. When a log-context statement derives either from record-specific `log.*` data, set `flatten_data: true` and enable the alpha `transform.flatten.logs` feature gate; otherwise records in the same batch can affect one another unexpectedly. In v0.156 the gate is disabled and `flatten_data` is `false` by default. The option is logs-only and incurs copying, hashing, and regrouping overhead.

### `logdedup` paths use dot-notation only

The processor accepts `include_fields` / `exclude_fields` (not `fields`). Paths must start with `attributes.` or `body.` and use dot-notation. Bracket notation (`attributes["service.name"]`) is rejected, and `resource[...]` paths are not addressable. Default behavior dedups on the full record, which is usually what's wanted.

### `k8sattributes` cannot extract `k8s.cluster.name`

The processor's `metadata` list is restricted to pod-level identity. Cluster name has to come from `resourcedetection` or a static `resource` processor that reads it from an env var.

### Path syntax changed in v0.120

In older configs you may see `span_event.*` — current syntax is `spanevent.*`. Cache paths now require the context prefix: write `span.cache["x"]` not just `cache["x"]`. Plain `cache` is only valid in profile/profilesample contexts where it's the documented path. Paste-from-old-config is the most common source of regressions.

### `Base64Decode` is deprecated

Use `Decode(value, "base64")` instead. The same `Decode` converter handles `base64-raw`, `base64-url`, `base64-raw-url`, and IANA character set encodings. Keep `Base64Decode` only if pinned to a pre-v0.141 collector.

### `routing` request context is deprecated

In routing connector configs, use `otelcol.client.metadata["key"][0]` for HTTP/client metadata or `otelcol.grpc.metadata["key"][0]` for gRPC metadata. The old `context: request` plus `request["key"] == "value"` form is deprecated in v0.156.

### `Bool` converter coercion is loose

`Bool` preserves booleans, treats any non-zero integer or double as `true`, and uses Go boolean parsing for strings (`1`, `t`, `T`, `TRUE`, `true`, `True`, and their false equivalents). Invalid strings and unsupported types error; `nil` returns `nil`. Don't assume Python-like truthiness for arbitrary strings.

### Verify before publishing

YAML/OTTL gotchas like the above pass the eye test. Use the [telemetrygen verification recipe](../otel-telemetrygen/SKILL.md#verifying-a-collector-config) (otelcol-contrib + file exporter + telemetrygen) to confirm the snippet does what the surrounding prose claims, especially before shipping to a customer or production.

## Best practices

1. **Validate inputs** before conversion: `where IsString(x)`, `where x != nil`.
2. **Order conditions by selectivity** (cheap and most-selective first). `span.kind == SPAN_KIND_SERVER and IsMatch(...)` lets the kind check short-circuit before the regex runs.
3. **Cache expensive operations** in `<context>.cache`: `set(span.cache["url_parts"], Split(span.attributes["http.url"], "/"))`, then read from cache.
4. **Use the most specific context.** `datapoint` for metric-attribute work beats walking `metric.data_points` from the metric context.
5. **Escape regex correctly.** Use `\\d+`, `\\.`, `\\s+` in OTTL strings (single backslash in YAML becomes a literal).
6. **Prefer `keep_keys` to a long list of `delete_key`** when shaping output — easier to read, fails closed.

## Versioning notes

Recently added (still useful to know which release introduced them when supporting users on older collectors):

| Feature | Since |
|---------|-------|
| Exemplar context (transform `metric_statements` only) | v0.156 |
| Routing connector context inference (`condition` with context-qualified paths) | v0.156 |
| Routing connector `request` context deprecated; use `otelcol.*` metadata paths | v0.156 |
| Log-only `flatten_data` via alpha `transform.flatten.logs` feature gate | v0.103 |
| `connector.routing.defaultErrorModeIgnore` feature gate | v0.155 |
| `otelcol.*` context via enabled-by-default `ottl.contexts.enableOTelColContext` feature gate | v0.147 |
| Filter processor top-level `trace_conditions`, `metric_conditions`, `log_conditions` | v0.146 |
| Profile / ProfileSample contexts | v0.124 / v0.132 (Development) |
| Cache paths require context prefix; `spanevent` rename | v0.120 (breaking) |
| `delete_index` editor | v0.145 |
| `span.flags` path | v0.145 |
| `truncate_all` UTF-8 safe default (`utf8_safe` parameter) | v0.148 (behavior change) |
| `Substring` optional `utf8_safe` parameter (default `false`) | v0.156 |
| `SpanID` / `TraceID` accept hex strings | v0.142 |
| `flatten` `resolveConflicts` parameter | v0.139 |
| `Base64Encode` | v0.147 |
| `Decode`, deprecates `Base64Decode` | v0.141 |
| `Bool` converter | v0.143 |
| `ExtractGrokPatterns` | v0.130 |
| `URL`, `UserAgent` | v0.127, v0.134 |
| `IsInCIDR` | v0.146 |
| `Murmur3Hash*`, `XXH3`, `XXH128` | v0.129, v0.135 |
| `Sort`, `Index`, `SliceToMap` | v0.125, v0.126, v0.128 |
| `UUIDv7`, `ParseSeverity`, `CommunityID` | v0.138, v0.133, v0.131 |
| `stringify_all` editor; dynamic keys when indexing converter results | v0.155 |

## References

- `references/contexts.md` — context paths and enums
- `references/functions.md` — editors and converters with signatures
- `references/quick-reference.md` — recipes, regex patterns, troubleshooting
- Upstream: <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl>