OpenTelemetry Collector Pipeline Configuration Guide

The OpenTelemetry Collector is a vendor-neutral pipeline that ingests telemetry, transforms it and exports it somewhere useful. Its configuration is deceptively simple — four component classes and a service section — and most production incidents around it come from two mistakes: processors applied in the wrong order, and pipelines that reference components nobody defined. This guide walks the structure, the ordering rules that the collector actually enforces, and a complete configuration you can start from.

The Four Component Classes

• Receivers — how data gets in: otlp (gRPC 4317 / HTTP 4318), prometheus, filelog, fluentforward, zipkin, hostmetrics.
• Processors — optional transforms applied in the exact order listed in the pipeline: memory_limiter, k8sattributes, resourcedetection, attributes, filter, transform, batch.
• Exporters — where data goes: otlp (gRPC), otlphttp, prometheusremotewrite, loki, debug.
• Connectors — components that are both exporter and receiver, joining two pipelines; useful for deriving metrics from traces or logs from metrics.

Components are identified as type[/name], so you can define several receivers of the same type as long as the names are unique: otlp, otlp/2, otlphttp/internal.

Processor Ordering Rules

1. memory_limiter first — it must be able to reject data before expensive processing begins.
2. Enrichment that needs request context (k8sattributes, resourcedetection) next, because batch clears that context.
3. Filtering and transformation (filter, attributes, transform) next, so you enrich only what you keep.
4. batch last — batching data that is about to be dropped or mutated wastes both memory and CPU.

A Complete Working Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  filelog:
    include: [ /var/log/app/*.log ]
    start_at: end
  hostmetrics:
    collection_interval: 30s
    scrapers: [ cpu, memory, disk, filesystem, network, load ]

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  resourcedetection:
    detectors: [ env, system ]
  attributes/scrub:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: service.environment
        value: production
        action: upsert
  filter/drop_health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'
  batch:
    send_batch_size: 1024
    timeout: 5s

exporters:
  otlphttp/backend:
    endpoint: https://telemetry.example.com:4318
    headers:
      x-api-key: ${env:OTEL_API_KEY}
  prometheusremotewrite:
    endpoint: https://prom.example.com/api/v1/write
  debug:
    verbosity: basic

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  pprof:
    endpoint: 127.0.0.1:1777
  zpages:
    endpoint: 127.0.0.1:55679

service:
  extensions: [ health_check, pprof, zpages ]
  telemetry:
    logs:
      level: info
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ memory_limiter, resourcedetection, attributes/scrub, filter/drop_health, batch ]
      exporters: [ otlphttp/backend ]
    metrics:
      receivers: [ otlp, hostmetrics ]
      processors: [ memory_limiter, resourcedetection, batch ]
      exporters: [ prometheusremotewrite ]
    logs:
      receivers: [ otlp, filelog ]
      processors: [ memory_limiter, resourcedetection, batch ]
      exporters: [ otlphttp/backend ]

Validate Before You Ship

otelcol --config=customconfig.yaml --dry-run   # or: validate
curl -s http://127.0.0.1:13133/                # health_check
curl -s http://127.0.0.1:55679/debug/pipelinez | head

The zpages extension is the fastest way to debug a silent pipeline: it shows which components are running and whether data is flowing. If the collector starts but nothing arrives at the backend, check the pipeline's exporter list first, then the backend's authentication headers — a wrong API key typically surfaces as repeated retry logs rather than a hard failure.

Related reading: Fluent Bit to Elasticsearch pipeline and Alertmanager routing configuration.

原文链接：https://opentelemetry.io/docs/collector/configuration