Explain The Collector — Pipelines in Observability

3 / 4

OpenTelemetry

The Collector — Pipelines

Run a vendor-neutral pipeline to receive, process, and export telemetry.

The OpenTelemetry Collector is a standalone service that sits between your instrumented apps and your backends, decoupling them. It is built from three pluggable stages: receivers ingest data (OTLP, plus Prometheus, Jaeger, Zipkin, host metrics and more), processors transform it (batching, memory limiting, attribute editing, filtering, and tail-based sampling), and exporters send it onward to one or many backends. You assemble these into pipelines per signal in a single config file.

A mail distribution centre: it accepts parcels in any format (receivers), repackages, labels, and screens them (processors), then routes each to the right destination courier (exporters) — senders never deal with the destinations directly.

Key Concepts

Two deployment patterns dominate. As an agent, the Collector runs on each host or as a Kubernetes DaemonSet/sidecar close to the apps. As a gateway, a central Collector cluster receives from many agents and does heavier processing before export. The Collector is where you implement tail-based sampling (decide to keep a trace after seeing all its spans, so you keep the slow/error ones), scrub sensitive attributes, add metadata, and fan out to multiple backends — all without touching application code.

What the interviewer is probing.

What the interviewer is probing.

1. "What is the collector for?" *Probing:* the purpose. *Stalls:* "Collecting." *Moves up:* receiving, processing and exporting telemetry outside the application, so sampling, redaction and routing change without redeploying services.

1. "What is the collector for?"

2. "What are the pipeline stages?" *Probing:* the structure. *Stalls:* "In and out." *Moves up:* receivers, processors and exporters, composed per signal type.

2. "What are the pipeline stages?"

3. "What is the difference between agent and gateway deployment?" *Probing:* the architecture. *Stalls:* "Nothing." *Moves up:* agent runs beside each workload for local collection; a gateway tier centralises processing such as tail-based sampling that needs the whole trace.

3. "What is the difference between agent and gateway deployment?"

4. "Why can tail-based sampling not run on the agent?" *Probing:* the depth. *Stalls:* "It can." *Moves up:* it needs every span of a trace in one place to decide, and those spans come from different hosts.

4. "Why can tail-based sampling not run on the agent?"