Datadog
Log Management & Pipelines
Ingest, parse, and control the cost of logs while keeping them searchable and correlated.
Datadog Log Management ingests logs from the Agent, cloud services, and libraries, then runs them through processing pipelines that parse raw text into structured attributes (status, duration, user, trace_id). Structured logs are what make full-text and faceted search fast and what enables one-click pivots between a log line and its trace.
A mailroom that opens every letter and files the important ones in fast-access drawers while boxing the rest in cheap off-site storage — retrievable later if an audit needs them.
Key Concepts
1
A key concept is the split between ingestion and indexing. Everything you send is ingested and can be processed, but you choose — via index filters and exclusion rules — which logs are actually indexed (made searchable and retained) versus archived cheaply to object storage. Patterns and Log Rehydration let you keep costs down while still being able to bring archived logs back when an investigation needs them.
2
What the interviewer is probing.
What the interviewer is probing.
3
1. "What does a log pipeline do?" *Probing:* the purpose. *Stalls:* "Stores logs." *Moves up:*
parses, enriches, remaps attributes and can redact, so logs arrive structured and queryable rather
than as raw text.
1. "What does a log pipeline do?"
4
2. "Why is structured logging worth the effort?" *Probing:* the argument. *Stalls:* "It is
tidy." *Moves up:* you can filter and aggregate by field — with free text you are reduced to
substring search, which does not scale.
2. "Why is structured logging worth the effort?"
5
3. "How do you control log cost?" *Probing:* the commercial reality. *Stalls:* "Log less."
*Moves up:* indexing only what is queried and archiving the rest, with exclusion filters on
high-volume low-value lines.
3. "How do you control log cost?"
6
4. "How do logs connect to traces?" *Probing:* the integration. *Stalls:* "They do not." *Moves
up:* injecting trace and span ids into log lines, so one click moves from a slow trace to exactly
its log output.
4. "How do logs connect to traces?"