Datadog
Monitors, SLOs & Alerting
Turn signals into alerts that page the right people and track service-level objectives.
A monitor in Datadog is a saved query plus a condition that evaluates continuously and changes state (OK, Warn, Alert, No Data). Monitor types cover metrics, logs, APM, anomalies, forecasts, outliers, and composite logic, so you can alert on a threshold, a sudden change, or a predicted breach. Notifications route through integrations (PagerDuty, Slack, email) using template variables and tag-based routing.
A smoke detector tuned to actual fires, not toast. And a monthly "budget" of allowed alarms — once you have used too many too fast, you know to stop shipping risky changes.
Key Concepts
1
On top of monitors, SLOs let you express reliability as a target — e.g. 99.9% of requests succeed over 30 days — backed by either metric ratios or monitor uptime. The remaining error budget tells you how much failure you can still absorb, which turns alerting from "something is wrong" into "are we spending reliability faster than allowed". Good alerting minimises noise: alert on symptoms users feel, not every underlying cause.
2
What the interviewer is probing.
What the interviewer is probing.
3
1. "What makes a good alert?" *Probing:* the discipline. *Stalls:* "It catches problems." *Moves
up:* it is actionable, urgent and user-visible — if nobody does anything at 3am, it should be a
dashboard, not a page.
1. "What makes a good alert?"
4
2. "What is an error budget?" *Probing:* the SLO concept. *Stalls:* "A limit." *Moves up:* the
allowed unreliability under the SLO — spending it is the signal to stop shipping features and fix
reliability.
2. "What is an error budget?"
5
3. "Why alert on symptoms rather than causes?" *Probing:* the principle. *Stalls:* "Causes are
better." *Moves up:* users feel latency and errors; CPU being high may be fine. Symptom alerts catch
causes you did not anticipate.
3. "Why alert on symptoms rather than causes?"
6
4. "What is alert fatigue and how do you fix it?" *Probing:* the practical angle. *Stalls:*
"Ignore them." *Moves up:* too many low-value alerts train people to dismiss everything — fixed by
deleting alerts, not by adding more routing.
4. "What is alert fatigue and how do you fix it?"