Prometheus
Alerting, Rules & Long-term Storage
Evaluate alert rules, route them via Alertmanager, and extend retention beyond one node.
Prometheus separates alert evaluation from alert handling. Recording rules pre-compute expensive PromQL expressions on a schedule and store the result as a new series, which speeds dashboards. Alerting rules evaluate a PromQL condition (e.g. error ratio > 1% for 10m) and, when it holds, emit an alert to Alertmanager. The "for" clause prevents flapping by requiring the condition to persist.
Prometheus is a sensor that raises its hand; Alertmanager is the dispatcher that groups the raised hands, ignores duplicates, and calls the right on-call team — while a warehouse (remote storage) keeps the long-term archive.
Key Concepts
1
Alertmanager is a separate component that receives alerts and decides what humans see: it groups related alerts, deduplicates identical ones from HA pairs, applies silences and inhibition (suppress a noisy alert when a bigger one fires), and routes by labels to receivers like PagerDuty or Slack. For history beyond a single node's disk, remote_write streams samples to scalable backends — Thanos, Cortex, Grafana Mimir, or vendor stores — giving long-term, horizontally scalable storage while Prometheus stays the collection engine.
2
What the interviewer is probing.
What the interviewer is probing.
3
1. "What does Alertmanager do that Prometheus does not?" *Probing:* the separation. *Stalls:*
"Alerting." *Moves up:* grouping, deduplication, silencing, inhibition and routing — Prometheus only
evaluates the rule and fires.
1. "What does Alertmanager do that Prometheus does not?"
4
2. "What is the for clause on an alert rule?" *Probing:* the detail. *Stalls:* "A delay." *Moves
up:* the condition must hold for that duration before firing, which is what stops a one-scrape blip
paging anyone.
2. "What is the for clause on an alert rule?"
5
3. "What is a recording rule for?" *Probing:* the performance angle. *Stalls:* "Nothing." *Moves
up:* precomputing an expensive expression on a schedule, so dashboards query a cheap series instead.
3. "What is a recording rule for?"
6
4. "How do you keep metrics beyond local retention?" *Probing:* the architecture. *Stalls:*
"Bigger disks." *Moves up:* remote write to Thanos, Mimir or Cortex, which also gives a global query
view across Prometheus instances.
4. "How do you keep metrics beyond local retention?"