Prometheus
Alerting, Rules & Long-term Storage
Evaluate alert rules, route them via Alertmanager, and extend retention beyond one node.
Prometheus separates alert evaluation from alert handling. Recording rules pre-compute expensive PromQL expressions on a schedule and store the result as a new series, which speeds dashboards. Alerting rules evaluate a PromQL condition (e.g. error ratio > 1% for 10m) and, when it holds, emit an alert to Alertmanager. The "for" clause prevents flapping by requiring the condition to persist.
Prometheus is a sensor that raises its hand; Alertmanager is the dispatcher that groups the raised hands, ignores duplicates, and calls the right on-call team — while a warehouse (remote storage) keeps the long-term archive.
Key Concepts
1
Alertmanager is a separate component that receives alerts and decides what humans see: it groups related alerts, deduplicates identical ones from HA pairs, applies silences and inhibition (suppress a noisy alert when a bigger one fires), and routes by labels to receivers like PagerDuty or Slack. For history beyond a single node's disk, remote_write streams samples to scalable backends — Thanos, Cortex, Grafana Mimir, or vendor stores — giving long-term, horizontally scalable storage while Prometheus stays the collection engine.