Prometheus
Architecture & Pull-based Scraping
Understand why Prometheus pulls metrics over HTTP and how its single-node TSDB is laid out.
Prometheus is the de-facto open-source metrics system, a CNCF graduated project built around a pull model. Instead of services pushing data, the Prometheus server periodically scrapes an HTTP /metrics endpoint exposed by each target, fetching a plain-text exposition of current values. Targets are found via service discovery (Kubernetes, Consul, file-based) so the scrape list updates as the fleet changes.
A night watchman doing rounds: instead of every room calling in, the watchman visits each on a fixed schedule and notes the reading — and immediately knows if a room fails to answer the knock.
Key Concepts
1
Scraped samples land in a local time-series database on disk, optimised for append-heavy writes and compressed into blocks. A single Prometheus server is intentionally standalone and reliable — no clustering, no external dependencies — which makes it simple to run but means one server's data and retention are bounded by its disk. The pull model also gives a free health signal: if a scrape fails, the target's "up" metric goes to 0.
2
What the interviewer is probing.
What the interviewer is probing.
3
1. "Why does Prometheus pull rather than receive pushes?" *Probing:* the design rationale.
*Stalls:* "It is simpler." *Moves up:* the server controls the rate, can tell a down target from a
silent one, and targets need no knowledge of the monitoring system.
1. "Why does Prometheus pull rather than receive pushes?"
4
2. "When does pull not work?" *Probing:* the honest limit. *Stalls:* "Never." *Moves up:*
short-lived batch jobs that finish between scrapes — which is what the Pushgateway exists for, used
sparingly.
2. "When does pull not work?"
5
3. "How are targets discovered?" *Probing:* the mechanism. *Stalls:* "A list." *Moves up:*
service discovery against Kubernetes, cloud APIs or a file, so scaling does not require editing
configuration.
3. "How are targets discovered?"
6
4. "What is the storage model?" *Probing:* the detail. *Stalls:* "A database." *Moves up:* a
local time-series database on the node, which is why long retention and high availability need
remote write.
4. "What is the storage model?"