Elastic (ELK)
Beats, Logstash & Ingestion
Collect, parse, and enrich data on its way into Elasticsearch.
Data reaches Elasticsearch through two complementary tools. Beats are lightweight, single-purpose shippers installed at the edge: Filebeat tails logs, Metricbeat collects metrics, Packetbeat sniffs network data, and so on. They are low-overhead and good at reliably forwarding raw data. (Elastic Agent now consolidates these behind one agent managed by Fleet, but the Beats model is still the mental foundation.)
Couriers (Beats) pick up raw parcels from every doorstep and bring them to a sorting plant (Logstash) that opens, labels, and repackages them before they go on the shelves (Elasticsearch).
Key Concepts
1
Logstash is the heavier processing pipeline, structured as input → filter → output. Its filters — grok for regex parsing, mutate, date, geoip, and many more — transform messy text into clean structured fields before indexing. A common architecture sends Beats → (optionally a queue like Kafka) → Logstash → Elasticsearch, separating cheap collection from expensive transformation. Alternatively, Elasticsearch ingest pipelines run lightweight processors inside the cluster for simpler parsing without a separate Logstash tier.