Design Ad Click Aggregation / Real-time Analytics

hard

Scale: 1M+ events/s; sub-second freshness on dashboards Storage: Raw events to lake; aggregates compact in OLAP Google, Meta, TikTok

Case StudyStreamingAnalytics

Ad click aggregation ingests billions of click events per day, computes per-ad per-minute counts in near real time, emits to dashboards and billing. The same shape serves any streaming analytics — pageviews, IoT telemetry, fraud signals.

Scale1M+ events/s; sub-second freshness on dashboards

StorageRaw events to lake; aggregates compact in OLAP

Key Concepts

1. Ingest. Edge collectors (JS on publisher page, mobile SDK) batch and forward to Kafka (or Kinesis, Pub/Sub), partitioned by ad_id — all events for one ad on one partition, preserving order and enabling stateful processing. Collectors stateless; Kafka backpressure protects downstream.

1. Ingest.ad_id

2. Stream processing. Flink (or Spark Streaming, Kafka Streams) subscribes, groups by ad_id, applies tumbling windows (1 minute typical), emits aggregates. Watermarks handle late events — bounded lateness. Allowed lateness folds late events into a delayed update. State (per ad_id) stored in RocksDB inside Flink, with checkpointing for durability.

2. Stream processing.

3. Exactly-once vs at-least-once. Exactly-once in Flink: idempotent producers + transactional sinks. Expensive. Most pipelines use at-least-once + dedup at the sink (idempotent INSERT or upsert with click_id). Dedup state has a TTL so it doesn't grow forever. For billing, exactly-once matters; for analytics, near-exact is fine.

3. Exactly-once vs at-least-once.

4. Output sinks. OLAP store (Druid, ClickHouse, Pinot) for sub-second dashboard queries. Data lake (S3 + Parquet) for raw events — replay, ad-hoc analytics, fraud forensics. Billing pipeline (separate, reconciled) for invoicing. Fraud detector reads raw stream in parallel; cleaned events feed billing.

4. Output sinks.

5. Hot keys and architecture. One viral ad floods one partition. Mitigate: producer pre-aggregation (SDK buffers counts before send), key salting ((ad_id, salt) then merge), special routing for top ads. Lambda vs Kappa: Lambda (batch + stream + serving merge) is battle-tested but complex; Kappa (stream-only with replay) is simpler and increasingly the default.

5. Hot keys and architecture.(ad_id, salt)

High-level design

Edge collector → Kafka (partitioned by ad_id, long retention).
Stream processor (Flink) → windowed aggregation → dedup → emit.
Sinks: OLAP (Druid/ClickHouse) for dashboards; data lake (S3) for replay; billing pipeline for invoicing.
Side: fraud detector reads raw stream; flags / removes bad clicks.
Dashboards / API → OLAP query.

Components

Edge collector (stateless, batches to Kafka).
Kafka (partitioned by ad_id; week+ retention for replay).
Stream processor (Flink): keyed state per ad_id, tumbling windows.
Dedup state (RocksDB-backed in Flink).
OLAP store (Druid, ClickHouse, Pinot) for sub-second queries.
Data lake (S3 + Parquet).
Billing pipeline (separate; exactly-once or reconciled).
Fraud detection service.
Dashboards (Superset, Grafana, internal).

Stream processing details

Windowing: tumbling 1-minute (non-overlapping) for billing; sliding for trends.

Watermarks: declare 'I've seen all events up to time T'; bounded lateness.

Allowed lateness: late events trigger updates within window.

State backend: RocksDB-based for large state; in-memory for small.

Checkpointing: durable snapshots for recovery.

Exactly-once: Flink + Kafka transactions; idempotent sinks.

Architecture choice: Lambda vs Kappa

Lambda: batch (nightly Spark over raw events) + stream (Flink for fresh approximate); serving merges. Complex, two code paths, but battle-tested for correctness.

Kappa: stream-only; reprocess by replaying log when needed. Simpler. Depends on log retention.

Most modern designs are Kappa with Flink + Kafka long retention.

Some still use Lambda for billing because nightly Spark on raw events gives ironclad correctness.

Hot keys

One ad goes viral; 80% of traffic to one partition.

Mitigations:
- Producer pre-aggregation: SDK buffers and sends counts, not individual events.
- Salt keys at producer: split into (ad_id, salt) partitions; consumer merges.
- Special-case: route top ads to dedicated processors.
- Repartition mid-stream: hash by (ad_id, time bucket) instead of just ad_id.

Trade-offs

Exactly-once: achievable with Flink + transactional sinks, but slower and more complex.

At-least-once + dedup: simpler, requires idempotent sinks; standard in practice.

Druid vs ClickHouse: Druid for time-series rollups; ClickHouse for general OLAP.

Lambda: correctness via batch backup; double maintenance.

Kappa: simpler; depends on log retention and reprocessing performance.

Billing accuracy > analytics accuracy. Separate paths.