Amazon Web Services services
15 / 24
Amazon Web Services

Streaming — Kinesis Data Streams & Firehose

Ingest high-volume ordered event streams and know when a stream beats a queue.

Kinesis handles continuous streams of records that several consumers read independently.

A conveyor belt past several workstations, rather than a counter where each item is handed to one person. Everything stays on the belt for a while, and each station takes what it needs at its own speed.

Key Concepts

1
    Data Streams   you manage shards; records kept 24h to 365 days;
                   many consumers read the same data at their own pace
    Firehose       fully managed delivery to S3, Redshift or OpenSearch;
                   no shards, buffers and writes, near-zero operations
2
The difference from SQS is the one to state.
    SQS       a message is consumed and DELETED. One consumer wins.
    Kinesis   records stay for the retention period. Every consumer
              keeps its own position and can re-read.
3
So you reach for Kinesis when several systems need the same events, or when you may need to replay history after fixing a bug.
4
A shard is the unit of capacity and of ordering.
    per shard:  1 MB/s or 1,000 records/s in,  2 MB/s out
    ordering is guaranteed WITHIN a shard, never across shards
5
The partition key chooses the shard, and this is where designs go wrong:
6
    partitionKey = customerId   -> all of one customer's events ordered
    partitionKey = "constant"   -> everything on one shard: a hot shard
                                   capped at 1 MB/s, however many you have
7
A hot shard is the Kinesis equivalent of a hot partition, and interviewers probe for it.
8
Scaling means resharding — splitting or merging shards — which is a real operation, unlike a queue that simply absorbs more. On-demand mode scales automatically for a higher price.
9
Enhanced fan-out gives each consumer its own 2 MB/s pipe instead of sharing the shard's read capacity, which matters once more than two or three consumers read the same stream.
10
Firehose is the one to pick when the destination is storage. It batches by size or time and writes to S3 in Parquet, with no shards to manage. The cost is latency — buffering is 60 seconds or more.
11
Kinesis or MSK? MSK is managed Apache Kafka: use it when you want Kafka's ecosystem or already have Kafka skills. Kinesis is simpler and integrates more tightly with AWS.
12
What the interviewer is probing.1. "How is Kinesis different from SQS?" Probing: consumption semantics. Stalls: "Kinesis is faster." Moves up: SQS deletes a message once consumed; Kinesis retains records so several consumers read independently and any can replay.
13
2. "Your throughput is capped despite many shards. Why?" Probing: the hot shard. Stalls: "Add more shards." Moves up: a constant or low-cardinality partition key sends everything to one shard; the key must spread the load.
14
3. "What ordering does Kinesis guarantee?" Probing: the scope of the guarantee. Stalls: "Everything is ordered." Moves up: ordering within a shard only, never across the stream.
15
4. "When is Firehose the better choice?" Probing: the two products. Stalls: "They are the same." Moves up: when the destination is simply S3, Redshift or OpenSearch — no shards to manage, at the cost of at least a minute of buffering.
Use cases:kinesisstreamingshardspartition-keyfirehose