Apache Kafka
Delivery Semantics
Choose between at-most-once, at-least-once, and exactly-once — and know what each really costs.
Every messaging system delivers under one of three guarantees. At-most-once means messages may be lost but never duplicated (commit offset before processing). At-least-once means messages are never lost but may be redelivered (process, then commit) — the most common default. Exactly-once means each message affects the result once, with no loss and no duplicate.
At-least-once is mailing a letter with delivery confirmation — it always arrives, but if confirmation is lost you might send it again. Exactly-once is a bank transfer with a transaction id the bank uses to ignore any duplicate submission.
Key Concepts
1
Kafka achieves effective exactly-once within Kafka using the idempotent producer (dedupes retries via a producer id and sequence number) plus transactions, which atomically write to multiple partitions and commit consumer offsets in the same transaction (the read-process-write loop). The consumer must set isolation.level=read_committed to skip aborted records.
2
For systems writing to an external database, true end-to-end exactly-once usually still relies on idempotent writes or the outbox pattern, because Kafka's transactions don't extend into your database.
3
What the interviewer is probing.
What the interviewer is probing.
4
1. "Explain the three semantics." *Probing:* the fundamentals. *Stalls:* "They deliver
messages." *Moves up:* at most once risks loss, at least once risks duplicates, exactly once needs
idempotent producers and transactional reads and writes.
1. "Explain the three semantics."
5
2. "Which do you design for in practice?" *Probing:* the judgement. *Stalls:* "Exactly once."
*Moves up:* at least once with idempotent consumers — it is simpler, cheaper and covers the same
correctness need.
2. "Which do you design for in practice?"
6
3. "What is the scope of Kafka exactly-once?" *Probing:* the honest limit. *Stalls:*
"Everything." *Moves up:* reads and writes within Kafka — a side effect such as sending an email or
calling an external API is outside the transaction.
3. "What is the scope of Kafka exactly-once?"
7
4. "How do you make a consumer idempotent?" *Probing:* the implementation. *Stalls:* "Check for
duplicates." *Moves up:* a natural idempotency key written atomically with the work, so reprocessing
the same record changes nothing.
4. "How do you make a consumer idempotent?"