Explain Deduplication & Ordering (FIFO) in Messaging

6 / 8

Amazon SQS

Deduplication & Ordering (FIFO)

Get exactly-once processing and strict order out of SQS FIFO using group and dedup ids.

FIFO queues deliver two guarantees through two ids. Ordering is governed by MessageGroupId: messages sharing a group id are delivered in strict order, one in-flight at a time per group, while different group ids are processed in parallel. Choosing the group id (e.g. per orderId or per customer) is exactly like choosing a Kafka partition key — it sets the granularity of ordering versus parallelism.

Group id is assigning each customer their own single-file queue at a bank — within a customer everything is in order, but many customers are served in parallel. Dedup id is the bank ignoring a duplicate deposit slip with the same reference number submitted twice.

Key Concepts

Exactly-once is governed by MessageDeduplicationId: within a 5-minute deduplication window, SQS treats any message with a repeated dedup id as a duplicate and silently drops it. You either supply the id explicitly (e.g. a business key) or enable content-based deduplication, which hashes the message body to derive it. This makes producer retries safe — resending the same logical message does not create a duplicate.

The cost is throughput: per-group serialization and the dedup machinery cap FIFO well below Standard, mitigated by batching and high-throughput mode.

What the interviewer is probing.

What the interviewer is probing.

1. "How does FIFO deduplication work?" *Probing:* the mechanism. *Stalls:* "It checks content." *Moves up:* a deduplication id — explicit or a content hash — within a five-minute window.

1. "How does FIFO deduplication work?"

2. "What is the limitation of that window?" *Probing:* the honest limit. *Stalls:* "None." *Moves up:* a duplicate arriving after five minutes is accepted, so application-level idempotency is still needed for retries over longer gaps.

2. "What is the limitation of that window?"

3. "When would you supply an explicit deduplication id?" *Probing:* the judgement. *Stalls:* "Never." *Moves up:* when the body can differ for the same logical event — a timestamp or trace id in the payload breaks content-based hashing.

3. "When would you supply an explicit deduplication id?"

4. "How do ordering and parallelism coexist?" *Probing:* the design. *Stalls:* "They do not." *Moves up:* through message group ids — order within a group, parallelism across groups.

4. "How do ordering and parallelism coexist?"