Explain Deduplication & Ordering (FIFO) in Messaging

6 / 8

Amazon SQS

Deduplication & Ordering (FIFO)

Get exactly-once processing and strict order out of SQS FIFO using group and dedup ids.

FIFO queues deliver two guarantees through two ids. Ordering is governed by MessageGroupId: messages sharing a group id are delivered in strict order, one in-flight at a time per group, while different group ids are processed in parallel. Choosing the group id (e.g. per orderId or per customer) is exactly like choosing a Kafka partition key — it sets the granularity of ordering versus parallelism.

Group id is assigning each customer their own single-file queue at a bank — within a customer everything is in order, but many customers are served in parallel. Dedup id is the bank ignoring a duplicate deposit slip with the same reference number submitted twice.

Key Concepts

Exactly-once is governed by MessageDeduplicationId: within a 5-minute deduplication window, SQS treats any message with a repeated dedup id as a duplicate and silently drops it. You either supply the id explicitly (e.g. a business key) or enable content-based deduplication, which hashes the message body to derive it. This makes producer retries safe — resending the same logical message does not create a duplicate.

The cost is throughput: per-group serialization and the dedup machinery cap FIFO well below Standard, mitigated by batching and high-throughput mode.