Design a Payment System

hard

Scale: 100-10K TPS depending on scale; payment latency sub-second Storage: Append-only ledger grows forever; archival tier for old entries Stripe, Square, PayPal

Case StudyPaymentsConsistency

A payment system is judged on what it does when something goes wrong. Any design can move money when everything works.

Scale100-10K TPS depending on scale; payment latency sub-second

StorageAppend-only ledger grows forever; archival tier for old entries

Key Concepts

The problem that defines the whole thing. You send a charge to the bank. The network drops before the reply comes back. You now cannot tell whether the customer was charged.

The problem that defines the whole thing.

Try again and you might charge them twice. Do nothing and you might have lost a real payment. A timeout is not the same as a failure, and your system has no way to tell them apart.

The fix is an idempotency key. The client makes up a unique key for this payment attempt and sends it with the request. It sends the same key when it retries.

The fix is an idempotency key.

The server stores the key the moment it starts work. If a request arrives with a key it has already seen, it does not charge again — it returns the result of the first attempt. Two requests, one charge, and retrying is now safe.

Make the key required, not optional. Client libraries retry on their own, and if the key is optional somebody will leave it out.

Store money as whole numbers. Use paise, not rupees. Floating point cannot hold 0.1 exactly, and the tiny errors add up until your totals stop matching the bank's. Every amount carries its currency, and you refuse to compare amounts in different currencies.

Store money as whole numbers.

Never store a balance you overwrite. Record every movement as two entries that cancel out: one account is debited, another credited, and the pair sums to zero. A balance is then the sum of the entries.

Never store a balance you overwrite.

Nothing is ever edited. A mistake is corrected by adding a correcting pair, so the error and the fix are both visible afterwards. This is what makes a payment system auditable, and it is the difference between a payments answer and a database answer.

Authorising and capturing are two steps. Authorising checks the card and reserves the money. Capturing actually moves it. Shops authorise when you order and capture when they ship, which is why a cancelled order leaves a pending charge that disappears a few days later.

Authorising and capturing are two steps.

Model them as one step and refunds, partial shipments and expiry all become impossible to represent honestly.

Webhooks lie about order. The payment processor tells you asynchronously that a payment settled. Those messages arrive twice, out of order, or not at all. Check the signature, handle each event ID once, and never assume the order. payment.captured arriving before payment.authorized is normal.

Webhooks lie about order.payment.capturedpayment.authorized

Reconciliation is the safety net. Every day, compare your records with the file the processor sends, line by line. Anything on one side and not the other is a break, and a human works through it. A payment system without this finds out about its bugs from customers.

Reconciliation is the safety net.

What the interviewer is probing.

What the interviewer is probing.

1. "Design a payment system." Probing: whether you start from failure. Stalls: request, charge, update balance, respond. Moves up: naming the timeout problem in the first minute.

1. "Design a payment system."

2. "The response times out and the client retries." Probing: idempotency, precisely. Stalls: "Check if a payment already exists." That races with itself — two requests can both find nothing. *Moves up:* a unique key stored before the charge, and the stored result returned on a retry.

2. "The response times out and the client retries."

3. "How do you store amounts?" Probing: the floating point trap. Stalls: "A decimal type." *Moves up:* whole numbers of the smallest unit, with the currency alongside.

3. "How do you store amounts?"

4. "A refund is issued. How does the balance change?" Probing: ledger versus a balance column. Stalls: "Subtract from the balance." Moves up: add a correcting pair of entries; the balance is derived and the history survives.

4. "A refund is issued. How does the balance change?"

5. "The processor sends the same webhook twice, out of order." Probing: whether you trust the network. Stalls: "Process them in order." Moves up: verify the signature, handle each event ID once, assume nothing about order, reconcile daily.

5. "The processor sends the same webhook twice, out of order."

High-level design

API: idempotency-key → idempotency store check → execute or replay.
Charge flow: validate → fraud check → call PSP/network → record outcome → ledger entry.
Ledger: Postgres / Spanner; double-entry; append-only.
Multi-step: state machine in DB; saga for cross-service.
Settlement: nightly reconciliation against bank/network files.
Webhooks: at-least-once delivery to merchants with signature.

Components

API gateway + idempotency store (Redis or Postgres).
Fraud / risk service (rules + ML).
Tokenization vault (PCI-scoped, isolated).
PSP / network integration (Visa, Mastercard, ACH).
Ledger (Postgres / Spanner / CockroachDB) with double-entry.
State machine / saga orchestrator.
Reconciliation jobs (Spark over ledger + statements).
Webhook dispatcher (signed, retried).
Settlement engine.
Dispute / chargeback workflow.

The double-charge race, concretely

  Timeline of the classic incident
    t0  client POSTs /payments
    t1  server calls the processor; card is charged
    t2  network drops before the response returns
    t3  client times out, retries
    t4  without an idempotency key: second charge, same customer

  With an idempotency key
    t0  client generates key K (UUID), POSTs /payments with K
    t1  server INSERTs (K, status=in_progress) -- unique index on K
    t2  server calls processor, stores the result against K
    t3  response lost
    t4  client retries with the SAME K
    t5  server's INSERT fails on the unique index -> reads the stored
        result -> returns the original outcome. One charge.

  What to store against the key
    key, request fingerprint (hash of amount+currency+payee),
    status, processor reference, response body, created_at.

  Why the fingerprint matters
    If a retry arrives with the same key but a different amount, that is a
    client bug or an attack. Reject with 422 rather than silently
    returning the old result.

  In-progress retries
    A retry that arrives while the first is still running should block
    briefly and then return 409, not start a second processor call.

Ledger model

Double-entry, append-only. Every payment writes balanced entries.

  Customer pays 500.00 INR for an order:
    entry 1   debit   customer_receivable      50000   (paise)
    entry 2   credit  merchant_payable         50000
                                              -------
    sum                                            0

  Refund of 200.00 INR later:
    entry 3   debit   merchant_payable         20000
    entry 4   credit  customer_receivable      20000

  Balance of any account = SUM(credits) - SUM(debits) over its entries.
  No row is ever updated. A mistake is corrected by a compensating pair,
  which leaves the error and its correction both visible.

  Fields every entry needs
    entry_id, transaction_id, account, direction, amount_minor, currency,
    created_at, and the idempotency key or processor reference that caused
    it.

  Why not a balance column
    Two concurrent updates to a balance need a lock held across the
    transaction, and a crash mid-update leaves a number nobody can audit.
    Summing entries is slower and always explicable; snapshot periodically
    if the sum gets expensive, keeping the entries as the source of truth.

Failure modes in production

Retry without idempotency key
Symptom: duplicate charges, usually on mobile networks.
Cause: client library retries transparently on timeout.
Fix: key generated once per attempt and reused across retries -- and
make it required by the API, not optional.

  Webhook replay
    Symptom: a settled payment is credited twice.
    Cause: handler not idempotent on event id.
    Fix: unique index on the processor's event id; second delivery is a
    no-op that still returns 200 so the processor stops retrying.

  Partial refund arithmetic
    Symptom: refunds exceed the original charge by a few paise.
    Cause: percentage splits rounded independently per line.
    Fix: allocate rounding remainder to one line deterministically and
    assert the split sums to the original amount.

  Reconciliation break nobody owns
    Symptom: a small daily mismatch persists for weeks.
    Cause: breaks are logged but not assigned.
    Fix: treat a break like an incident -- owner, deadline, and a written
    cause. The value is in the causes you find, not in the report.