Design a Payment System

hard

Scale: 100-10K TPS depending on scale; payment latency sub-second Storage: Append-only ledger grows forever; archival tier for old entries Stripe, Square, PayPal

Case StudyPaymentsConsistency

A payment system processes charges, refunds, transfers, and the bookkeeping that records them. Stripe, Adyen, Square, PayPal all share the architecture: idempotency on every API call, durable state machines, a strongly consistent ledger, and rigorous reconciliation against the bank/card-network statements.

Scale100-10K TPS depending on scale; payment latency sub-second

StorageAppend-only ledger grows forever; archival tier for old entries

Key Concepts

1. Idempotency is non-negotiable. Every POST takes an Idempotency-Key header. First call executes, response stored. Replay with same key returns stored response — doesn't re-execute. Networks between merchant ↔ gateway ↔ network are lossy; a 200 OK lost in transit looks identical to a charge that never happened. Idempotency keys retained 24h+ (outlast retry windows). Stripe's published design retains them this long.

1. Idempotency is non-negotiable.Idempotency-Key

2. Ledger as source of truth. Most production systems use double-entry bookkeeping: every transaction has 2+ entries that sum to zero (debit + credit). A $100 charge: -$100 to customer account, +$100 to merchant revenue. Balances computed via SUM(entries WHERE account = X) — never mutate stored balances. Append-only; historical state always recoverable. Postgres or Spanner-class. Schema is the hardest part to change later — design carefully.

2. Ledger as source of truth.-$100+$100SUM(entries WHERE account = X)

3. Multi-step flows: state machines + sagas. Card payment: authorized → captured → settled (or voided, refunded). Each transition persisted before any external call; recovery walks from persisted state. Saga for cross-service flows: sequence of local transactions, each with a compensating action if the next fails (charge → ship → email; ship fails → refund). 2PC rarely used — doesn't compose across organizations.

3. Multi-step flows: state machines + sagas.authorized → captured → settledvoidedrefunded

4. Reconciliation. Nightly job compares your ledger against bank/card-network settlement files (ACH NACHA, card network reports). Which charges they settled, which declined, which chargebacks. Discrepancies flagged for ops. Async settlement (T+1 to T+5) means your ledger and theirs are out of sync until reconciled.

4. Reconciliation.

5. PCI scope and risk. Card numbers (PAN) tokenized; vault holds raw PAN in isolated PCI scope. Everything else operates on tokens. Fraud / risk upstream of payment: rule-based + ML scoring; high-risk → decline or 3DS step-up. Webhooks to merchants are at-least-once with signature. Chargebacks can arrive weeks later — ledger handles non-linear timing naturally.

5. PCI scope and risk.

High-level design

API: idempotency-key → idempotency store check → execute or replay.
Charge flow: validate → fraud check → call PSP/network → record outcome → ledger entry.
Ledger: Postgres / Spanner; double-entry; append-only.
Multi-step: state machine in DB; saga for cross-service.
Settlement: nightly reconciliation against bank/network files.
Webhooks: at-least-once delivery to merchants with signature.

Components

API gateway + idempotency store (Redis or Postgres).
Fraud / risk service (rules + ML).
Tokenization vault (PCI-scoped, isolated).
PSP / network integration (Visa, Mastercard, ACH).
Ledger (Postgres / Spanner / CockroachDB) with double-entry.
State machine / saga orchestrator.
Reconciliation jobs (Spark over ledger + statements).
Webhook dispatcher (signed, retried).
Settlement engine.
Dispute / chargeback workflow.

Idempotency in depth

Key passed by client (UUID or content hash).
First call: lock on (idempotency_key); execute; store (request hash, response).
Subsequent calls: return stored response. Validate request hash matches (else: 422 with different body).
Retention: 24h+ (outlast retry windows).
Edge case: first call fails mid-write. Use a status field (pending → completed) so retries can resume safely.

Ledger design

Double-entry: every transaction has 2+ entries summing to zero.
Append-only: never UPDATE; corrections are new offsetting entries.
Balance: SUM(entries WHERE account = X). Cache for read performance.
Schema: (transaction_id, account, debit, credit, currency, timestamp, metadata).
Multi-currency: keep entries in transaction currency; FX rates as separate ledger.
Audit: immutable; complete history of every cent.

Saga pattern

Multi-step flow with compensations: each step's failure triggers undo for prior steps.
Orchestrator-based: central saga manager dispatches steps and compensations.
Choreography-based: services react to events; no central coordinator.
Example: charge → ship → email. If ship fails, refund (compensation for charge).
Idempotency on each step + compensation makes it safe under retries.

Trade-offs

Strong consistency in ledger non-negotiable: Postgres or Spanner-class.

Idempotency keys must outlive retry windows (24h+).

Sagas vs 2PC: sagas compose across services and orgs; 2PC requires shared transaction manager.

Networks are async — design for delayed authorization callbacks.

Never silently retry payment — always check status first.

Multi-region: usually region-pinned with cross-region reconciliation; truly global ACID via Spanner is heavyweight.