Payment system (Stripe-like ledger)
Move money without ever charging a customer twice or losing a cent when networks time out. Every mutation carries an end-to-end Idempotency-Key, records immutable double-entry ledger postings (sum of debits = sum of credits) in an ACID transaction with an outbox, and reconciles daily against external card networks.
Component labels and connections
- Clients (client) · calls: Load balancer
- Load balancer (loadBalancer) · calls: Payment gateway 1, Payment gateway 2
- Payment gateway 1 (service) · calls: Payment orchestrator
- Payment gateway 2 (service) · calls: Payment orchestrator
- Payment orchestrator (service) · calls: Idempotency key store, Double-entry ledger DB, Transactional outbox / settlement queue
- Idempotency key store (dedupStore) · calls: none
- Double-entry ledger DB (database) · calls: none
- Transactional outbox / settlement queue (queue) · calls: none
- PSP capture + reconciliation workers (worker) · calls: Double-entry ledger DB
Requirements
Functional
- Authorize, capture, and refund payments via external Payment Service Providers (PSPs / card networks) and settle balances to merchants.
- Record every balance movement in an immutable double-entry ledger where every transaction's debits equal its credits.
- End-to-end idempotency: retrying a timed-out charge with the same Idempotency-Key returns the original result without double-charging.
- Daily automated reconciliation between internal ledger entries and external PSP settlement files.
Non-functional
- Zero double charges (`duplicateEffect`) and zero lost acknowledged transactions (`ackedWriteLost`).
- Strict monetary precision: integer minor units (cents) plus currency code — never IEEE-754 floats.
- Auditability: ledger entries are append-only; corrections are compensating reversal entries, never `UPDATE` or `DELETE`.
Worked estimates
Transaction volume
≈ 5,800 charges/s average, ≈ 20,000/s peak (Black Friday / flash sales)
500M payments/day ÷ 86,400
Ledger rows
≈ 400 GB/day ≈ 150 TB/year of immutable audit rows
500M payments/day × ~4 ledger postings (buyer, merchant, fee, settlement) × 200 B
Idempotency key store
≈ 64–128 GB — fits in memory + durable transactional table
500M keys/day × 128 B × 24–48 h active TTL
External PSP call latency
never hold a database row lock across an external PSP network call
card network rails take 200–1,000 ms p99
API
POST /v1/payments
Request: Header: Idempotency-Key (UUID) · Body: { amountCents, currency, source, merchantId }
Response: 200 { paymentId, status, ledgerTxId } (replayed on duplicate key)
POST /v1/payments/{id}/refunds
Request: Header: Idempotency-Key · Body: { amountCents, reason }
Response: 200 { refundId, compensatingEntries[] }
GET /v1/payments/{id}
Request: —
Response: 200 { paymentId, status, amountCents, entries[] }
Data model
- payments — store: ledgerdb
payments: state machine (`CREATED → AUTH_IN_PROGRESS → AUTHORIZED → CAPTURED → SETTLED` or `FAILED`).
ledger_entries: immutable `(entryId, txId, accountId, direction: DEBIT|CREDIT, amountCents, currency, createdAt)`. Account balance is the sum of entries (materialised into a versioned checkpoint for fast reads).
idempotency_keys: `(merchantId, idempotencyKey) → { requestHash, status, responsePayload }`, committed atomically with the payment state transition.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
End-to-end idempotency keys and state machines
An idempotency key flows across three hops: `Client → Payment API → External PSP`. On arrival, the orchestrator atomically inserts `(merchantId, idempotencyKey, status = IN_PROGRESS)` before touching the PSP. If a concurrent retry arrives while the first call is still in flight, it sees `IN_PROGRESS` and either waits or returns `409 Conflict (retry shortly)`. If the first call completed, the stored response is returned verbatim. If the client accidentally reuses the same key with different parameters (`amountCents`), compare the stored request hash and reject with `422`.
Double-entry bookkeeping and avoiding hot merchant accounts
Every financial event writes at least two immutable rows in one ACID transaction: e.g. a \$10.00 charge debits `buyer_clearing` by `1000` cents and credits `merchant_payable` by `970` cents and `platform_fee` by `30` cents. If every charge to a huge merchant (e.g. Amazon) locked a single merchant balance row to update a running total, that row would bottleneck at a few hundred TPS. Instead, append ledger entries without updating a single hot counter, or shard the merchant's sub-accounts across N buckets (`merchant_payable#0..N-1`) and sum them on read.
Sagas, transactional outbox, and T+1 reconciliation
Updating the ledger DB and publishing an event to a queue cannot be done with a naive dual write (if the process crashes in between, the ledger and downstream systems diverge). Commit the outbox message in the same local DB transaction as the ledger entries, and let a relay worker publish it at-least-once. Finally, every night a reconciliation job diffs the internal ledger against the PSP's settlement file (T+1), automatically flagging any orphan authorizations, missing captures, or fee discrepancies for finance review.
Failure scenarios
Acknowledgements from the payment service are lost
Scenario and expected result
From t = 1.5 s to 1.58 s every response from the payment orchestrator to gateway 1 is dropped after the ledger write has already committed. Gateway 1 times out after 50 ms and retries with the same `Idempotency-Key`.
Expected: The idempotency store intercepts every retry and replays the committed response without appending a second debit/credit pair. Zero duplicate charges (`clean: true`). Compare the no-dedup alternative.
Settlement / outbox workers crash
Scenario and expected result
The PSP capture & settlement workers crash from t = 1.3 s to 1.9 s.
Expected: Because payment intents and ledger entries are durably committed with an outbox queue, synchronous charge acceptance stays at > 99% goodput while settlement catches up once workers restart.
Ledger database leader fails over
Scenario and expected result
The primary ledger database node crashes at t = 1.5 s and promotes a semi-sync replica by t = 2.0 s.
Expected: Charges pause during the ~500 ms leader election and fail closed rather than risking split-brain or unrecorded writes. Semi-sync replication and fencing ensure every acknowledged payment survives on the new primary.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
No idempotency store: retry timeouts blindly
- When the payment orchestrator commits the charge and the response packet is dropped, the gateway times out and retries. Without atomic `Idempotency-Key` deduplication, the customer's card and ledger are debited twice (`duplicateEffect`). Run the `lost-acks` scenario to see the violation.
Reference
- Inspect the architecture above and run the same scenario.
Mutable `balance` column instead of an append-only double-entry ledger
- Doing `UPDATE accounts SET balance = balance - 1000` overwrites history. When a bug, partial retry, or settlement mismatch occurs, there is no audit trail proving where money came from or went, and no self-checking invariant (`SUM(debits) == SUM(credits)`).
Reference
- Inspect the architecture above and run the same scenario.
Distributed 2PC across internal DB and external PSP
- External card networks do not speak XA two-phase commit, and holding database locks across a 500 ms HTTP call to Visa/Mastercard exhausts connection pools. Instead, persist a `PENDING` payment + idempotency key locally, call the PSP outside the DB transaction forwarding the idempotency key, and record the outcome via a state-machine transition / saga.
Clarifying questions
- Are we building the merchant-facing payment gateway/ledger, or card-issuing rails?
- Do we support multi-currency transactions and foreign-exchange conversion?
- How are payouts to merchants scheduled (instant vs T+2 rolling settlement)?
- What happens when an external PSP times out and neither confirms nor denies a charge?
Common mistakes
- Storing money as `FLOAT` or `DOUBLE` instead of integer minor units (`amountCents`) + currency.
- Checking the idempotency key with a non-atomic `SELECT` before `INSERT` (check-then-act race).
- Holding a database transaction open while waiting on a 500 ms HTTP call to an external PSP.
- Mutating account balance columns in place without an immutable double-entry audit log.
- Having no reconciliation loop to detect drift between internal state and external bank/PSP settlement files.
Self-assessment rubric
requirements / levels 1-4
- 1/4 Charge a card.
- 2/4 Charge, refund, and store status.
- 3/4 Idempotency, double-entry ledger, integer cents, and daily reconciliation.
- 4/4 Defines exact failure semantics across client retries, DB crashes, and PSP timeouts.
estimates / levels 1-4
- 1/4 None.
- 2/4 TPS.
- 3/4 Average/peak TPS, ledger storage growth, and idempotency key footprint.
- 4/4 Identifies external PSP latency (~500 ms) vs internal DB lock duration (< 5 ms).
api / levels 1-4
- 1/4 POST /pay.
- 2/4 Includes amount and currency.
- 3/4 Mandatory `Idempotency-Key` header, integer minor units, and status/refund endpoints.
- 4/4 Specifies behaviour for concurrent in-flight duplicates and mismatched payload hashes.
data model / levels 1-4
- 1/4 Payments table.
- 2/4 Payments + balances.
- 3/4 Payment state machine + immutable double-entry postings + idempotency table.
- 4/4 Transactional outbox and sharded sub-accounts for high-throughput merchants.
high level / levels 1-4
- 1/4 API calls Stripe.
- 2/4 API + DB + PSP.
- 3/4 Gateway, orchestrator, idempotency store, ledger DB, outbox queue, settlement workers.
- 4/4 Separates synchronous DB state transitions from external PSP network calls.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Mentions idempotency.
- 3/4 Atomic idempotency locking, double-entry zero-sum invariant, and outbox pattern.
- 4/4 Saga compensation (refund/void), hot-account sharding, and T+1 settlement reconciliation.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions monitoring.
- 3/4 Duplicate charges, ledger imbalance, and PSP drift as top risks.
- 4/4 Continuous `SUM(debits) == SUM(credits)` invariant checks and reconciliation alerts.