What payment reconciliation systems teach you about distributed consistency
Eventual consistency in payment reconciliation is a business requirement, not a technical compromise. Idempotency keys and reconciliation windows replaced two-phase commits.
In computer science courses, distributed consistency is presented as a technical tradeoff. CAP theorem. Linearizability versus eventual consistency. Two-phase commit versus saga patterns. In payment reconciliation, the same concepts show up, but the framing is different. Consistency is a business constraint with dollar values attached to every failure mode.
At FinanceOps, our reconciliation system matches records across banks, payment processors, and our internal ledger. These systems have different clocks, different schemas, different event delivery guarantees, and different definitions of what constitutes a completed transaction. Making them agree is the core problem.
By this point I cared less about sounding smart and more about making the tradeoff legible. It also builds on what I learned earlier in “Running Kafka at startup scale is a decision you will regret exactly once.” The systems had enough history that every database or eventing opinion had receipts behind it. That is the same posture I now bring to longer-lived experiments like ftryos and pipeline-sdk: if the constraint is real, say it plainly and design around it.
Why Two-Phase Commit Failed Us
Our first attempt at reconciliation tried to enforce strict consistency. When a payment event arrived, we would confirm the transaction state across all parties before recording it as reconciled. This is the two-phase commit approach: prepare, verify, commit.
It failed for three reasons:
- Banks do not respond in real time. Some bank APIs have response times measured in minutes during batch processing windows. A two-phase commit that waits for a bank to confirm a transaction state can block for indefinitely long periods.
- Payment processors batch their settlement reports. Stripe settles daily. Some bank processors settle every 48 hours. You cannot achieve strict consistency across systems that operate on fundamentally different time scales.
- Network partitions are not theoretical in fintech. Bank API outages happen weekly. Payment processor maintenance windows happen monthly. Any consistency model that requires all parties to be available simultaneously will have unacceptable downtime.
The key insight was that eventual consistency is not a compromise in payment reconciliation. It is the business requirement. The business does not need instant reconciliation. It needs accurate reconciliation within a defined window. That window is typically 24-48 hours for most transaction types.
Reconciliation Windows and Idempotency Keys
We replaced two-phase commit with a reconciliation window model. Every transaction gets a window: a time period during which we expect all parties to report their view of the transaction. The reconciliation engine runs continuously, matching records as they arrive from different sources.
- When a payment event arrives from any source, it gets recorded with its idempotency key and source timestamp.
- The matching engine runs every 60 seconds, looking for transaction records from all expected sources within the reconciliation window.
- When all sources agree on the transaction state, the transaction is marked as reconciled.
- When sources disagree, the transaction is flagged for investigation with the specific discrepancy recorded.
- When the reconciliation window closes and a source has not reported, the transaction is flagged as missing, not as failed.
Idempotency keys are the linchpin. Every transaction has a unique key that is consistent across all sources. When Stripe sends a webhook for the same payment twice, the idempotency key ensures we do not double-count it. When a bank reports a transaction that our processor has not confirmed yet, the idempotency key lets us match them when the confirmation arrives hours later.
What This Teaches About Distributed Systems
Payment reconciliation is a masterclass in practical distributed consistency because the constraints are unavoidable and the consequences of getting it wrong are measured in dollars, not error logs.
- Design for temporal disagreement. Systems that participate in reconciliation will have different views of the truth at any given moment. Your architecture must tolerate this as normal, not exceptional.
- Define your consistency window explicitly. “Eventually consistent” without a defined window is meaningless. Our SLA is 24-hour reconciliation for domestic payments, 48 hours for international. These numbers drive the architecture.
- Make idempotency a first-class concern. Every write operation in the reconciliation pipeline is idempotent. Every event can be safely replayed. This is not optional. It is the foundation that makes eventual consistency workable.
- Separate detection from correction. The reconciliation engine detects discrepancies. A separate process handles correction. Mixing these concerns creates systems that are both fragile and opaque.
The Broader Lesson
This is the phase where individual scars finally turned into repeatable operating principles. I cared less about sounding clever and more about leaving behind a system that stayed sane without me in the room. That is how I build ftryos and pipeline-sdk too.
Distributed consistency is not a technical problem you solve once. It is a business constraint you manage continuously. The architecture reflects the business reality, not the other way around.
Every distributed system I have worked on since building the reconciliation engine looks different to me now. The question is never “how do I achieve strong consistency?” The question is “what is the consistency window the business actually needs, and what happens when records disagree within that window?” If you can answer those two questions precisely, the technical architecture follows naturally. If you cannot, no amount of clever engineering will save you.
Payment reconciliation taught me that the hardest distributed systems problems are not technical. They are organizational. Getting three external parties to agree on an idempotency key format is harder than implementing the matching algorithm. Getting the business to define an acceptable reconciliation window is harder than building the engine that enforces it. The engineering is the easy part. The alignment is the work.
Payment reconciliation is where distributed systems theory meets financial regulation. Every inconsistency has a dollar amount attached to it, and every dollar amount has a compliance reporting requirement. The reconciliation engine we built handled ten million transactions before it needed its first major refactor, and the refactor was driven by performance requirements, not correctness issues. That durability came from treating reconciliation as a first-class domain problem, not a background batch job.