E-wallets — what matters more than code in payment systems

I started building my first e-wallet in 2010. Then came a payment system, a dealer portal, payment applications. Almost ten years later, by then in telecom, I found myself facing the same task again — designing a wallet and payment infrastructure for subscribers.

Between those two points everything changed: languages, frameworks, the approach to architecture, the way we deployed. One thing didn't — the list of things payment systems break on. It turned out to be remarkably stable, and almost none of it is about code directly.

Money is not a number in a table

The most common early mistake is to model a balance as a column on the user row and change it with UPDATE.

You shouldn't, and it isn't about performance. A number in a column can't answer the question "why does this person have exactly this much money?" And you will be asked that question — by the user, by accounting, by a regulator, at the least convenient moment.

The right model is a ledger of operations from which the balance is derived. Every movement of money is a separate immutable record: from where, to where, how much, when, on what grounds. The balance is a computed value, not the source of truth.

A few rules follow, and it's better to accept them up front:

• Never delete. A wrong operation isn't erased — it's compensated with a reversing entry. History must stay complete.
• Never edit retroactively. A correction is a new record, not a patch on an old one.
• Store everything that explains the decision. The rate, the fee, the external transaction id, the provider's response. Six months later you won't reconstruct it otherwise.

That ledger feels excessive right up until the first investigation. After it, it feels like the only sane option.

Idempotency is the main lesson

If you take one term away from this article, make it this one.

Networks are unreliable. The request went out, the response was lost. The mobile app timed out and retried. The user saw "error" and pressed the button again. An external system timed out and re-sent its callback.

In all of these cases, the money must not be charged twice.

The solution is conceptually simple: every operation has a unique key generated by the initiator. A repeat request with the same key doesn't create a new operation — it returns the result of the existing one. Not "error, duplicate", but precisely the same response as the first time.

It sounds trivial, but in practice the requirement spreads across the whole system: the API, callback handlers, background jobs, retries. Idempotency can't be "added later" — you either build it into the model from the start or rewrite half the system.

A payment's state is a state machine

A payment isn't simply "succeeded" or "failed". Between them lives the most interesting state of all: unknown.

We sent a request to the bank and got no answer. The money may have been charged. It may not have been. Until we find out for certain, the operation is neither completed nor cancelled — it hangs.

So a payment needs an explicit set of states and explicit transition rules: created → processing → succeeded / declined / needs investigation. And it needs a mechanism that deals with the hanging ones: re-querying the provider for status, timeouts, automatic cancellation, or escalation to a human.

A system with no "I don't know" state will sooner or later either lose money or create it out of thin air.

Reconciliation isn't optional

The moment a second system appears — a bank, a processor, an operator, a partner — two versions of the truth appear with it. Your database says one thing, their export says another. Not because anyone is cheating, but because some operations landed during an outage, a timeout or a nightly maintenance window.

Discrepancies will always exist. The only question is whether you learn about them within a day from an automated reconciliation, or a month later from accounting.

That makes regular reconciliation as mandatory a part of a payment system as processing payments itself. And it needs a real tool: not "export two CSVs and diff them in Excel", but a proper discrepancy report where each item can be worked through.

Observability matters more than it seems

In the argument "I was charged and didn't get the service", the winner isn't the one who's right — it's the one who has the log.

A payment system has to store the actual exchange with external systems: what exactly was sent, what exactly came back, when. It saves you both in incident analysis and in negotiations with a partner whose documentation disagrees with their API's behavior — and it almost always does.

It's also worth alerting not only on errors but on silence: if the payment flow suddenly drops to zero, that's bad news even when there isn't a single exception in the logs.

The hardest part isn't technology, it's trust

The technical side of a payment system is hard but finite. The other side is harder.

When someone's money disappears — even for ten minutes, even because of a provider delay — they aren't thinking about your architecture. They're thinking they've been cheated. And rebuilding that trust costs far more than preventing the failure.

That leads to things engineers don't always find obvious:

• Clear statuses instead of technical errors. "Payment is processing; the money will return within an hour if it doesn't go through" beats "Error 500".
• Support must see what the system sees. If an agent can't answer "where is my money", the system is badly designed, however elegant it is inside.
• Predictability beats speed. A payment that always takes ten seconds is better than one that's usually instant but occasionally hangs for a day.

What I took from two runs at the same problem

The gap in technology between the 2010 wallet and the 2019 one is enormous. But if you asked me what actually determined the outcome in both cases, I wouldn't name the stack.

What determined it was the data model (a ledger versus a number in a column), the attitude to failure (designed for, or hoped away), and the ability to explain every last cent six months after the fact.

That's probably the core difference between fintech and ordinary development. In most systems a bug means inconvenience. Here it means someone else's money — and that changes the priorities entirely.