The half of the system nobody sees

Operations staff at a bank keep the posting application open in a tab behind whatever they are actually doing. A posting that fails needs attention inside a window the bank guarantees, and nobody was looking at the page. So the page title itself flips to an error marker, and the warning rides in the browser tab where somebody working on something else entirely will still catch it. That is the sort of problem this page is about. There is a longer version of it further down, along with three others.

I have been a full-time developer at The Bancorp since August 2016, ten years as of this August. I write the code behind the screen: the part that decides what is true and refuses to guess when it does not know. The source is not mine to show you, and a job title on its own is not evidence of anything, so this page is what the years actually consisted of.

A bank runs on a great many small systems nobody outside it has ever heard of. Something has to open an account. Something else has to move money between accounts overnight, decide whether an account is allowed to go below zero, and settle up in the morning with the partner on the other side. Something has to take the file a card processor sends back at the end of the day and turn it into the ledger entries an accountant will put their name to. Something has to notice that a letter came back undelivered and start the work that fixes the address before the next one goes out. None of it is glamorous. All of it has to be right, because when it is wrong, it is wrong about somebody's money.

I have worked on thirty-two of those systems. Nine of them I started myself, from an empty folder and a first day. Four of the nine are, by the record of who wrote which line, near enough mine alone. The rest run from systems I built the majority of alongside one or two other people, down to a lending platform with a large team on it where my part was one specific corner of the thing.

The written record I am drawing on starts in May 2018, so the first twenty-one months of those ten years are not in it. I have not counted them anywhere on this page, which means every figure here is an undercount rather than a flattering one.

That is the scale. What follows is the shape of it, then the decisions I keep making across all of them, then four problems that were worth the telling.

The systems I own outright

Owning something outright means two separate things and they are worth keeping apart. One is that I started it: an empty folder, my name on the first day of it, every decision after that mine to make or to defend. The other is that I wrote most of what is in it. They usually travel together, they are not the same claim, and rounding one into the other is the easiest lie to tell about work nobody can go and look at. So they are stated separately here.

The ones I started

Nine systems began with me. Four of those are also, by the record of who changed what, near enough entirely mine.

The general-ledger upload system for deposit accounting is the largest. It builds, checks and posts the ledger entries that deposit accounting runs on, from templates through to postings that have to land the same day, and three much older programs feed it from outside. Ninety-nine percent of the recorded changes to it are mine. The design constraint that shaped the whole thing is that the person who enters a figure must not be the person who approves it, which is a rule about people rather than about software, and the software has to make it true anyway.

The internal real-time messaging bus is not an application at all. One system announces that something happened and another system, written by somebody else, in a different decade of the same estate, hears about it without either of them knowing the other exists. It is the piece I am most careful about, because the failure mode of a message bus is that everything downstream of it looks fine and is quietly wrong.

The admin tool over internal program settings and their audit history is the boring one that everything else depends on. It reads and edits the configuration values other systems run on, across three separate databases, and keeps a full history of who changed which value and when. The audit trail is written by the database itself rather than by the application, so a change made behind the tool's back is still recorded.

The bulk source-repository mirroring utility is the smallest thing on this page and it is here on purpose. It is a throwaway internal tool, and it was built with the same structure, the same configuration handling and the same test suite as the applications, because the alternative is a script nobody can safely change eighteen months later.

The other five I started are carried in part by other people, and I say so plainly. The account services desk for bank operations staff is the one I wrote the majority of: customer and account lookup, address changes, returned mail, email updates, and the nightly job that clears the queue behind all of it. The internal IT ticketing and asset register is the longest-running thing I have ever worked on, live from December 2018 to now, and roughly half of it is mine. The instant-payment rail and its settlement process, and the wire-handling and transfer-communication jobs, I started and then largely handed on. The last is a repository whose only content is a note saying there is no source in it yet, which is a small thing to admit in writing and better than an abandoned half-application with a promising name.

The ones I did not start

The largest single body of work I have is the enterprise transactional email platform, and I did not found it. Every internal system that needs to send a customer an email hands it to that platform, and it is the piece with the most callers and therefore the least freedom to change. Eight of every ten recorded changes to it are mine, across nearly eight years of it, and the interesting engineering in it is not a feature. It is an entire test suite whose only job is to prove that a rewrite still speaks to its old callers exactly as it did before.

Below that sits the ordinary weight of a bank. The overnight sweep, overdraft and settlement processor moves money between internal accounts and an outside partner every night and has to do the steps in a fixed order, because money movement is not a set of events that can arrive in any sequence. The ACH and government payment processing suite parses the acknowledgement files a payment network sends back, in formats where a single character in the wrong position is a very expensive file. The fraud alert triage platform is where analysts work through alerts and decide who gets restricted. The insurance-backed line of credit tracker follows loans and the policies pledged against them. The securities-based lending platform for advisors is a large multi-team codebase where my part was one specific corner, and I am not going to describe it as more than that.

For seventeen of the thirty-two the record does not say who started them, and there are two different reasons for that. Six had their history re-imported during a platform migration, which destroyed the evidence outright. For the other eleven the record simply never says. Either way this page claims nothing about who started them. One of the six, the project template most of the organisation's web applications were copied from, is ninety-four percent mine by recorded change, and I still will not claim I founded it, because the evidence for that was destroyed by a migration and not by me being modest.

The decisions that keep recurring

Thirty-two systems over ten years, and the same seven arguments in nearly every one of them. These are not preferences I would abandon under pressure and they are not things I read in a book. Each one is here with the reason I hold it and with what it costs me, because a decision presented without its cost is advertising rather than engineering, and every one of these is arguable.

One endpoint, one file

Every HTTP operation is its own class, carrying its own request and response types, instead of a controller holding a dozen loosely related actions.

Controllers bloat. Finding one operation means scanning hundreds of lines or hunting for a method by name, and that is before the pile of decorators a conventional controller carries. The framework I use does not merely permit one endpoint per file, it expects it, and it keeps that file small enough to read in one go. The hard work belongs in the service behind it, not in the thing that answers the request. Finding what I am looking for got dramatically faster the day I switched.

The cost is files. A five-operation feature is five endpoint classes plus its contracts, where a controller would have been one file, and somebody who likes seeing a whole feature on one screen will hate it. I would still rather scroll a folder than scroll a file. It is the house pattern on the general-ledger upload system, the admin tool over internal program settings, and the account services desk for bank operations staff.

Never throw out of a service

Every service returns a result envelope. Failure is a value the caller has to handle, and only the outermost layer turns that value into an error response. The rule is written down inside the repositories that follow it, in those words.

I do not like what exception handling tells you, which is almost nothing. Did this really fail, or was the exception planned? Did it come from the thing I just called, or bubble up from four layers below it? Do you want to attach a message explaining what the caller should do about it? Too bad. A result envelope behaves the way I want code to behave, and it leaves room to say more than that something went wrong.

The cost is that every call site has to unwrap, every time, including the ninety percent of call sites where nothing interesting can happen. Where a failure genuinely is exceptional, an envelope is more ceremony than a throw would have been, and I pay that ceremony everywhere to get the clarity where it matters. The account services desk and the enterprise transactional email platform both run on it.

The browser never touches the API

A server-side hop sits between the browser and the API. The browser talks to that hop, the hop proves who is calling and forwards. On one system the browser cannot reach the API at all, by design.

This one is not a preference. It came out of the security environment I work in: individual users are not to reach an API directly, and that needed a hard line rather than a policy. Only service accounts may call one. The hop in front is the thing that makes it true instead of merely stated.

The cost is real and I will not pretend otherwise. It is another deployable to build, host and monitor, another hop of latency on every request, and every endpoint effectively exists twice: once on the API, once as the pass-through in front of it. On a system with no compliance pressure behind it that is a lot of machinery for a rule nobody is enforcing. It is how the admin tool over internal program settings, the general-ledger upload system and the account services desk are all built.

Permissions belong to actions, not to records

The standard set is read, edit and admin, and it is not enough. Approving, back-dating, auto-reversing, voiding, changing a settlement date and editing the one field that grants somebody else access are each their own permission, separate from ordinary edit.

The extra ones arrived by two routes and I am not going to tidy that into a single story. Some were designed in while planning the application, because it was obvious up front that approving a figure is a different act from typing one. Others were added later, once it became clear that a role already handed out could do something nobody intended it to do. Both happened, repeatedly.

The cost is administration. More permissions means more roles, more combinations, and somebody has to hold a map of which role can do what. A three-permission model is genuinely easier to run, right up until the day it is wrong about something expensive. The general-ledger upload system, the overnight sweep, overdraft and settlement processor, and the internal IT ticketing and asset register all carry decomposed permissions.

Legacy gets marked, not deleted

A repository that has been superseded but is still standing gets a LEGACY prefix on its name, and it keeps that prefix until somebody confirms it is safe to remove from the main branch.

The prefix is aimed at people, not at machines. It stops the next person making changes to something that has already been replaced, which is a mistake that is easy to make and expensive to unpick. Keeping the old thing is the rollback plan: renaming a project is a change anyone can reason about, and hunting through history for the exact commit to revert to is not.

The cost is that dead repositories stay in the estate, visibly, and somebody has to keep deciding when each one is finally safe to remove. That decision gets deferred, which is how you end up with prefixes older than some of the people looking at them. The enterprise transactional email platform carries two solutions side by side on exactly this basis, with the older target framework documented as intentional rather than as an oversight.

Shared contracts, so old callers move on their own schedule

A contract package targets both the current runtime and the decade-old one, so callers on either side share the same types. Where that is not possible, the old and the new ship alongside each other as two packages rather than as one breaking upgrade.

This one was a bear. Wrappers and converters carry the old request models next to the new ones that gained functionality, and unit testing was the thing that made it survivable, probably more so than on any other project I have worked on. The alternative was telling teams on the old runtime that they had to move before they could have the new feature, which in practice means they do not get the new feature.

The cost is two runtimes to keep a package building against, and a lowest common denominator on anything the old target cannot express. You end up designing to the limits of a framework version you would rather forget. The internal real-time messaging bus, the payment solutions support platform and the shared internal package estate are all built this way.

Record the correction, and label an inference as an inference

A decision that gets superseded is struck through and dated rather than quietly overwritten. Where an implementer reasoned their way to an answer but nobody actually ruled on it, the entry says so, states the reasoning and states the alternative reading, and waits.

On the ACH and government payment processing suite the written record is checked mechanically against the code: retired terms surviving unmarked, citations of decisions that do not exist, verification claims whose covered files have since moved. One entry in that record documents that a file was excluded from scanning on the premise that a test covered it, that no such test existed, that this was the third instance of the same defect, and the date it was resolved.

The cost is that the record gets longer and messier to read than a clean one would be, and somebody has to keep the checks working or they rot into decoration. A tidy record is easier to read and quietly less true, and I would rather have the messy one.

Four problems worth the telling

None of these are the biggest things I have built. They are the four where the interesting part is not the fix but the reason the problem was hard to see in the first place, which is the part a list of technologies can never tell you.

The alert that lives in the browser tab

The most reliable place to put a warning for somebody who is not looking at your application is the strip along the top of their browser.

Operations staff keep the general-ledger upload system pinned in a background tab and spend their day in front of something else. A posting that errors needs attention now, not whenever somebody next clicks back. Posting time on that system carries a stated guarantee, so a failure sitting unnoticed for an hour is a broken promise rather than an inconvenience, and one of the approvers said as much.

Every obvious answer is the wrong shape. A banner, a toast, a badge on a count: all of them are perfectly visible to a person already looking at the page, and completely invisible to a person who is not, which is the only case that actually matters. So the page title itself flips to an error marker whenever errored posts exist, and the alert rides in the tab label where a person who is doing something else entirely will still catch it out of the corner of their eye. It has worked well enough that nobody has asked for anything else.

It cost something. Non-production environments are permanently full of stale errored posts, so the indicator is lit there almost all of the time, and the developer documentation has to say outright that this is not an application failure. An always-on warning in the environment developers spend the most time in is exactly how a warning gets trained out of people.

A setting of zero that could have read as no setting at all

The strictest possible configuration value and a configuration value that failed to load look identical if you check for the wrong thing.

The general-ledger upload system caps how far back a posting may be dated, and that cap is configuration. Zero is a legitimate value for it, meaning no back-dating is allowed at all. The natural way to ask whether configuration has arrived yet is to ask whether anything came back, and a cap of zero comes back as nothing.

Follow that through and it inverts. The client that reads the setting as not yet loaded does not enforce it, so the strictest setting anyone could configure would have produced the loosest behaviour the system has: back-dated rows straight through, silently, with the configuration screen showing the rule sitting right there. Nothing in the data would have looked wrong.

What is in the code is a one-line property that tracks loaded-ness by whether the floor date is set rather than by whether a count is above zero, with a four-line comment beside it explaining the trap. I would like to tell you I spotted this and defended against it. I did not. I checked the date because the date is always there and it was the easiest thing to write, and the safety fell out of an ordinary habit rather than out of a diagnosis. The rest of that path was deliberate: the rule is enforced on the client and the API no longer re-checks it, so when the cap cannot be loaded at all, every back-dated row is blocked while today and future rows still pass. Degraded and safe rather than degraded and permissive.

The test suite that is a bug list

Thirty-one tests that fail on purpose, and a red run that reads as the to-do list for whoever opens the repository next.

The submission and approval service behind new accounts runs on validators written fifteen years ago that hold thirty-one live defects, and the rules for the work said plainly that the tests were not to modify the production source. That is the whole trap. A characterisation test asserts what the code currently does, so writing one over a defect locks the defect in and calls it correct. The suite goes green, everybody relaxes, and the bugs have quietly become the specification.

So each test asserts what the code was meant to do instead, wrapped so that it fails while the defect is alive. The failure message names the defect, the production file and line, and what the code should do instead. Fix the source and the test turns green by itself, at which point the wrapper comes out. Each one carries a trait so a build gate can exclude them deliberately rather than by accident, and a defect that escapes as an exception declares its exception type, so an unrelated crash gets rethrown untouched instead of being quietly filed as a known bug.

This came out of a push to get unit, integration and end-to-end testing onto every internal repository, and fixing was explicitly not the goal of that push. The audience is the next developer to open the repository. They see a wall of failures, they clear those first, and then they do whatever they came in to do. The cost is that a permanently red suite has to be explained to every single person who runs it, and the invariant worth recording is the failure count rather than the pass count, because pass counts drift every time anyone adds a test.

The cleanup job that deleted live databases

A four-hour timezone difference in a housekeeping query turned a routine cleanup into something that destroyed a colleague's work mid-review, and it arrived looking like a login problem.

I wrote the end-to-end test infrastructure for the tool that hands out the next available account number. It provisions a throwaway database for each run and sweeps up the orphans left behind by runs that died, comparing each database's creation timestamp against a cutoff.

The creation timestamp is recorded in the server's local time. The cutoff was computed in a timezone four hours ahead. So the cutoff sat permanently in the future, every scratch database read as older than the grace period, and the sweep deleted databases belonging to runs that were still going. It destroyed a reviewer's probe database in the middle of their review, and what they saw was not a sweep behaving badly. It was a login that stopped working for no visible reason.

I cannot claim the find. It was surfaced by an automated review pipeline I built for exactly this kind of work: it reviews code deeply, raises an issue, validates the issue, attempts a fix, and has that fix validated and approved by the agent that raised it. The fix itself is one function call, comparing local time to local time, and it ships with the reasoning in a comment next to it. The more useful part is what had to happen first. The sweep could not be tested at all until it was refactored to take its prefixes and its grace period as parameters, because a creation timestamp is not something you can set, and with a hard-coded one-hour grace no test database could ever get old enough to reach the code path. It was untestable by construction, which is precisely why the defect was able to live there.