The situation
The organization operates a national coordination-of-benefits registry — the system that identifies where a member has overlapping health-plan coverage and determines which payer is primary. It ingests roughly 250 million member records per week from health-plan payers, about 1.2 TB per week compressed at the raw layer.
The entire system ran inside a third-party vendor's stack: SSIS, SQL Server, VB.NET, and a Java identity-matching service.
Which produced an uncomfortable asymmetry. The organization owned the data and carried the regulatory exposure, but had:
- No visibility into the processing logic
- No direct control of unit cost
- No ability to change matching or primacy rules without going through the vendor
That is the worst position to be in with regulated data. You are answerable for outcomes produced by a system you cannot inspect, on a cost base you cannot influence, using rules you cannot change. Every question a regulator might ask has to be routed through the party whose commercial interest is served by the answer taking a while.
The programme was to move it onto a lakehouse the organization actually controls. I led the data architecture and data engineering workstream on the client side.
What client-side authority actually means
This is worth being precise about, because "led the architecture workstream" can mean anything from writing a document to holding a veto.
I set the target-state direction, ran architecture review, and held formal approval authority over the data integration architecture, the security model, and the test strategy produced during initiation. Deliverables did not pass out of the initiation phase without that approval.
That is a different job from designing a system. It is deciding what is acceptable, in an environment where the people producing the designs have their own delivery pressures, and where accepting something merely workable during initiation becomes permanent by build.
Target-state architecture
A medallion design with six layers, each with a specific job:
Landing → Bronze → EMPI → Silver → Primacy → Gold
Plus dedicated schemas for configuration, exceptions, and platform telemetry — deliberately separated rather than scattered through the pipeline layers.
Those three are the ones that get skipped under delivery pressure and are painful to retrofit. Configuration as data rather than embedded in code is what lets a matching rule change without a deployment. Exceptions as a first-class destination is what stops unmatched records from disappearing into a log file. Telemetry as its own schema is what makes "is the pipeline healthy?" a query rather than an opinion.
Putting EMPI and Primacy in as named layers matters too. Identity resolution and payer primacy are the two places where this platform makes decisions that affect real people's coverage. Giving each its own layer means the output of each is inspectable, testable, and attributable — rather than being an intermediate state inside a larger transformation.
Architecture Review Board
I chaired and served on the weekly technical review arbitrating design decisions across the data, application, infrastructure, and security tracks.
The part I would call out is this: decisions were held open as tracked ADRs rather than being closed prematurely.
Architecture review boards fail in one of two directions. They rubber-stamp, and the design drifts. Or they force closure on every question in the meeting where it is raised, which converts genuine uncertainty into a decision made on the worst available information and then treated as settled.
Recording something as an open decision record — with the options, the constraints, and what would need to be true to choose — is how you keep the question live without blocking delivery. It also leaves a trail explaining why the architecture is what it is, which is the artefact everybody wants two years later and nobody writes at the time.
Governance and access model
I approved the Unity Catalog topology, the RBAC/ABAC model, row-level payer isolation, and the column-masking policy for PHI and PII.
Row-level payer isolation is the load-bearing control here. This is a registry aggregating data from competing health plans in order to work out coverage overlap. The entire proposition depends on each payer's data being usable in aggregate while remaining invisible to the others. That is not a preference — it is the condition under which payers participate at all.
Getting that wrong is not a security incident. It is the end of the business model.
Scope note
This covers the architecture and initiation phase — twelve weeks, completed mid-2026. The write-up describes the target-state design, review structure, and governance model established during that phase, not the delivery that follows it.
What transfers
- Owning the regulatory exposure without owning the system is the real risk, and it is usually invisible on an architecture diagram. Ask who can answer a regulator's question without making a phone call.
- Give the decision points their own layers. If identity resolution and primacy determination are buried inside a transformation, neither is testable, and neither is explainable when challenged.
- Configuration, exceptions, and telemetry deserve their own schemas. All three get folded into pipeline code under delivery pressure and all three are painful to extract later.
- Hold decisions open as ADRs rather than forcing closure. Premature certainty is more expensive than a tracked open question.
- Approval authority has to be formal to be real. Advisory input on a design gets overridden by the delivery schedule. Sign-off does not.