Retiring a Paid Data Vendor

The problem

A national provider directory was built on self-attested and vendor-verified data. Two structural problems sat underneath it.

Trust was binary and invisible. A downstream consumer could not tell whether a provider's license number came from a state licensing board or from a form the provider filled in three years ago. Both arrived in the same column, looking identical. There was no machine-readable measure of how much to trust any given value — so every consumer either trusted everything or trusted nothing.

Primary source verification was expensive and lagging. Credentialing data was bought from a third-party verification vendor: per-record cost, refresh latency, and coverage gaps — including in states that publish their own licensing rosters for free.

The mandate was to build a gold enrichment layer that re-sources provider credentials onto authoritative primary sources, attaches a per-field confidence score and provenance stamp to every value, and progressively retires the paid vendor state by state.

Approach

A medallion architecture (bronze → silver → gold) producing 21 gold tables and 519 business columns, orchestrated as scheduled jobs.

Provenance on every field

Every enriched field carries four columns instead of one:

LicenseNumber	LicenseNumberVerifiedSource	LicenseNumberVerifiedOn	LicenseNumberConfidenceScore
A12345	CA Board of Registered Nursing	2026-07-31	1.00

Downstream systems can filter, rank, or route on trust rather than guessing at it. This is the difference between a data product and a data dump: the consumer can make their own risk decision because the evidence travels with the value.

Corroboration-based confidence scoring

The original rubric scored on recency alone — 0.95 if verified after attestation, 0.85 within 180 days, and so on. I replaced it with a source-authority and corroboration rubric:

• 1.00 — authoritative source, corroborated by the provider's self-attestation
• 0.95 — authoritative source, nothing to corroborate against
• 0.85 — authoritative source disagrees with self-attestation
• NULL — no provider linkage, or no value

The 0.85 case matters most. A recency-based score hides disagreement; this one surfaces conflict as a signal rather than silently picking a winner. Freshness moved into the VerifiedOn column where it belongs, so a stale-but-authoritative value is no longer indistinguishable from a fresh guess.

Re-sourcing onto authoritative registries

Four significant rewrites, each a source-of-truth change rather than a cosmetic one:

Table	Before	After
Provider spine	Membership-ID spine — only providers already known to the organization	National NPPES registry — 7.4M individual NPIs, with the membership ID demoted to a nullable attribute
DEA	Paid verification vendor	Federal DEA registry, linked via state license → taxonomy → NPI
Controlled substances	Paid verification vendor	9 state registries, vendor as backfill only
Medicare opt-out	Paid verification vendor	CMS federal file

The spine re-sourcing was the pivotal call. It flipped the platform from "enrich the providers we already have" to "cover the national provider population, and attach our identity where it exists." That is a strategic change disguised as a schema change.

Onboarding 34 state boards

The bulk of the work. Each board is a hybrid two-branch union: the board's published roster loads as authoritative, while the paid vendor remains as backfill only for professions that board does not cover — most "medical board" extracts are physicians-only, so nursing, pharmacy, and dental stay on the vendor until a matching registry exists.

The hard part was de-duplication across the seam. License numbers are formatted inconsistently between state boards, the national registry, and the vendor: letter prefixes, board-type codes, leading zeros. A naive numeric-core match collapsed genuinely distinct licenses. A strict match left duplicates behind.

The solution was a provider-scoped collision-recovery dedup: match on numeric core, but treat a collision as safe only when it resolves within a single provider. That recovered 28,082 records the prior exact-match logic had wrongly suppressed — records that had simply been missing from the directory.

Engineering discipline

• Environment parameterization so one notebook body runs against dev, test, UAT, and production with zero hardcoded catalog names
• Asset bundles with per-target overrides; jobs read notebooks directly from git
• CI/CD with automatic deployment to lower environments and approval gates above them
• Validation cells in every notebook — row counts, score distributions, and linkage rates run as part of the job, so a regression appears in the run output rather than three weeks later in a customer complaint
• A generated data dictionary and coverage-metrics pack for executive leadership, turning pipeline internals into a business-readable progress narrative

Outcome

Gold tables in production	21 tables / 519 columns
Provider spine	7.4M individuals; 2.84M matched to an internal ID (38.3%)
License records	11.59M, of which 10.99M (94.9%) from state boards
State boards onboarded	34, up from 22 the prior cycle
Paid verification fully retired	14 states
Vendor records remaining	595K (5.1%), down from ~6.9%
DEA registrations	2.17M loaded; 68.2% linked to a provider
Controlled-substance registrations	491K, up from 198K; 74.8% registry-sourced
Locations	1.55M records; 99.5% provider-linked

Beyond the numbers, the reporting pack made vendor retirement plannable. The leadership deck ranks remaining targets by records eliminated, so the next quarter's roadmap is driven by measured cost reduction rather than intuition.

What transfers

1. Provenance belongs on the field, not in a README. If trust cannot travel with the value, downstream consumers will invent their own rules for it.
2. Score authority, not recency. Recency is a property worth recording, but it is not a measure of truth.
3. Make disagreement visible. A confidence rubric that cannot express "these two sources conflict" is hiding your most valuable signal.
4. Validation belongs inside the job. Row counts and distribution checks that run on every execution catch regressions when they happen, not at quarter close.
5. Re-sourcing a spine is a strategy decision. Which population you cover is determined by which key you build on.