If you are evaluating a reporting vendor, "our engine validates your data" is not information. Every vendor says it. The useful question is narrower and much harder to dodge: what, specifically, does it check — and how do you know the checks work?
This post answers both for our engine, in the form we think is actually checkable: a coverage map with counts, the six areas those checks cover, a description of the harness that exercises them, and an explicit section on what this evidence does not establish.
| What we validate | Count |
|---|---|
| Measure types covered | 8 — blood pressure, PHQ-9, weight/BMI, HbA1c, LDL-C, eGFR, uACR, MSK PROM |
| Distinct validation verdict codes | 12 |
| Rule families | 4 — validity-window boundaries, cadence/alignment machine, provenance & collection method, code recognition |
| Conformance test scenarios | 47, across 7 scenario families |
| Defect classes exercised end-to-end in the benchmark harness | 14 |
Those are the numbers we publish, and they are deliberately coverage numbers rather than performance numbers. More on that choice below, because it's the part of this post that says the most about how we operate.
Every check attaches a machine-readable verdict code, a human-readable reason, and a suggested fix — and exclusions are reported per beneficiary, per rule. That last detail matters more than it sounds: a validator that tells you "3,000 records failed" has handed you a second problem. One that tells you which beneficiary, which rule, and what to do about it has handed you a work queue.
1. Validity windows. Collection-to-submission windows per measure — the 15-day device and PROM window, the 1-year versus 2-year lab windows, including the condition-dependent overrides (HbA1c runs on a 1-year window on the diabetes and prediabetes tracks and 2 years otherwise). Every edge gets boundary-pair tests on both sides, plus a timezone-rollover fixture, because "one day late" is a bug class that only ever shows up at the boundary.
2. Cadence and alignment. The full per-beneficiary submission clock: the 60-day baseline deadline whose miss causes unalignment, the rolling 70–110-day quarterly windows keyed to the prior accepted submission, the day-425 final deadline, and the track-specific early-success windows. Boundary pairs on each. (We walked this machinery in detail in the two clocks and the rolling window.)
3. Provenance and collection method. No manual entry where a device measure is required. The ≥3-reading validated-cuff requirement for blood pressure. Self-report permitted only where CMS allows it — weight and PROMs — and that one carries an over-rejection guard test, because a validator that rejects legitimate self-report is failing in the direction nobody notices until submissions are already thin. CLIA and accreditation required for labs. Device identity and validation status required. And pipeline provenance — a raw payload hash and transform version — attached to every observation.
4. Code recognition and FHIR conformance. Unknown measure codes rejected. Bundle profile, Parameters wrapper, participant-ID format, value-set bindings, and US Core conformance failures are all surfaced as validation errors, per our reading of CMS's draft spec.
5. Transport and lifecycle. The full async submit contract — OAuth, POST, 202, poll to a terminal state — plus the eligibility-check, align, and unalign operations with their input-validation and subscription side effects.
6. Guardrails. Automated tests assert that dashboard and alert copy stays administrative-only, with no clinical language, per our FDA enforcement-discretion posture. To be explicit, because this is a place vendors get slippery: that is a posture we hold ourselves to and test for, not a clearance. We hold no FDA clearance and claim none.
Coverage counts are only as good as the thing that exercises them, so here is the harness, described plainly.
We generate a seeded synthetic cohort — every patient synthetic, no PHI anywhere, and the pipeline hard-refuses to run unless the synthetic-only flag is set. We then plant deliberate defects in it: 14 distinct defect classes, each injected into dedicated synthetic beneficiaries, each mapping to a validity rule the engine is supposed to enforce. Expired collection windows. Per-measure window overrides that should fire where the generic window wouldn't. Collection timestamps dated after the submission. Missing timestamps. Manual keying where device capture is required. And so on.
Then the whole cohort — clean records and planted defects together — runs the full path: engine → packaging → submission to our own test server built to CMS's draft ACCESS reporting spec → poll to a terminal state. Not a unit test of the rule function; the actual end-to-end path a real submission would take.
Two things about that server, stated as directly as we can: it is ours, and it is not a CMS system. It is our implementation of CMS's published draft submission contract. Bundles reaching a terminal accepted state means our bundles satisfied our implementation of a published draft. We describe it that way every time, including in sales conversations, because the alternative phrasing is the single most common oversell in this category.
Run the harness twice and you must get the same answer, or none of the coverage above means anything.
All randomness flows from one pinned seed — the cohort, the observation streams, and the placement of every injected defect are all drawn from it, so they regenerate identically on the next run. That alone is not sufficient, and the reason is a failure mode worth knowing about even if you never look at our code: a pipeline that iterates over an unordered collection can produce different results under a different hash seed, silently, on a machine otherwise identical to yours. So the pipeline pins the order at the points where it would otherwise be left to chance: beneficiary references are sorted before a bundle is assembled, and per-rule results are sorted before the report is written.
There's a second design decision inside that worth stealing. Two runs on two machines can never match on everything — wall-clock timings differ, host fingerprints differ, generation timestamps differ. A blanket "byte-identical output" promise is therefore either false or quietly excludes so much that it shows nothing. So we separate the substantive output from the volatile: every count and rate is regenerated from the pinned seed and must match, while wall-clock throughput is expected to vary with hardware and is recorded with the hardware it ran on rather than promised. And the report is generated from the machine-readable metrics file, so every number in it traces back to a measured value by construction rather than by transcription.
We keep an internal reproduction kit that does exactly this against frozen results. It isn't a public download today — that's a real limitation, and it's the honest state of things rather than a coming-soon.
Here is the part that we suspect tells you more about us than the coverage table does.
We run that harness and it produces performance figures — catch rates, false-positive counts, per-class tables. We deliberately don't publish them. When those numbers were reviewed internally, the decision was to publish the coverage story — the number of checks and the areas covered — and keep the numeric performance reports internal. This post honors that scope, ten days later, when nobody outside would have known the difference.
What the internal reports document, in the one sentence we're cleared to carry over: "all 14 injected defect classes are caught with the exact expected verdict code, with zero false positives on the clean stream, deterministically reproducible under a pinned seed."
We're not going to dress that up with the underlying figures, and here's the reasoning we'd want a buyer to hold us to: those are test results over data we generated, containing defects we injected. Publishing precise-looking accuracy percentages from a test we designed against ourselves would give you a number with the appearance of external validity and none of the substance. The coverage map is the honest artifact — it tells you what the engine attends to, which is the thing you can actually compare against your own requirements.
Over-doing this section is the point of it.
The defects are ours. We defined all 14 classes and planted them in data we generated. That exercise shows the engine enforces its own rule table correctly and deterministically. It says nothing about defect classes outside that table — and 14 classes are not the universe of real-world defects. Real pipelines produce failure modes nobody has enumerated: novel formats, partner-specific mappings, upstream corruption we've never seen.
This is plumbing evidence, and only plumbing evidence. No clinical claim, no outcome claim, no representativeness claim can be read from any of it. Nothing here says anything about patient outcomes, about care, or about what happens to your attainment.
The data is synthetic. No PHI touches any of this, which is a privacy feature and an evidentiary limitation at the same time. Synthetic data is generated by assumptions; real data is generated by reality, and reality is messier.
The accepting server is ours, and the spec is a draft. ACCESS reporting requirements are still moving. The engine is built to the published draft contract as it stands; the rules are data, versioned, and expected to change as CMS publishes more.
We have no third-party audit of any of this, and no reference customers. Both are on the list of things we don't have, which we keep in the open rather than waiting to be asked.
If you take one thing from this post and use it against us, use this: ask any reporting vendor for their coverage map — measure types, rule families, verdict codes, and how each rule is tested at its boundaries — and ask what their evidence doesn't establish. The second question is the one that separates vendors quickly. A vendor who can't name a limit either hasn't looked or isn't telling you.
Ours are above. If your list has something on it that our map doesn't cover, that's the most useful thing you could send us.
---
Outcome Rail builds reporting infrastructure for ACCESS participants. If you want to walk the coverage map against your own measure set, or you think we're missing a rule family: [email protected].
Sources: Coverage counts and the six covered areas come from our own validation coverage summary, the artifact approved for publication under our internal benchmark review; rule mechanics (validity windows, the 60-day and 70–110-day and day-425 clocks, collection-method requirements) trace to CMS primary sources carried in the earlier posts in this series — CMS ACCESS Model page, the ACCESS Request for Applications (Appendix C), and the ACCESS Payment Amounts & Performance Targets paper. All benchmark data is synthetic and contains no PHI; the submission target is our own test server built to CMS's draft ACCESS reporting spec, never a CMS system. This post states no measure-target values, no clinical, outcome, or representativeness claims, no FDA clearance, and no payment claims; it is not legal advice.
Device, lab, and PROM data in; compliant FHIR submissions out — validity windows, cadence clocks, and provenance rules enforced before CMS ever sees the bundle. We're onboarding a small founding cohort of design partners this quarter.