Project-derived · pre-deployment · sanitized

Claims and Evidence Register (project-derived, sanitized)

A register of every claim a patient-facing system intends to make, set against the evidence that would actually support it. Each row records the claim as it would be worded, the evidence held, what that evidence supports rather than what it is cited for, the gap between the two, and the specific study or check that must exist before the claim may be made at all.

What this artifact is for

The rest of my governance pack answers organisational questions: who owns a risk, who may override the system, what is monitored, who approves a change. This register answers the question that comes before all of them, and that a governance structure cannot fix once it has been skipped: does the evidence support what the system says about itself?

It is a different instrument from a validation report. A validation report states what was measured. This register starts from the other end, from the sentence a patient, a clinician, a sponsor, or a regulator will actually read, and works backwards to whether anything in the evidence base entitles the project to say it. The two do not automatically match. A model can be validated and still be described in language its validation does not license.

Every row therefore separates two things that are usually collapsed: the evidence a claim cites, and what that evidence supports. The gap between them is the governed object.

This artifact is drawn from the governance pack of the patient-facing cardiometabolic digital twin I am building at the HKU-Avnet Joint AI Laboratory. It is sanitized, and it is pre-deployment in a specific sense worth stating plainly: the claims below are the claims the project intends to make, not claims it currently makes. Several rows record that the evidence required does not yet exist. That is the register working, not the register failing.

Why I built this (builder’s perspective)

Two things led here. The first is eight years of peer review for machine-learning venues, where the single most common defect is not a wrong method but a claim that has quietly outgrown the experiment underneath it: a result generalised past the setting it was measured in, a gain that no ablation attributes, a comparison run on a protocol the authors also designed. Nothing about that pattern is specific to machine learning. It is what happens whenever the party making a claim is also the party that chose how to test it.

The second is closer to home. Earlier this month I audited my own governance pack and found that documents registered as evidence for other claims did not contain what they were registered as, and that two claims resting on them were therefore false. That audit worked on documents. It left the harder layer untouched, because a document can exist, be correctly named, and still describe a performance the data does not support.

So this register turns the same instrument on my own system’s performance and clinical claims, before there is any external pressure to make them. Writing it while the honest answer to several rows is no evidence yet is the entire point. A claims register drafted after a launch deadline arrives is a marketing review.

Fixed metadata

Claims and evidence register

The gap taxonomy

The middle columns are the reusable part. Across domains, claims fail against their evidence in a small number of recurring ways, and naming the type is what makes the required fix specific rather than a general instruction to be careful.

The one that is hardest to catch

Six of the seven types can be found by reading carefully. Circularity cannot, because every individual document is internally consistent. It appears when the same party defines the claim, designs the evaluation, selects the comparator, and reports the result, and the failure is invisible at document level precisely because nothing is wrong at document level.

In medical AI this arrives in familiar clothing: an evaluation protocol written by the developer, a comparator built by the developer rather than an accepted standard of care, a retrospective cohort selected after the results were seen, a threshold chosen on the same data it is reported on. Each step is defensible alone. Together they produce a number that measures the design of the test more than the performance of the system.

The structural control is not sharper reading. It is the same separation this pack applies everywhere else: the party that delivers a component does not certify it. For claims, that means the evaluation protocol and the comparator are fixed before results are seen, by someone who does not answer for the result. Where that separation cannot be arranged, the honest move is not to claim less confidently. It is to record, in this register, that the evidence is developer-controlled, so that a later reader can weigh it accordingly.

Reading is not verification

A register can be filled in accurately by someone who only ever read the evidence. Some rows require recomputation, and I keep these separate because they are the ones that catch defects reading does not reach:

• Recompute the headline metric from the reported confusion matrix or raw counts, and check it against the number in the text.
• Recompute calibration, not only discrimination, since a well-ranked model can be systematically wrong about absolute risk at the threshold where it is acted on.
• Re-split by subgroup rather than accepting the subgroups the report chose to show, and record which subgroups fall below a usable sample size.
• Check the operating threshold is the one the metric was reported at, and that it was not selected on the same data.

Where I cannot recompute, the register says so. A row verified by reading and a row verified by recomputation are different strengths of evidence, and collapsing them is the same error this artifact exists to prevent.

The design principle

A claim is governed when the project can answer, for each sentence it intends to say about the system: where the sentence will appear, what evidence is cited for it, what that evidence supports as distinct from what it is cited for, which gap type separates the two, who owns closing it, and what specific study or check must exist before the sentence may be used.

No claim reaches a patient, a sponsor, a regulator, or a manuscript before it has a row here. A claim without a row is not a small documentation gap. It is a statement nobody has been made responsible for supporting.

What changed, and why

This section is permanent; each revision adds to it rather than replacing it, so that the page can be checked rather than merely read.

v0.1 (30 August 2026). First publication. Three rows are recorded with no supporting evidence at all, and one row downgrades a claim the project could plausibly have made in good faith, that clinicians had reviewed the system, to what the review actually covered. Both are deliberate. A claims register whose first version shows every claim already supported would be evidence that the register was written to be shown rather than used.

Related note: An index that lists a file is not evidence the file contains anything applies the same instrument one layer down, to the documents a governance pack claims to hold.

← Back to Governance Portfolio