23 July 2026
From promise to evidence: what institutionalized agent governance actually asks for
There is a quiet but important change happening in how AI agents are governed. The early language was voluntary: organizations pledged to use agents “responsibly.” The direction of travel now, visible in the way national AI safety bodies are formalizing testing agreements and operational guidance, is toward institutionalized controls: named risks, defined controls, pre-deployment and ongoing tests, monitored signals, and an audit trail a reviewer can inspect. The question is no longer “do you promise to be careful?” It is “show me the test that proves it, and the log that proves it held.”
I will not spend this note summarizing the policy trend. What is more useful, and harder, is to show what that trend looks like when you actually try to implement it on one system. So this note is built around a tool: the agent risk-control matrix I built for the patient-facing medical digital twin I am developing.
Why a registry is not enough. In the same governance pack, I keep an agent registry that lists every agent, its owner, its PHI boundary, and its permission tier. That answers what exists and who is responsible. It does not answer the question an auditor or a safety regulator will actually ask: for this specific risk, where is the control, and how do you know it works? A registry is a roster. Institutionalized governance needs a proof.
The move that makes it work: organize by risk, not by agent. The matrix has one row per risk, not per agent, because the same risk (acting without human approval, accessing PHI beyond scope, retaining memory it should not, operating after a safety trigger) recurs across many agents. For each risk it records five things:
- a control (the mechanism that limits the risk);
- a control type (preventive, detective, or corrective);
- a test (how the control is checked before deployment and re-checked on every model, prompt, tool, or PHI-scope change);
- a monitored signal (what a dashboard shows if the control is failing, so failure is seen before an incident, not during one);
- an audit-evidence record (the log or artifact a reviewer can inspect without taking the team’s word for it).
That last column is the whole point. “We require human approval for patient-facing actions” is a promise. “Here is the approval gate, here is the pre-deployment test that showed an unapproved action is blocked, here is the dashboard metric that would flag a bypass, and here is the approval log with approver, timestamp, and action reference” is evidence. The matrix is the structure that turns the first sentence into the second.
This connects to something I wrote earlier this week. In an earlier note on model and cloud dependency, the underlying move was the same: a dependency you hope is safe becomes governed only when it has a named owner and a documented fallback. Here, an agent you promise is safe becomes governed only when each risk has a test and an evidence trail. Both are the same shift stated twice: from promise to evidence. That, more than any single regulation, is where medical AI governance is going. The organizations that will cope with it are the ones that can hand a reviewer not a pledge, but a gate, a test, a dashboard, and a log.