Clinical AI architecture · 20 August 2026 · Pathways
Clinical safety rules should not be model output
Safety-critical clinical screening should be implemented as deterministic, versioned rules rather than model output, because clinical governance requires reproducibility rather than accuracy alone. A rule that fires whenever its conditions are met can be tested, audited and re-run months later to produce the same answer — a property a probabilistic system cannot offer.
There is a reasonable-sounding argument that if a language model screens perioperative risk more accurately than a rule set, it should screen perioperative risk. The argument is wrong, and the reason it is wrong is not about accuracy at all.
Governance asks a different question
A perioperative committee reviewing a case does not primarily ask whether the system was right. It asks what the system was told, what it concluded, and whether that conclusion follows from the input. Six months later, in a morbidity and mortality review, it asks the same questions again — and expects the same answers.
That is a reproducibility requirement, and it is stricter than an accuracy requirement. A system that is right 97% of the time but cannot tell you why it was right in this case, and might answer differently if asked again, fails review even when it is right.
What deterministic buys you
A deterministic rule has three properties that matter here.
It is testable. “Flag an identified SGLT2 inhibitor with no documented withholding plan for the surgery date” is a condition you can write test cases against. You can enumerate what should fire and what should not, and you can run that suite on every change.
It is versioned. Every output records the rule version that produced it. When a case is re-examined in September, you can re-run it against the rule set as it stood in March rather than against the current one, which is the difference between an audit and an argument.
It cannot be talked out of it. This one sounds glib and is the most important. A rule fires whenever its conditions are met. It does not have a bad day, it is not swayed by the surrounding narrative, and it does not become less confident because the rest of the chart looks reassuring. For the specific set of findings that exist to protect patients, that inflexibility is the feature.
Where models belong
None of this is an argument against using models. It is an argument about which part of the pipeline they occupy.
Models are very good at the messy front end: conducting a conversational interview that follows up appropriately, reading a photograph of a medication box including handwriting, classifying an uploaded document, turning free narrative into structured findings. These are tasks where the input is unbounded and brittle rules genuinely fail.
The architectural line is drawn after that. Extraction and interpretation produce structured findings; those findings are verified against their source; and only then does a deterministic engine screen them. The model’s output is an input to the rules, never a substitute for them.
Two consequences worth stating
Confidence is shown, not hidden. If an extracted value is 86% confident, the profile says so, and the missing confirmation is listed explicitly. A system that presents uncertain extractions with the same visual weight as verified ones is making a clinical judgement it has no standing to make.
The system states what it does not know. Missing investigations and unconfirmed timings appear on every profile. This is the part most easily skipped, and it is what separates a screening tool from a reassurance tool — a clean profile that has not been told about the echocardiogram is not the same thing as a clean profile.
The design rule
Use models where the input is unstructured and the cost of an error is a human reviewing something unnecessarily. Use deterministic rules where the input is structured and the cost of an error is a patient harmed. Verify the boundary between them, and record which version of everything produced the answer.
That is not a compromise between safety and capability. It is what makes the capable part deployable in a setting that has to answer for its decisions.