# Clinical safety rules should not be model output

> Safety-critical clinical screening should be implemented as deterministic, versioned rules rather than model output, because clinical governance requires reproducibility rather than accuracy alone. A rule that fires whenever its conditions are met can be tested, audited and re-run months later to produce the same answer — a property a probabilistic system cannot offer.

*Source: https://getpathways.ai/insights/deterministic-rules-in-clinical-ai*

There is a reasonable-sounding argument that if a language model screens
perioperative risk more accurately than a rule set, it should screen
perioperative risk. The argument is wrong, and the reason it is wrong is not
about accuracy at all.

## Governance asks a different question

A perioperative committee reviewing a case does not primarily ask whether the
system was right. It asks what the system was told, what it concluded, and
whether that conclusion follows from the input. Six months later, in a morbidity
and mortality review, it asks the same questions again — and expects the same
answers.

That is a reproducibility requirement, and it is stricter than an accuracy
requirement. A system that is right 97% of the time but cannot tell you *why* it
was right in this case, and might answer differently if asked again, fails
review even when it is right.

## What deterministic buys you

A deterministic rule has three properties that matter here.

**It is testable.** "Flag an identified SGLT2 inhibitor with no documented
withholding plan for the surgery date" is a condition you can write test cases
against. You can enumerate what should fire and what should not, and you can run
that suite on every change.

**It is versioned.** Every output records the rule version that produced it. When
a case is re-examined in September, you can re-run it against the rule set as it
stood in March rather than against the current one, which is the difference
between an audit and an argument.

**It cannot be talked out of it.** This one sounds glib and is the most
important. A rule fires whenever its conditions are met. It does not have a bad
day, it is not swayed by the surrounding narrative, and it does not become less
confident because the rest of the chart looks reassuring. For the specific set
of findings that exist to protect patients, that inflexibility is the feature.

## Where models belong

None of this is an argument against using models. It is an argument about which
part of the pipeline they occupy.

Models are very good at the messy front end: conducting a conversational
interview that follows up appropriately, reading a photograph of a medication box
including handwriting, classifying an uploaded document, turning free narrative
into structured findings. These are tasks where the input is unbounded and
brittle rules genuinely fail.

The architectural line is drawn after that. Extraction and interpretation produce
structured findings; those findings are verified against their source; and only
then does a deterministic engine screen them. The model's output is an input to
the rules, never a substitute for them.

## Two consequences worth stating

**Confidence is shown, not hidden.** If an extracted value is 86% confident, the
profile says so, and the missing confirmation is listed explicitly. A system that
presents uncertain extractions with the same visual weight as verified ones is
making a clinical judgement it has no standing to make.

**The system states what it does not know.** Missing investigations and
unconfirmed timings appear on every profile. This is the part most easily
skipped, and it is what separates a screening tool from a reassurance tool — a
clean profile that has not been told about the echocardiogram is not the same
thing as a clean profile.

## The design rule

Use models where the input is unstructured and the cost of an error is a human
reviewing something unnecessarily. Use deterministic rules where the input is
structured and the cost of an error is a patient harmed. Verify the boundary
between them, and record which version of everything produced the answer.

That is not a compromise between safety and capability. It is what makes the
capable part deployable in a setting that has to answer for its decisions.