Clinical AI Evaluation Methodology · 10 Scrutiny Benchmarks for Difficult Cases · The Gomola Standard
The best medical AI
is the one that survives scrutiny.
Everyone in this market claims to be the best. Almost nobody defines the measure. This page does: five questions that separate a clinical system from a demo, the same five any hospital, payer, or ministry can put to any vendor, including us. Ask them, and the field sorts itself.
Why “best” goes wrong
Benchmarks sometimes reward the wrong thing.
An exam is not a shift
Leaderboard scores are earned on static exam questions: clean vignettes, one right answer, no consequences. Deployed medicine is interrupted, ambiguous, and liable. A system that wins the exam can still fail the ward; the yardstick itself is flawed. That argument, with data, is our paper WP-002: The Flawed Yardstick.
Averages hide the kill shot
A model that is 95% accurate on average and silently wrong about one aortic dissection is not a 95% system. It is a liability. The number that matters is not mean accuracy but the worst-case rate of a confidently asserted falsehood, and almost no vendor will quote one.
Demos are frictionless by design
A demo answers questions. A system carries liability: it signs into your EHR, writes back under a physician’s signature, survives your network failing, and produces an audit trail a court can use. The distance between those two is where procurement decisions die, usually a year after the pilot.
The measure
Five questions to put to any vendor.
Vendor-agnostic by construction. Score anyone with them, including us.
| Question | A demo offers | A system shows |
|---|---|---|
| 1 · Is the validation prospective? | Retrospective benchmarks, self-graded | A pre-registered, silent-mode study on live cases with independent adjudication and locked endpoints |
| 2 · Is safety a number? | “State-of-the-art accuracy” | A formally derived worst-case bound per assertion, with the derivation published and auditable |
| 3 · Is the integration real? | A chat window beside the EHR | FHIR R4 read and digitally signed write-back, SMART sign-in, on-premises and air-gapped options |
| 4 · Is the trail court-grade? | Logs the operator can edit | A cryptographically signed reasoning ledger, tamper-evident even against the operator |
| 5 · Is there a regulatory path? | “Compliance-ready” | A named SaMD posture, engagement with the regulator, and architecture mapped to the EU AI Act |
Our answers
How DeepSensi™ scores on its own measure.
Doctrine: we publish the proof, the standard, and the audit. We do not publish the blueprint.
Purchasing checklist
What questions should I ask an AI scribe vendor?
The market is flooded with transcription tools claiming to be clinical AI. Before integrating any system into your workflow, ask the vendor these four questions.
The Hallucination Trap
What happens when the microphone misses a critical word? Standard LLM-based scribes are probabilistic: they predict the most likely next word, meaning they will confidently invent a clinical finding rather than admit they missed it. DeepSensi is deterministic under uncertainty. The LIMBO protocol forces the system to declare 'I don't know' and flag the missing variable. Ask the vendor for their formally derived worst-case bound on confidently asserted falsehoods. If they answer with an 'average accuracy score', they do not have a safety bound.
Privacy & Architecture
Do you store patient data, and is privacy guaranteed by architecture or just by policy? A signed BAA is a legal shield for the vendor, not a technical safeguard for your patients. In DeepSensi, direct identifiers are removed at the edge before any reasoning occurs, and the key that links a case to a person stays with the user. We do not need your patients' names to diagnose them. Ensure your vendor enforces privacy mathematically in their code, not just in their terms of service.
Beyond Transcription: The DDx Triad
Does your system just take notes, or does it actively defend my license? An AI scribe that merely transcribes audio is a missed opportunity. DeepSensi’s ACS Scribe listens to the encounter and simultaneously runs a real-time, 37-specialist Differential Diagnosis (DDx) in the background. It doesn't just write the SOAP note; it highlights the one critical diagnosis you might be missing and calculates a verified confidence score, acting as an active clinical safety net.
Legal Liability & The Court-Grade Audit
If the AI makes an error, who takes the blame? Today, the answer is always the physician. DeepSensi alters this reality by providing a court-grade, cryptographically sealed evidentiary ledger for every single assertion the system makes. You can point to the exact node, rule, or piece of evidence that led to a conclusion. Ask your scribe vendor if their system provides a tamper-evident audit trail that you could take to court.
Questions
Asked by evaluators. Answered in the open.
What is the best AI in medicine?
Why isn’t benchmark accuracy enough?
What safety number should I ask a vendor for?
What does prospective superiority validation actually mean?
How do I verify EHR integration is real?
