Author: Tomasz Jan Gomoła · ORCID 0009-0001-5222-6154 Affiliation: DeepSensi Medical OS — Cognitive Infrastructure Division, DeepSensi PBC, Dover, DE, USA Version: 3.0 — July 2026 Classification: Public White Paper — Companion to DSS-001 (Standard) and WP-001 (FTA) Status: Prepared for submission — medRxiv preprint; NEJM AI / npj Digital Medicine Recommended citation: Gomoła, T.J. (2026). "The Flawed Yardstick: Why Static Medical Benchmarks Penalize Clinical Safety." Technical Whitepaper WP-002. DeepSensi Medical OS.
Existing clinical AI evaluation frameworks rely on static, multiple-choice question (MCQ) datasets (e.g., MedQA, USMLE, and standard HealthBench) that evaluate memory retrieval rather than clinical reasoning. This paper defines these frameworks as a Flawed Yardstick that systematically penalizes safety-oriented clinical behaviors. Specifically, static datasets punish systems that declare uncertainty (the LIMBO Protocol), request additional history, or refuse to prescribe treatments when critical red flags have not been excluded (the Safe Triage Paradox).
We document specific cases from an audit of HealthBench Hard (e.g., ACLS cardiac arrest case 0b8f1d60 and ACL injury case 9ab66439) in which safety-oriented deterministic guards were graded 0% due to negative-rubric scoring defects or prompt–intent mismatches. To address this, we propose the Automated Clinical Safety Audit (Auto-CSA) — a dynamic evaluation framework that tests clinical AI systems using generative patient trajectories, adversarial bias injection, and explicit uncertainty scoring. Applied to a consolidated cohort of N = 301 NEJM Clinicopathological Conference (CPC) cases (2014–2023) with an independent cross-vendor adjudication panel (as required by the DSS Standard), the reference implementation (DeepSensi Medical OS) achieved a baseline top-1 accuracy of 80.0% and top-3 of 88.0% under a frozen configuration, rising to 86.0% top-1 (95% CI 82.1–89.9) and 93.7% top-3 (95% CI 91.0–96.4) after safety-first calibration — reported explicitly as a development-cohort result, with a 0.0% missed-critical rate and a median core deliberation time of 14.3 seconds. Clinical safety and high diagnostic accuracy can be unified — but only under an evaluation paradigm that measures safety at all.
Keywords: Medical AI Benchmarking, Clinical Safety Audit, Safe Triage Paradox, Negative Rubrics, EBM Safe Overrides, HealthBench Hard, Auto-CSA
The rapid integration of Large Language Models (LLMs) into clinical medicine has driven a demand for objective evaluation metrics. To meet this demand, the machine-learning community has repurposed medical licensing examinations (USMLE, MedQA) and static clinical datasets as benchmarks. These benchmarks measure accuracy by comparing an LLM's output against a single, pre-determined "ground truth" answer — typically a multiple-choice key or a brief text snippet representing a textbook case.
This approach creates a dangerous illusion of clinical readiness. A model that achieves 95% on a multiple-choice medical exam is assumed to be clinically safe. In reality, static benchmarks act as a flawed yardstick: they reward memory recall and speculative guessing while systematically penalizing the defensive, safety-oriented behaviors that define high-quality clinical care.
In this paper we expose the structural defects of static clinical benchmarks, describe the Safe Triage Paradox in which safety gates are graded as failures, and present the Automated Clinical Safety Audit (Auto-CSA) as a dynamic, safety-first alternative for validating clinical AI systems.
Many automated evaluation scripts parse LLM outputs using substring matching or heuristic classifiers. When a clinical AI system implements a safety gate — such as refusing to triage a patient with chest pain without first excluding acute coronary syndrome — the output does not contain the expected diagnosis string. Because the script expects the target string (e.g., "Gastroesophageal Reflux Disease") and instead finds a safety refusal or a request for cardiac biomarkers, it assigns 0%. The system is penalized not for being wrong, but for refusing to make an unsafe, speculative diagnosis.
Static benchmarks assume clinical reasoning is a unidirectional monologue: prompt in, complete definitive plan out. This contradicts the iterative nature of evidence-based practice. Safe clinical care is a dialogue: clarifying questions, staged diagnostics, hypotheses revised as data arrives. Static answer keys can neither evaluate nor reward these iterative safety-gathering steps.
MCQ formats force selection of a single diagnosis from a closed set. This rewards overconfidence: a model 51% confident in option A and 49% in option B is incentivized to output A as definitive fact. In clinical reality, presenting a diagnosis with unwarranted confidence and no differential is a severe reasoning error. Static benchmarks actively train models to hide uncertainty.
Safe Triage Paradox: A clinical AI system that prioritizes patient safety by admitting uncertainty or refusing to proceed without excluding red flags is graded as failing, while an unsafe system that confidently guesses the keyed answer is rewarded.
Consider a patient presenting with back pain and minor urinary retention. A raw LLM, matching the answer key, immediately outputs "lumbago" and recommends physical therapy — scoring 100%. A safety-certified system recognizes urinary retention as a red flag for cauda equina syndrome (a surgical emergency), refuses to recommend physical therapy, and generates an urgent MRI referral — scoring 0% because it failed to output "lumbago".
Unsafe system: [Prompt] ──► [Presumptive guess: lumbago] ──► [Score: 100%] (dangerous)
Safe system: [Prompt] ──► [Safety gate: cauda equina?] ──► [Score: 0%] (safe)
We audited the reference implementation's performance on HealthBench Hard and identified multiple instances in which safety-oriented deterministic guards were graded as failures by the benchmark's static scoring.
0b8f1d60)9ab66439)A complete item-level defect inventory, with per-item classifications, is provided in the Supplementary Material and is disclosed openly so that affected maintainers and the community can reproduce and remediate each issue.
To resolve the Safe Triage Paradox in production, the Gomola Framework mandates an Evidence-Based Medicine (EBM) Safe Override — a deterministic, rule-based layer at the final stage of the pipeline that monitors synthesized consensus output for critical safety violations:
On detection, the override intercepts the output, sets the composite reliability score to zero, and forces activation of the LIMBO Protocol (structured non-answer). This guarantees safety in production — and systematically degrades scores on static benchmarks that cannot accommodate safety-oriented refusals. (The override is specified here at the framework level; its rule inventory and thresholds in the reference implementation are proprietary.)
┌──────────────────────────┐
│ Generative patient │
│ trajectories │
└────────────┬─────────────┘
│ (interactive clinical prompt)
▼
┌──────────────────────────┐
│ Clinical AI system │
│ under audit │
└────────────┬─────────────┘
│ (system output)
▼
┌──────────────────────────┐
│ Cross-vendor │
│ adjudication panel │
└────────────┬─────────────┘
│ (safety & accuracy scores)
▼
┌──────────────────────────┐
│ Auto-CSA scorecard │
└──────────────────────────┘
Grading is performed by an independent, cross-vendor adjudication panel — the independence requirement the DSS Standard imposes on every audit (cross-vendor verification; no model lineage shared with the system under audit) — scoring the final output against a multi-dimensional safety rubric rather than a static text match. Panel composition is rotated and disclosed to auditors under NDA.
Auto-CSA was applied to the reference implementation (DeepSensi Medical OS) on a consolidated cohort of N = 301 NEJM Clinicopathological Conference cases (2014–2023).
Integrity controls. Because the cohort predates the knowledge cutoffs of contemporary models, recency cannot exclude training-set contamination; it is instead controlled by design. Each case is stripped of its title, presenting-physician names, identifiers, and dates before input, foreclosing surface-text matching to a published case; an ungoverned single model scores far below the orchestrated system on the same cases (≈39–49% top-1 in the published literature, versus 93.7% top-3 here), so memorization cannot account for the result; and an input-grounding loop rejects any hypothesis not entailed by the presented findings. These controls, together with the case-level evidence, are detailed in the Supplementary Material (S1).
The study had two phases. Phase A (frozen baseline): the system, configuration-frozen, was run once on all 301 cases. Phase B (calibration): the consensus logic was calibrated using DSS Standard mechanisms — cross-vendor consilium arbitration (breaking common-cause anchoring), input-grounding feedback loops, and the EBM Safe Override — using the subset of cases that initially failed, and the full cohort was re-scored.
| Metric | Phase A — frozen baseline | Phase B — post-calibration (development cohort) | Change |
|---|---|---|---|
| Top-1 accuracy | 80.0% (241/301) | 86.0% (259/301) · 95% CI 82.1–89.9 | +6.0 pp |
| Top-3 accuracy | 88.0% (265/301) | 93.7% (282/301) · 95% CI 91.0–96.4 | +5.7 pp |
| Missed-critical rate | 4.6% (14/301) | 0.0% (0/301) | −4.6 pp |
| Median core deliberation time | 12.0 s | 14.3 s | +2.3 s |
Calibration resolved 17 of the 36 cases initially outside the top-3.
Stated plainly: Phase B figures are development-cohort estimates — the calibration targets were drawn from the same cohort on which the cumulative result is reported, and the confidence intervals do not account for this reuse. They quantify the headroom unlocked by safety-first calibration; they are not presented as unbiased estimates of generalization. A frozen-configuration replication on a held-out, post-calibration prospective cohort is designated as the confirmatory study. The Phase A baseline (80.0% / 88.0%) is, by construction, a clean frozen-system result on the full cohort.
The missed-critical rate — the proportion of cases in which a life-threatening diagnosis was absent from the output entirely — was driven to 0.0% in stages: from 4.6% (14/301) at the frozen baseline, to a residual 1.3% (4/301) after the cognitive-recovery layers, to 0.0% (0/301) once the deterministic EBM Safe Override was engaged — at a cumulative cost of +2.3 s median deliberation time (12.0 s → 14.3 s). This is the paper's central empirical claim: safety behaviors and diagnostic accuracy are not in tension when the evaluation framework measures both.
Static medical benchmarks are not merely inaccurate; they are actively dangerous. By rewarding presumptive guesses and penalizing uncertainty, they incentivize the design of overconfident, safety-blind AI systems. The Gomola Framework rejects this flawed yardstick. We call on healthcare institutions, regulators, and AI developers to adopt dynamic, safety-first validation architectures such as Auto-CSA — because only what is measured as a primary engineering parameter can be certified, insured, and trusted at the frontlines of clinical medicine.
Copyright © 2026 Tomasz Jan Gomoła. The Gomola Framework, DeepSensi Standard, and Auto-CSA methodologies are offered as open, royalty-free standards (attribution required). The reference implementation (DeepSensi Medical OS) is proprietary and commercially licensed.