Research

Background. Published sepsis AI models routinely report AUC > 0.90 on ICU benchmarks. We pre-registered a falsification protocol to test whether these benchmarks measure biological sepsis or administrative artifacts. Methods. 286,510 ICU encounters from MIMIC-IV and eICU-CRD. Labels were regenerated using three independent sepsis definitions (Sepsis-3 clinical criteria, biomarker thresholds, and billing-code derived). Model performance was compared across label regimes. Results. Models trained on billing-code labels retain AUC > 0.85 even when key biological features are removed, while models trained on Sepsis-3 clinical criteria degrade to AUC 0.68 under the same ablation. The gap replicates on eICU. Performance on the billing benchmark is largely explained by care-process features observed after sepsis onset. Conclusion. Benchmark performance reflects hospital coding practice as well as biology, and the pooled clinical claim did not survive falsification. Model evaluation should separate administrative from biological signal before clinical claims are made.

Background. Public ICU datasets are dominated by academic medical centers, but the majority of US ICU admissions happen in community hospitals. Methods. 136,864 encounters from eICU-CRD stratified by hospital type. We trained sepsis models on AMC-only cohorts and evaluated on community cohorts, and vice versa. Results. Cross-institution-type transfer loses 11–19 AUC points. The gap persists after standard fine-tuning and is partially attributable to differences in nursing documentation cadence, lab-test ordering patterns, and antibiotic-administration timing, all of which are inputs to common sepsis models. Conclusion. Institution-type is a first-class covariate in ICU AI evaluation. FDA submissions relying on AMC-trained models should include community-hospital validation cohorts as a standard practice.

Background. Clinical risk stratification tools rely on published ICU mortality estimates that are often decades old, cohort-specific, or derived from meta-analyses with limited subgroup resolution. Methods. Pre-registered observational study, 201,905 ICU encounters from MIMIC-IV and eICU-CRD. Six primary strata: Diabetes, MI, AF, Seizure, PE × age > 70; COPD × age < 50. Observed mortality computed and compared to consensus-published estimates. Results. Five of six strata show divergences exceeding 60%, cross-validated on the independent dataset. The seizure × elderly stratum shows a +510% divergence, the largest literature gap in the series. COPD × young adults shows the inverse (−63.6%, suggesting published estimates are inflated by selection bias). Conclusion. Bedside risk stratification using literature-derived mortality estimates is systematically miscalibrated for elderly ICU patients. Clinical AI trained on or calibrated to these estimates inherits the same miscalibration.