Designing usable machine learning decision support for clinical deterioration
Early warning scores are widely implemented across acute hospital settings to detect patients who are clinically deteriorating. These decision support tools alert clinicians to physiological changes, yet they routinely fall short when it comes to identifying the root cause of the decline. Recognising that a patient is unstable is only the first step; clinicians must quickly determine whether the driver is sepsis, cardiac failure, respiratory compromise, or another etiology to start targeted management. Machine learning models offer the potential to analyse multi-parameter patient data and suggest likely etiologies along with relevant treatment options. However, presenting complex predictive outputs to frontline medical staff remains a substantial hurdle. If an interface is difficult to interpret or fails to explain its recommendations, it can create distraction rather than clinical utility.
Designing and testing the decision support interface
To address this gap, the researchers conducted a sequential exploratory mixed-methods study focused on eliciting clinician requirements and measuring interface usability. The first phase involved qualitative focus groups with 26 clinicians specialising in critical care or hospital medicine. The goal of these sessions was to establish specific design requirements for a machine learning clinical decision support interface tailored to patient deterioration.
Following focus group feedback, the study team refined a prototype clinical decision support tool. They then proceeded to a quantitative usability testing phase involving 23 critical care and hospital medicine clinicians in a controlled laboratory setting. The testing compared an experimental electronic health record interface featuring the machine learning clinical decision support tool against a control interface without the tool. Each clinician reviewed five scripted patient cases per interface, completing a washout period between testing conditions to minimise order effects. The researchers measured key usability dimensions, specifically effectiveness, efficiency, and user satisfaction, analysing differences between the control and experimental conditions using Wilcoxon signed-rank tests.
User requirements and laboratory usability results
The focus groups yielded five specific design requirements from participating clinicians. First, clinicians required visibility into the rationale driving the model's recommendations. Second, they needed ways to assess how potential data quality issues within the electronic health record might impact the accuracy of the model's suggestions. Third, they asked for relevant patient data to be consolidated on a single display screen to minimise screen switching. Fourth, they requested graphic visualisations of clinical information. Fifth, they emphasised that the decision support tool should be integrated directly into pre-existing clinical care pathways.
In the usability evaluation with 23 clinicians, the experimental interface incorporating these design features outperformed the control interface across multiple metrics. Statistical testing showed that the experimental tool was associated with statistically significant improvements in diagnostic accuracy, lower perceived clinician workload, and higher user satisfaction (all p < 0.05). These gains were demonstrated in the controlled laboratory environment across the scripted deterioration scenarios.
Implications for informatics design and clinical workflow
These findings highlight the practical value of user-centred design when introducing machine learning tools into clinical workflows. Clinicians do not simply want automated alerts; they require contextual information, including data quality indicators and algorithmic rationale, to make safe decisions. Placing graphic data and recommendations onto a single screen directly targets the information overload clinicians experience when assessing unstable patients. Furthermore, aligning predictive outputs with pre-existing care pathways ensures that suggested treatments fit established institutional practices. For health informatics teams and educators, these results demonstrate that building clinical trust requires transparency about model logic and data integrity rather than relying on predictive performance alone.
Limitations of simulated usability testing
Several limitations must be kept in mind when interpreting these results. First, the evaluation was conducted in a controlled laboratory environment using five scripted patient cases per interface, which cannot duplicate the noisy, high-interrupt environment of live clinical practice. Second, the usability phase relied on a small sample size of 23 clinicians, limiting the generalisability of the findings across broader clinician groups or different hospital environments. Third, while the laboratory trial demonstrated improvements in diagnostic accuracy, perceived workload, and satisfaction, the study did not measure clinical outcomes or patient safety metrics in real care settings. Further research is necessary to determine if these usability benefits persist when the tool is integrated into live electronic health records during actual emergency care.
Reference
Nguyen OT, Wiegmann DA, Haddad J, Doerr SA, Ahmad A, Oguss M, et al.. Development and usability testing of a machine learning clinical decision support tool for suggesting etiologies and treatment of clinical deterioration. Journal of critical care. 2026. doi:10.1016/j.jcrc.2026.155748. PMID: 42759259.
This research digest summarises a recently published article. It was drafted with AI assistance from the article's abstract and reviewed by the author before publication.
Comments