The reaper rarely phones ahead: Mortality prediction tools good, not great (Review)

Doctors are generally lousy at predicting death in terminally ill patients, and in ICU patients with indeterminate outcomes. Mortality prediction models have proliferated to improve our performance, but in the critical care literature, have mostly shown high predictive accuracy only at the tail ends of probability (high probability of survival or death).

Siontis et al (led by the great epidemiologist John Ioannidis, author of the awesome-titled and influential paper, "Why Most Published Research Findings Are False") review all the literature published in a single year (2009) on validated clinical decision rules predicting mortality.

They reviewed 94 papers with 240 assessments of 118 mortality prediction tools. These included critical care metrics like APACHE, SAPS, and SOFA, as well as MELD, the Pneumonia Severity Index, and many others. Nine of the tools (including the aforementioned) were assessed 4 or more times that year, and the analysis centered on these.

APACHE was the best studied (in 19 studies that year). Its area-under-the-curve was 0.79, considered good (very good > 0.80, excellent > 0.90). Most of the other rules performed worse.

Most interesting to me: Reported accuracy of the tool was inversely related to the impact factor of the journal that published it. Authors politely suggest this tendency may be due to poor methodology in the strongly positive studies.

A significant limitation was that this was just one year of literature. Authors acknowledge a comprehensive review would take years and hundreds of full-time researchers.

Ioannidis does not shy away from clear, assertive conclusions, and this paper includes this gem:

Our systematic evaluation of a large number of seemingly well-validated predictive tools reported in the recent literature shows that these tools are not very accurate and that there is wide variation in their predictive accuracy for death. Most of the tools included in our analysis are not sufficiently accurate for wide use in clinical practice.

Siontis GCM et al. Predicting Death: An Empirical Evaluation of Predictive Tools for Mortality. Arch Intern Med 2011;171(19):1721-1726.