The empirical variance estimator for computer aided diagnosis: lessons for algorithm validation

Alex F. Mendelson, Maria A. Zuluaga, Lennart Thurfjell, Brian F. Hutton, Sébastien Ourselin

Research output: Contribution to journalConference paperpeer-review

1 Citation (Scopus)


Computer aided diagnosis is an established field in medical image analysis; a great deal of effort goes into the development and refinement of pipelines to achieve greater performance. This improvement is dependent on reliable comparison, which is intimately related to variance estimation. For supervised methods, this can be confounded by statistical issues at the comparatively small sample sizes typical of the field. Given the importance of reliable comparison to pipeline development, this issue has received relatively little attention. As a solution, we advocate an empirical variance estimator based on validation within disjoint subsets of the available data. Using Alzheimer's disease classification in the ADNI dataset as an examplar, we investigate the behaviour of different variance estimators in a series of resampling experiments. We show that the proposed estimator is unbiased, and that it exceeds the estimates of naive approaches, which are biased down. Because the estimator avoids independence assumptions, it is able to accommodate arbitrary validation strategies and performance metrics. As it is unbiased, it is able to provide statistically convincing comparison and confidence intervals for algorithm performance. Finally, we show how the estimator can be used to compare different validation strategies, and make some recommendations about which should be used.

Original languageEnglish
Pages (from-to)236-243
Number of pages8
JournalLecture Notes in Computer Science
Publication statusPublished - 2014
Externally publishedYes
Event17th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2014 - Boston, MA, United States
Duration: 14 Sep 201418 Sep 2014


Dive into the research topics of 'The empirical variance estimator for computer aided diagnosis: lessons for algorithm validation'. Together they form a unique fingerprint.

Cite this