How assessment drives learning in neurosurgical higher training

Certifying the competence of neurosurgeons is a process of critical importance to the people of Australia and New Zealand. This process of certification occurs largely through the summative assessment of trainees involved in higher neurosurgical training. Assessment methods in higher training in neurosurgery vary widely between nations. However, there are no data about the 'utility' (validity, reliability, educational impact) of any national (or bi-national) neurosurgical training system. The utility of this process in Australia and New Zealand is difficult to study directly because of the small number of trainees and examiners involved in the certifying assessments. This study is aimed at providing indirect evidence of utility by studying a greater number of trainees and examiners during a formative assessment conducted at a training seminar in Neurosurgery in April 2005. Aim: To evaluate an essay examination for neurosurgical trainees for its validity, reliability and educational impact. Methods: A short answer essay examination was undertaken by 59 trainees and corrected by up to nine examiners per part of question. The marking data were analysed. An evaluation questionnaire was answered by 48 trainees. Eight trainees who successfully passed the Fellowship examination who had also taken the short essay examination underwent a semi-structured interview. Results: The essay examination was found to be neither reliable (generalisability coefficient of 0.56 if the essay paper had comprised 6 questions) nor valid. Furthermore, evidence suggests that such an examination may encourage a pursuit of declarative knowledge at the expense of competence in performing neurosurgery. Conclusion: This analysis is not directly applicable to the Fellowship examination itself. However, this study does suggest that the effect of assessment instruments upon neurosurgical trainees' learning strategies should be carefully considered.