Reliability, ease of use and usefulness of I-MeDeSA for evaluating drug-drug interaction alerts in an Australian context

Background: Recently, attention has shifted to improving the design of computerized alerts via the incorporation of human factors design principles. The Instrument for Evaluating Human Factors Principles in Medication-Related Decision Support Alerts (I-MeDeSA) is a tool developed in the United States to guide improvements to alert design and facilitate selection of electronic systems with superior design. In this study, we aimed to determine the reliability, ease of use and usefulness of I-MeDeSA for assessing drug-drug interaction (DDI) alerts in an Australian context. Methods: Using the I-MeDeSA, three reviewers independently evaluated DDI alert interfaces of seven electronic systems used in Australia. Inter-rater reliability was assessed and reviewers met to discuss difficulties in using I-MeDeSA and the tool's usefulness. Results: Inter-rater reliability was high (Krippendorff's alpha = 0.76), however, ambiguous wording and the inclusion of conditional items impacted ease of use. A number of items were not relevant to Australian implementations and as a result, most systems achieved an I-MeDeSA score of less than 50%. Conclusions: The I-MeDeSA proved to be reliable, but item wording and structure made application difficult. Future studies should investigate potential modifications to the I-MeDeSA to improve ease of use and increase applicability to a variety of system configurations.