Researching the reliability of accreditation survey teams: lessons learnt when things went awry

Accreditation of health organisations, occurring in over 70 countries, is predicated upon the reliability of survey teams' judgements, but we do not know the extent to which survey teams are reliable. To contribute evidence to this issue, we investigated the reliability of two survey teams simultaneously assessing an organisation. The setting was a large Australian teaching hospital, and data were derived from interviews, observations and survey documents. Participants were from four groups: hospital staff, accreditation agency personnel and surveyors, and research staff. Thematic analysis was employed to identify significant factors that influenced the study. The two survey teams' ratings and recommendations demonstrated high levels of agreement. However, while a common understanding of the study existed, the research was compromised. There were difficulties enacting the study. Contrary to negotiated arrangements, the pressure of the study resulted in surveyors discussing evidence and their interpretation of standards. Uncontrollable circumstances (late changes of personnel), and unexpected events (a breakdown of working relationships), challenged the study. The twin lessons learnt are that a consistent survey outcome is likely to be reached when reliability of process and consistent application of standards are pursued, and research requires negotiating challenges and relationships.