The potential for experimenter bias effects in word recognition experiments

Word recognition experiments that involve comparisons between two different matched sets of words allow for possible experimenter bias if there are many equally well-matched pairs to choose from. The possible extent of this bias depends on the experimenter's intuitive knowledge of the likely difficulty of individual items. This is assessed by asking a number of experimenters to decide which of two frequency-matched words would produce the fastest reaction time in a lexical decision experiment, and then comparing their predictions with actual data. All experimenters demonstrated substantial above-chance accuracy, which was unrelated to the amount of experience they had in the field. It was concluded that the experimenters could potentially produce spurious effect sizes ranging from 16 to 38 msec.