How “diagnostic” criteria interact to shape synesthetic behavior: the role of self-report and test–retest consistency in synesthesia research

In the past few decades, researchers have established synesthesia as a genuine phenomenon, identified its characteristics (in particular, its automatic, specific and consistent nature), and developed “gold standard” inclusion criteria for research: synesthetes are participants that self-report synesthetic experiences and have consistent (beyond a “cutoff” score) inducer-to-concurrent pairings. While this approach has significantly advanced scientific progress, it can confuse interpretation of research findings due to its inherent circularity: consistency will always appear to be a defining characteristic of synesthesia so long as it is also an inclusion criterion for synesthesia studies. Here, we aim to clarify the relationship between self-report and consistency in “diagnosing”¹ synesthesia. In four experiments, we find that: (1) the optimal consistency cutoff score differs across languages; (2) self-reported synesthetes that “fail” consistency tests can still behave like synesthetes – to our knowledge the first objective evidence that “inconsistent synesthesia” is a genuine phenomenon; (3) Using self-report as the sole inclusion criterion does not significantly change the effect size of two measures of synesthetic behavior (the synesthetic Stroop and synesthetic color Palette); and (4) Consistency influences Stroop effect size in self-reported synesthetes only, but influences the Palette in both synesthetes and non-synesthete controls. We conclude that (in certain cases) self-report alone is a sufficient diagnostic criterion for synesthesia, and that synesthesia studies can increase explanatory power by using raw consistency scores as a covariate in analyses, rather than as an inclusion criterion.