Dialect classification is difficult for naive listeners, but perceptual learning tasks using sentence-length utterances have been shown to produce modest improvements in performance. The goal of the current study was to explore perceptual learning by naive listeners in a speeded dialect classification task with shorter, word-length utterances. In a series of experiments, participants were trained in a two-alternative forced-choice speeded dialect classification task (Cleveland vs. Cincinnati) with feedback and were then tested in the same task with novel talkers and novel words without feedback to assess learning. Variability in the stimulus materials in the training phase, including the number of talkers from each dialect and the number of different tokens produced by each talker, was manipulated across experiments to determine how variation in the input affected perceptual learning of dialect categories. The results revealed that training materials consisting of utterances produced by multiple different talkers from each dialect with multiple different tokens produced by each talker led to a significant improvement in dialect classification performance compared to a baseline condition without training. These findings suggest that dialect classification performance can improve with training on short, word-length utterances, but that robust dialect category learning requires high variability stimulus materials.