While intraspecific variation in aposematic signals can be selected for by different predatory responses, their evolution is also contingent on other processes shaping genetic variation. We evaluate the relative contributions of selection, geographic isolation, and random genetic drift to the evolution of aposematic color polymorphism in the poison frog Adelphobates galactonotus, distributed throughout eastern Brazilian Amazonia. Dorsal coloration was measured for 111 individuals and genetic data were obtained from 220 individuals at two mitochondrial genes (mtDNA) and 7963 Single Nucleotide Polymorphisms (SNPs). Four color categories were described (brown, blue, yellow, orange) and our models of frog and bird visual systems indicated that each color was distinguishable for these taxa. Using outlier and correlative analyses we found no compelling genetic evidence for color being under divergent selection. A time-calibrated mtDNA tree suggests that the present distribution of dorsal coloration resulted from processes occurring during the Pleistocene. Separate phylogenies based on SNPs and mtDNA resolved the same well supported clades, each containing different colored populations. Ancestral character state analysis provided some evidence for evolutionary transitions in color type. Genetic structure was more strongly associated with geographic features, than color category, suggesting that the distribution of color is explained by localized processes. Evidence for geographic isolation together with estimates of low effective population size implicates drift as playing a key role in color diversification. Our results highlight the relevance of considering the neutral processes involved with the evolution of traits with important fitness consequences.