The velocity dispersions of stars near the Sun are known to increase with stellar age, but age can be difficult to determine, so a proxy like the abundance of α elements (e.g., Mg) with respect to iron, [α/Fe], is used. Here we report an unexpected behavior found in the velocity dispersion of a sample of giant stars from the Radial Velocity Experiment survey with high-quality chemical and kinematic information, in that it decreases strongly for stars with [Mg/Fe] > 0.4 dex (i.e., those that formed in the first gigayear of the Galaxy's life). These findings can be explained by perturbations from massive mergers in the early universe, which have affected the outer parts of the disk more strongly, and the subsequent radial migration of stars with cooler kinematics from the inner disk. Similar reversed trends in velocity dispersion are also found for different metallicity subpopulations. Our results suggest that the Milky Way disk merger history can be recovered by relating the observed chemo-kinematic relations to the properties of past merger events.