Diamonds are the deepest accessible "fragments" of Earth, providing records of deep geological processes. Absolute ages for diamond formation are crucial to place these records in the correct time context. Diamond ages are typically determined by dating inclusions, assuming that they were formed simultaneously with their hosts. One of the most widely used mineral inclusions for dating diamond is garnet, which is amenable to Sm-Nd geochronology and is common in lithospheric diamonds. By investigating worldwide garnet-bearing diamonds, we provide crystallographic evidence that garnet inclusions that were previously considered to be syngenetic may instead be protogenetic, i.e., they were formed before the host diamond, raising doubts about the real significance of many reported diamond "ages." Diffusion modeling at relevant pressures and temperatures, however, demonstrates that isotopic resetting would generally occur over geologically short time scales. Therefore, despite protogenicity, the majority of garnet-based ages should effectively correspond to the time of diamond formation. On the other hand, our results indicate that use of large garnet inclusions (e.g., > 100 μm) and diamond hosts formed at temperatures lower than ~1000 °C is not recommended for diamond age determinations.