The cooling of solids by optical means only using anti-Stokes emission has a long history of research and achievements. Such cooling methods have many advantages ranging from no moving parts or fluids through to operation in vacuum and may have applications to cryosurgery. However, achieving large optical cryocooling powers has been difficult to manage except in certain rare-earth crystals but these are mostly toxic and not biocompatible. Through study of the emission and absorption cross sections we find that diamond, containing either nitrogen vacancy (NV) or silicon vacancy defects, shows potential for optical cryocooling and, in particular, NV doping shows promise for optical refrigeration. We study the optical cooling of doped diamond microcrystals ranging 10-250 μm in diameter trapped either in vacuum or in water. For the vacuum case we find NV-doped microdiamond optical cooling below room temperature could exceed |ΔT|>10 K for irradiation powers of Pin<100 mW. We predict that such temperature changes should be easily observed via large alterations in the diffusion constant for optically cryocooled microdiamonds trapped in water in an optical tweezer or via spectroscopic signatures such as the zero-phonon line width or Raman line.