It is commonly claimed that only Hamiltonians with a spectrum unbounded both above and below can give purely exponential decay. Because such Hamiltonians have no ground state, they are considered unphysical. Here we show that Hamiltonians that are bounded below can give purely exponential decay. This is possible when, instead of looking at the global survival probability, one considers a subsystem only: We show that the reduced state of the subsystem can exhibit a Markovian dynamics, and some (local) observables can decay exponentially. We conclude that purely exponential decay might not be as unphysical as previously thought.