While there is currently a considerable interest in vacuum-modifications of material properties, such as chemical reactivity, conductivity or phase transitions, theoretical predictions of such effects usually rely on effective single- or few-mode models. These models ignore the coupling to the infinite number of electromagnetic modes that the vacuum is actually comprised of and are therefore fundamentally incapable of making reliable predictions about the magnitude or even the sign of vacuum energy shifts. In this talk I will present a first-principle derivation of the ground state energy shift of a single molecular dipole in ultrastrong coupling cavity QED. This analysis provides a clear distinction between purely electrostatic and genuine vacuum effects and allows us to study both contributions as a function of the relevant system parameters. Finally, I will address the implications of these findings for experimental and theoretical research in this field.