Vacuum energy of a real Klein-Gordon field

Hamiltonian for a Klein-Gordon field can be written as - $$H= \int \frac{d^3p}{(2\pi)^3 \omega_{\vec{p}}}[a^{\dagger}_\vec{p}a_\vec{p}+\frac{1}{2}(2\pi)^3\delta^{(3)}(0).\tag{1}]$$ In one of my lecture notes on QFT, It is written that - In absence of gravity, we can neglect the second term in above equation which will lead us to- $$H= \int \frac{d^3p}{(2\pi)^3 \omega_{\vec{p}}}a^{\dagger}_\vec{p}a_\vec{p}.\tag{2}$$ It is obvious that second term in the first equation will make vacuum energy infinite. But Neglecting this term is the best we can do? Something is infinite and just for our convenience, we are putting it to zero. How is this logically and mathematically justifiable? Secondly, What is the role of gravity in neglecting this term? Why can we not neglect this term if gravity is present?