quantum field theory - Boosts are non-unitary!

Unlike rotations, the boost transformations are non-unitary. Therefore, the boost generators are not Hermitian. When boosts induce transformations in the Hilbert space, will those transformation be unitary? I think no. If that is the case, what is the physical significance of such non-unitary transformations corresponding to boosts in the Hilbert space?

Answer

On the actual Hilbert space of a consistent relativistic quantum mechanical system, the Lorentz transformations including boosts actually are unitary – which also means that the generators $J_{0i}$ are as Hermitian as the generators of rotations $J_{ij}$.

We say that the Hilbert space forms a unitary representation of the Lorentz group.

What the OP must be confused by is the fact that the ordinary vector representation composed of vectors $(t,x,y,z)$ is not a unitary representation of $SO(3,1)$. The $SO(3,1)$ transformations don't preserve any positively definite quadratic invariant constructed out of the coordinates $(t,x,y,z)$. After all, we know that an indefinite form, $t^2-x^2-y^2-z^2$, is conserved by the Lorentz transformations. So on a representation like the vector space of such $(t,x,y,z)$, the generators $J_{0i}$ would end up being anti-Hermitian rather than Hermitian.

But if you take a Lorentz-invariant theory with a positive definite Hilbert space, like QED, the formula for $J_{0i}$ makes it manifest that it is a Hermitian operator, which means that $\langle \psi |\psi\rangle$ is preserved by the Lorentz boosts! The complex probability amplitudes for different states $c_i$ behave differently than the coordinates $t,x,y,z$ above.

Note that the (non-trivial) unitary transformations of $SO(3,1)$ are inevitably infinite-dimensional. Finite-dimensional reps may be constructed out of the fundamental vector representation above and they are as non-unitary as the vector representation. But that's not true for infinite-dimensional reps. For example, the space of one-scalar-particle states in a QFT is a unitary representation of the Lorentz group. For each $p^\mu$ obeying $p^\mu p_\mu=m^2$, and there are infinitely (continuously) many values of such a vector (on the mass shell), the representation contains one basis vector (which are normalized to the Dirac delta function). The boosts just "permute them" along the mass shell which makes it obvious that the positively definite form is preserved when normalized properly.