nuclear physics - About mass defect

Here's how my book explains mass defect:

Particles inside the nucleus interact with each other - they feel attraction. The potential energy $U$ of such attraction is negative, because in absence of these forces we consider the potential energy to be zero. So we can write the total energy as: $$E=E_{rest}+U$$ Dividing $E$ by $c^2$ we obtain the mass, and because $U<0$ the mass of the nucleus is less than the sum of individual nucleons.

Now, I have problem with the $U$ term. We know that we can choose the zero level for PE arbitrarily. Thus, $E$ can't be defined well (up to constant). However, real measurements "obey" the standard convention of zero PE at infinity. So how can I solve the contradiction? (Obviously, I'm wrong, but I fail to understand why).

This question leads me to a more general question regarding the $E=mc^2$ relation. It follows that $m$ has no certain value when we're dealing with potential energies. Only the change in mass matters, because only the change in potential energy has physical meaning (and can be defined precisely). But mass is a quantity which we measure everyday very precisely, and there's no ambiguity in its value, despite the fact that the systems we measure include quite often some potential energy.