electromagnetism - What symmetry gives you charge conservation?

This is a popular question on this site but I haven't found the answer I'm looking for in other questions. It is often stated that charge conservation in electromagnetism is a consequence of local gauge invariance, or perhaps it is due to some global phase symmetry. Without talking about scalar or spinor fields, the EM Lagrangian that we're familiar with is: $$ \mathcal{L} = -\frac{1}{4}F^2 - A \cdot J + \mathcal{L}_\mathrm{matter}(J) $$ The equation of motion for $A$ is simply $$ \partial_\mu F^{\mu \nu} = J^\nu $$ From which it follows that $J$ is a conserved current (by the antisymmetry of the field strength). But what symmetry gave rise to this? I'm not supposing that my matter has any global symmetry here, that I might be able to gauge. Then so far as I can tell, the Lagrangian given isn't gauge invariant. The first term is, indeed, but the second term only becomes gauge invariant on-shell (since I can do some integrating by parts to move a derivative onto $J$). If we demand that our Lagrangian is gauge-invariant even off-shell, then we can deduce that $\partial \cdot J = 0$ off-shell and hence generally. But we can't demand that this hold off-shell, since $J$ is not in general divergenceless!

For concreteness, suppose that $$ \mathcal{L}_\mathrm{matter}(J) = \frac{1}{2} (\partial_\mu \phi) (\partial^\mu \phi) \qquad J^\mu \equiv \partial^\mu \phi $$ Then we find that $\phi$ (a real scalar) satisfies some wave equation, sourced by $A$. The equations of motion here constrain the form of $J$, but off-shell $J$ is just some arbitrary function, since $\phi$ is just some arbitrary function. Then it is clear that the Lagrangian is not gauge-invariant off-shell.

And this is a problem, because when we derive conserved quantities through Noether's theorem, it's important that our symmetry is a symmetry of the Lagrangian for any field configuration. If it's only a symmetry for on-shell configurations, then the variation of the action vanishes trivially and we can't make any claims about conserved quantities.

So here's my question: what symmetry does the above Lagrangian have that implies the conservation of the quantity $J$, provided $A$ satisfies its equation of motion? Thank you.