quantum field theory - Trace of stress tensor vanishes $implies$ Weyl invariant

You often see in textbooks the statement that ${T^\mu}_\mu = 0$ implies Weyl invariance or conformal invariance. The proof goes like

$$\delta S \sim \int \sqrt{g} T^{\mu\nu} \delta g_{\mu\nu} \sim \int \sqrt{g} {T^\mu}_\mu, $$

where I have varied the action with respect to the metric and assumed $\delta g_{\mu\nu} \propto g_{\mu\nu}$ (i.e. a Weyl transformation).

This does not seem to be completely general because I can imagine a Lagrangian containing matter fields with non-trivial conformal weights. Then the full variation under Weyl tranformation contains a term proprotional to the matter equation of motion.

So I would conclude that the correct statement is more like

$${T^\mu}_\mu = 0, \quad \& \quad \frac{\delta S}{\delta \phi} = 0\implies \textrm{Weyl invariant}$$

Is it true that Weyl invariance only holds when the matter fields are on-shell or am I missing something?