Is Newtonian gravity consistent with an infinite universe?

Let us assume that we have have an infinite Newtonian space-time and the universe is uniformly filled with matter of constant density (no fluctuations whatsoever), all of it at rest. By symmetry, the stuff in this universe should not collapse or change position in any way if gravity is the only force acting on it. Now, consider Gauss theorem. It says that within a spherical system (it says more that that but this will suffice), the gravitational force felt by any point will be the same as if all matter between the center and the point were concentrated at the center. The matter outside the sphere does not contribute any force). Thus, in such a system, the matter (stuff, I do not say gas so we can consider it continuous) will collapse towards the center of the sphere. We can apply this argument to any arbitrary point in our previous infinite homogeneous universe, and conclude that matter will collapse towards that point (plus the point is arbitrary). So, why is Gauss's theorem is not valid in this case?

I was signaled that this question is a duplicate and has been answered, however: The best I could get from the redirected question is this quote: "However, the mass can't be negative and the energy density is positive. This would force a violation of the translational symmetry in a uniform Newtonian Universe". It still doesn't give a satisfactory answer. For instance: how is that symmetry broken if we assume that there is no noise nor small density fluctuations in the system? How can you choose then the absolute "origin" that will break the symmetry? Still doesn't make sense to me.