newtonian mechanics - Contradiction on gravitational potential energy

I was reading the derivation of the gravitational energy of a point mass and I seem to have found a contradiction.

The derivation in my textbook is given as follows:-

Let there me a large fixed mass 'M' and a small mass 'm' placed at a distance of $r_1$ from mass M.

Now let there be some external force $F_{ext}$ that displaces mass m from $r_1$ to $r_2$.

Now according to the work energy theorem, we will have the following equation:- $$K_2 - K_1 = W_g + W_{ext}$$ If we make sure that the kinetic energy of the system does not increase:- $$0 = W_g + W_{ext}$$ $$W_g = -W_{ext}$$ The change in gravitational potential energy of the system is equal to the negative of the work done by the gravitational force. $$ U(r_2) - U(r_1) = -W_g$$ Now let $r_1 = r$ and $r_2 = r + dr$ where Dr is an infinitesimally small distance. $$ U(r_2) - U(r_1) = -\int F_g.dr$$ Since the gravitational force and the displacement are in opposite directions:- $$ U(r_2) - U(r_1) = -\int F_gdrcos(\pi)$$ $$ U(r_2) - U(r_1) = \int F_gdr$$ $$ U(r_2) - U(r_1) = \int \frac{GMm}{r^2}dr$$ $$ U(r_2) - U(r_1) = -\frac{GMm}{r}$$ Now let $r_1$ = $\infty$ and $r_2 = r$ $$ U(r) - U(\infty) = GMm(\frac{1}{\infty}-\frac{1}{r})$$ $$ U(r) - U(\infty) = GMm(0-\frac{1}{r})$$ Now we will assume that the potential at infinity is zero. $$ U(r) = -\frac{GMm}{r}$$

My query:-

But if we let $r_1 = \infty$ and $r_2$ = r , $r_1$ will be greater than $r_2$(since $\infty>r$ ). Our original assumption was that the mass m was moving away from the mass M and that's why the dot product of gravitational force and displacement was negative. But now we are assuming it is moving towards mass M (from infinity to a separation of r). This looks like a contradiction to me.