forces - How does centripetal acceleration produce the tangential velocity?

I just got started on my physics course, and already I am confused with this stuff called centripetal acceleration.

One thing I don't understand is what produces the velocity that is tangent to the circular path.

I searched on google, and got from PBS Education that there is a change in the velocity which produces the acceleration: But my textbook tells me that there is no change in the magnitude of the velocity only that its direction is changing.

On another website, there is a proof explaining that there is an inward acceleration because the change-in-velocity vector points inward which indicated that the acceleration vector should also be in the same direction.

If you wish to explain it in calculus, bear in mind that I don't know anything about it.

Answer

A small addition to @Hypnosifl's answer:

Key to understanding the circular motion is the fact that at any moment in time the acceleration (force keeping the horse / object in a circular path) is exactly perpendicular to the direction of motion. We have to do "almost calculus" to explain this... but maybe you will be able to follow:

Imagine we initially have a horizontal velocity, and the acceleration is vertical (downwards). How come the new velocity isn't faster than the old one?

Mental picture:

When the change in velocity due to acceleration $a$ for time $\Delta t$ is added to the initial velocity $v_{init}$, the new velocity has a magnitude

$$|v_{new}| = \sqrt{v_{init}^2 + (a\Delta t)^2}$$

When $a\Delta t$ is small, we can do a Taylor expansion and get

$$|v_{new}| = v_{init}\left(1 + \frac{(a\Delta t)^2}{2 v_{init}}\right)$$

Now if we look at a shorter and shorter interval $\Delta t$ we can see that the change in magnitude in velocity, which goes with the square of the interval, will become zero.

This means that if we keep the acceleration at right angles to the velocity (which is done in circular motion) then the speed will be constant - which is of course what we observe for circular motion of things like satellites, the moon, etc (barring small effects like tides, atmospheric drag etc - which are "imperfections" and not part of the essence of the orbital motion).