rotational dynamics - Ball Rolling Down An Inclined Plane - Where does the torque come from?

There is a ball rolling down an incline, with no slipping. If we consider the point of contact between the ball and the inclined plane to be the pivot point (for our torque calculations), then I have noticed that friction is no longer in play, and gravity (and normal force) is the reason for the torque. However, if we consider the center of mass of the ball to be the pivot point, friction is the only factor, since both the normal force and gravity effect the center of mass (so the net torque caused by them is 0, since the pivot is the center of mass). However, both of these produce different torques.. for different reasons. So my question is,

Is gravity the reason a ball rolls, or is friction? And why?

[Note: This is not homework, it is a genuine question I had while looking at some physics problems.]