newtonian mechanics - Mass distributions within two bodies such that the orbit is no longer Keplerian?

The trajectories of two point masses or spherically symmetric masses with respect to their center of mass are conic sections or Kepler orbits.

Consider that the bodies have finite size with respect to their separation and not necessarily uniform, or even spherically symmetric mass distributions.

In that case what are the constraints on their mass distributions and orientations such that their orbits are still Keplerian? Or does any deviation from spherical symmetry of one or both body immediately result in a non-Keplerian orbit?