classical electrodynamics - Why isn't it obvious that a particle doesn't interact with its own field, classically?

The Wheeler-Feynman absorber theory or any other theory that tries to avoid the notion of field as an independent degree of freedom has always been concerned about infinite self energy of a charged particle. I don't feel confident that I appreciate the gravity of this problem.

According to the theory of fields every charged particle has a field of its own whose strength varies with distance from the particle and this field is what which the particle uses to exert force on other charged particles. So, if the field is supposed to be an inalienable part attached to the particle that it uses to interact with other particles why would the field be used to exert a force on the particle itself and as such why an infinite self energy at the points very close to it? Am I ignoring any implication of this theory that leads to a logical debacle? I hope with your elaborate answers I can see why this isn't something to be deemed obvious and can appreciate the seriousness of the Abraham force.