optics - How does the Huygens–Fresnel principle apply to diffraction?

I'm currently trying to understand the mechanics behind diffraction and gain a better understanding of the Huygens–Fresnel principle, but I've reached a hurdle that is making me question my understanding of refraction.

Firstly, a quick explanation of my understanding of refraction (where the Huygens-Fresnel principle makes sense):

As I understand, refraction is a phenomena caused by light-matter interaction; when a light wave is incident on a material, its electric field component excites the particles that make up the material, which leads to emission of new EM waves. The Huygens–Fresnel principle makes sense to me, here, as the particles become radiators, and they radiate new EM waves in all directions.

So with this understanding, the bending of refracted light makes sense: when an incident wave is at an angle that isn't 0 degrees, the incident wave will interact with the particles at the surface, in a sequential manner. This is what causes the refracted light to bend from the normal, as the excited particles behave like a phased array, where they emit new EM waves sequentially.

So with refraction, I believe it is the particles (that make up the material) that are the sources of new waves described by the Huygens–Fresnel principle (the yellow dots in the image below). I suspect this may be where I'm going wrong(?)

Note that the above understanding is for refraction only. I am aware that the overall transmitted light is made up of the refracted light and (some of) the incident light.

Now for diffraction:

With diffraction (using the single split experiment for simplicity), the wave is partially absorbed and partially transmitted. The form of the transmitted wave is not the same as the original, due to interference caused by the new wavelets, emitted at each point of the wavefront.

The transmitted part does not interact with any particles (I'm ignoring atmospheric particles), so from a physical perspective, what are these emitters (at each point of the wavefront/the yellow dots in the image above)? And what causes them to emit new wavelets?

For refraction, the emitters are the particles that make up the material, and they emit new waves due to their interaction with the incident wave. So I understand how the Huygens-Fresnel principle works with refraction, I'm just not sure why I should consider every point of a wavefront to be a source of new EM wavelets with diffraction.

---

So as you can probably tell, my understanding of refraction is causing confusion with diffraction, which, in turn, is making me question my understanding of refraction.

Is my understanding of refraction correct? Are the points referred to by the Huygens–Fresnel principle actually the particles that make up the material for refraction? What's the deal with diffraction?