particle physics - Why is quark flavor just a SU(N) group?

In the standard model one has U(1) for electromagnetism, SU(2) for the weak sector and SU(3) for the color sector. One could say that in the quark part of the fermions, there are $$ \underbrace{6}_\text{flavors} \cdot \underbrace{3}_\text{colors} \cdot \underbrace{(2 + 1)}_\text{left and right}$$ “different” spinor particles. Those states can be grouped into color triplets and weak-isospin doublets and singlets depending on their chirality. For massive quarks, the weak-isospin symmetry is only approximate, of course. The Lagrangian stays invariant under the unitary transformations, therefore those are the gauge symmetries.

The weak-isospin doublets made up from up-down, charm-strange and top-bottom are just the same thing in different generations. Their only difference I think they have is the mass. Neglecting the CKM-matrix and the masses, I would think that there is no way to distinguish a down-quark from a strange-quark or bottom-quark. With masses, there is just this one difference.

One then generalizes the flavor symmetry to include the strange quark. The weak-isospin group is extended from SU(2) to just SU(3). This bugs me: The (approximate) SU(2) weak-isospin symmetry groups the (left-handed) up- and down-quarks together in a doublet. They are two states that differ by their electromagnetic charge. When adding the strange quark I would consider this to be another down-type quark (which is distinct from an up-type quark) from the second generation. Why does one put the strange quark onto equal footing with the other ones by including it with one $\mathrm{SU}(N_\mathrm F = 3)$ group? Shoudn't the strange-quark be treated similarly to the down-quark but different from the up-quark?

For any given force, it only sees part of it. So for the weak-isospin gauge symmetry, all the color states appear to be on equal footing. And the strong force does not care about the electric charge or the weak-isospin $z$-component. Is my problem in understanding that I distinguish up-type and down-type by quantum numbers that belong to other symmetry groups and the flavor would not see at all? Or is there something more to this?