Tracking down the locality assumption in a CHSH inequality derivation

CHSH inequality requires both locality and realism. I will equate here realism with counterfactual definiteness.

Now counterfactual definiteness tells us that given two different measurements on the same object, described by random variables $C_1$ and $C_2$, there exists a joint probability distribution for $C_1$ and $C_2$ (this is not always the case, search for the marginal problem and indeed we know that outcomes of measurements of noncommuting observables do not posses a joint probability distribution). Now if we can assume the existence of a joint probability distribution, then expectation values $E(C_1) + E(C_2)$ may be joined together to have $E(C_1 + C_2)$.

So suppose that we now have four random variables $A_1$ and $B_1$ local to Alice and $A_2$ and $B_2$ for Bob, which can take values $\pm 1$. The expression in the CHSH inequality is $$ |E(A_1 A_2) + E(A_1 B_2) + E(B_1 A_2) - E(B_1 B_2)| $$ Now if we can assume realism (counterfactual definiteness), there exists a joint probability distribution for the outcomes of all four random variables, and we can join the expectation values in the above together to get $$ \Bigl\lvert E\bigl(A_1 (A_2 + B_2) + B_1 (A_2 -B_2)\bigr)\Bigr\rvert $$ So now we do the standard trick where either $A_2$ and $B_2$ are equal or they are opposite, so that the first term is either $\pm 2$ and same for the second term.

Now to my question. I obviously used the realism assumption in the above. I assume locality means that the marginal distributions for $A_1$ and $B_1$ are independent of the choice of random variables Bob makes, $A_2$ or $B_2$ (but I otherwise allow for the correlation of outcomes, as long as it's independent of measurement choice, as there may be hidden variables that were encoded at the source of the state that produce correlations). Where did I use the locality assumption in this derivation? I would like if you either point out precisely where this assumption is needed in this calculation or argue that it is not needed with a convincing justification.

Edit: Given the answer below, it seems likely that the locality assumption may have been used at the point where we take $E(C_1) + E(C_2) = E(C_1 + C_2)$. However, it is still not understood why realism is not sufficient to reach this conclusion without the assumption of locality. As stated above, the question is why is the above reasoning not correct and locality assumption is needed, or alternatively I am also open to the idea that it might not be needed.

Edit 2: I have received a satisfactory answer to this question over at MathOverflow from @Steven Landsburg.

The existence of a joint probability distribution for all four variables is definitely sufficient to get the inequality. The existence of joint probability distributions for (A1,B1) and (A2,B2) (i.e. "realism") is definitely not sufficient to imply a joint probability distribution for all four. Locality is a sufficient additional assumption.

He also shows there that counterfactual definiteness for local outcomes is not a sufficient condition to obtain the CHSH inequality.