What in the joke makes him racist but not misogynistic?

There’s nothing in the joke saying she’s either the only black person or the only woman in the bar. In fact you could say that there’s more pointing to him being misogynistic in that he says “this woman” and not “this black person”.

To me it reads as both