Problem is that contrast is not a thing for the “middle” levels.
So, for a white or light gray background, you can have black as contrast color. And for a dark gray or black background, you can have white as contrast color.
But for middle gray, both white and black don’t really provide enough contrast.

A slider to at least choose between white, light gray, dark gray and black would still be cool, though.