Aside from the fact that that’s just a horrible thing to build:

It would only require some simple training for ChatGPT to receive a command such as “turn left” and understand how to translate that into a machine-readable language.

You can literally do this without AI. It’s just speech to text and mapping simple phrases to relevant functions. There’s no need to stick an unreliable AI into it.