Because all a large language model does is it predicts what’s the most likely next word that you’re going to use

This is technically speaking doubly wrong. The default GPT output or loss function involves the categorical distribution of the likelihood (not the most likely) of the next token (not word). Whether you pick the most likely token is a matter of sampling strategy.

Also, a LLM after instruction tuning or with RLHF will behave beyond predicting the simple token probability.