LLMs are good at playing you
lemmy.ml
Large language models (LLMs) are eerily human-like: in casual conversations, they mimic humans with near-perfect fidelity. Their language capabilities hold promise for some fields — and spell trouble for others. But above all, the models’ apparent intellect makes us ponder the fate of humanity. I don’t know what the future holds, but I think it helps to understand how often the models simply mess with our heads.

TL;DR: LLMs are just mimicking natural language and conversation. Fact checking and healthy skepticism is not part of their model. For example they can be easily tricked into advocating conspiracy theories, like a fake moon landing. Google Bard is even stating arithmetic falsehoods like 5*6 != 30