AI Predicts the Text of Answers

There's a common argument about AI that says it doesn't understand anything about the world. That no matter how impressive it looks, it's "just predicting the next token of text." So basically, no matter how impressive AI appears to be, it's just using math to do next-token prediction. But that's like saying human authors are just writing down the words that pop into their heads while they write.