Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete. It’s a fine zeroth-order approximation, and it is grounded in something real: transformer-based language models emit tokens autoregressively:

while not done:
    tokens.append(model.sample_next_token(tokens))

This certainly has the shape of something you might call a next-token predictor. During pre-training, the model repeatedly takes some prior tokens, looks at the token that actually followed them, and makes that token more likely to be sampled next. Conceptually, the training loop looks something like this:

for tokens in training_data:
    for position in range(1, len(tokens)):
        prior_tokens = tokens[:position]
        actual_next_token = tokens[position]
        model.make_more_likely(
            actual_next_token,
            after=prior_tokens,
        )

make_more_likely is, of course, doing a heroic amount of work here. Under the hood are loss functions, gradients, and parameter updates, but for this post we only care about their combined effect: the token that actually came next becomes more likely.

Crucially, every actual_next_token comes from an existing sequence in training_data. It’s probably fair to say that the base model also behaves as a next-token predictor: it is trained to predict next tokens as they occur in its training data.

But the LLMs we use are not just base models. They are post-trained, and a key part of modern post-training is reinforcement learning with verifiable rewards (RLVR). During pre-training, the model learns only from sequences that already exist in the training data. During RLVR, the model explores by generating new sequences and learning from their outcomes. Conceptually, the RLVR training loop looks something like this:

for task in training_tasks:
    for explored_tokens in model.explore(task):
        reward = evaluate_outcome(task, explored_tokens)
        for position in range(len(explored_tokens)):
            prior_tokens = task + explored_tokens[:position]
            explored_next_token = explored_tokens[position]
            model.make_more_likely(
                explored_next_token,
                after=prior_tokens,
                according_to=reward,
            )

make_more_likely is doing the same kind of work in both loops, but for a fundamentally different reason. During pre-training, it makes an actual_next_token more likely because that token appeared in the training data. During RLVR, it makes an explored_next_token more likely because the explored sequence containing it earned a high reward.

So while a post-trained LLM still has the shape of a next-token predictor, emitting tokens one at a time, it no longer learns only by predicting existing text. It also learns from new sequences produced through its own exploration.

Chess Analogy

A chess engine makes this distinction easier to see. Imagine two chess systems.

The first is trained on a large database of grandmaster games. It learns patterns in how grandmasters respond to different board positions. Given a new position, it predicts the move a grandmaster would most likely play next. That is a next-move predictor.

The second is an idealized chess engine that has explored every possible game. From that exhaustive exploration, it knows the probability of winning from every possible board position. Given a position, it chooses the move that leads to the highest probability of winning. Unlike the first system, it is not trained only on games that grandmasters already played. It also learns from games generated by its own exploration.

Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.

Closing Thoughts

I didn’t touch on other post-training techniques in this post, but they matter too. Reinforcement learning from human feedback (RLHF), for example, shifts the model away from imitating its pre-training data as a whole and toward simulating a helpful assistant. And as we saw in this post, RLVR goes further still: it allows an LLM to explore and learn from ideas never seen in its training data.

That is why “next-token predictor” is the wrong mental model. It describes the shape of the mechanism, one token emitted after another, while ignoring what that mechanism encodes. A simulation of a helpful assistant and knowledge discovered through exploration can both be encoded in exactly the same next-token loop.