At Pantograph, we're working on training fully general robotics models that can act autonomously for hours at a time. Especially in robotics, it's difficult to get diverse data at scale. Learning to act from internet video data could allow models to scale with compute, rather than being limited by small action datasets.

In this work, we develop a simple method for learning goal-directed behavior through pretraining on internet-scale video.

Usually, goal-directedness is taught in a post-training phase, which limits the extent to which it can generalize. Here, we learn goal-directedness during pretraining on internet-scale video, which greatly improves the models' ability to achieve complex goals.

Video games are a useful testing ground for robotics. We're starting with Minecraft because it's open-ended, and supports commanding diverse, long-horizon goals.¹1This model can only act within the one game, but we will be training larger models across a broader distribution of visual environments in the coming months.↩ Our largest model, which we call Pan, is a 4B parameter model that can fight mobs, explore to find specific objects, complete challenging platforming environments, and build structures as commanded. We find it is able to generalize to quite diverse environments that it has never seen before.

Reinforcement learning is interesting in part because it can produce models that are superhuman. If we can do reinforcement learning at scale on video, we may be able to produce models that are superhuman in many more areas all at once.

#Reinforcement learning from observation-only data

We view internet-scale videos as reinforcement learning trajectories that include only observations.²2This lens makes sense for some modalities and not others. Most text data, for example, can't be viewed as a written transcript of an agent interacting with an environment. It could likely work for audio, however.↩ It's difficult to do reinforcement learning from this kind of data because it lacks rewards and actions, but it turns out there are simple solutions for both.

Goal-conditioning is a simple way of side-stepping the need to specify a reward function. The idea is to use what happens later in a video as the goal for the earlier part, a trick sometimes called hindsight relabeling,³3Marcin Andrychowicz et al., Hindsight Experience Replay, NeurIPS 2017.↩ because it takes what may have been a failure at one goal, and relabels it as a success for whatever actually happens. In principle, you could use any description of what happens later in a video as the goal, but in this work we focus on using frames of video as goals.

Although videos don't contain actions, it's still possible to learn functions from video that only depend on states. This includes value functions,⁴4Typically, value functions are tied to a specific policy. In this very general setting, where any number of policies may be acting in any kind of environment, it makes more sense to use in-context learning to infer the tendencies and level of skill of the policies in a specific video. This way a single model can be trained that can act as a generic value function, capable of evaluating any policy's behavior.↩ which in goal-conditioned RL represent roughly the probability that a goal frame is achieved in the agent's future, as well as the next frame distribution for a goal-conditioned policy.⁵5Both of these can be modeled in a number of ways, including likelihood-based models, energy-based models, and contrastive learning. We're excited about future work that could study the trade-offs between these different approaches.↩

After this action-agnostic pretraining, a smaller dataset containing actions can be used to produce an agent capable of acting in the environment.⁶6Given a strong value function, learning to act is a much easier problem (Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018). A value function effectively converts the very difficult multi-step decision making problem of reinforcement learning into a single-step decision problem, which is generally much more tractable.↩

We think that this lens of viewing video as RL trajectories is fruitful, and there are many ideas yet to be explored. If this is the kind of work you're interested in, join us!

#Training

We pretrained a family of goal-conditioned models from scratch on about 500k hours of diverse videos of Minecraft gameplay. Afterwards, we post-trained them on about 2k hours of contractor trajectories including both videos and action sequences. The pretraining is action-agnostic, and the action space only comes in during post-training. (See Appendix for more details.)

#Evaluation

We created a suite of 104 evaluation environments. Each environment consists of an initial Minecraft save world and a goal save world. We load the goal save world once per rollout and render a single goal image from it as the model's prompt. We use a mix of programmatic and language model-based graders to assess model performance, depending on the environment.

#Comparisons

We compared to STEVE-1, an open source goal-conditioned policy for Minecraft. STEVE-1's goal conditioning comes from post-training on a contractor dataset with action labels, on top of VPT, a model pretrained by OpenAI using an inverse dynamics model to label internet videos with estimated actions.⁸8Shalev Lifshitz et al., STEVE-1: A Generative Model for Text-to-Behavior in Minecraft, 2023; and Bowen Baker et al., Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos, 2022.↩

We also trained a VLA baseline on the same post-training dataset of contractor trajectories that we used for Pan. This VLA model was initialized from Gemma 4, one of the strongest open-source language models with support for video inputs.⁷7Google DeepMind, Gemma 4 model card, 2026.↩ In total, much more compute went into Gemma 4 than into the models we trained, though our models have likely seen more Minecraft data through their pretraining.

#Results

The models trained with goal-conditioned pretraining are better able to accomplish their goals in our evaluation worlds, including for varied and off distribution goals. We find that model size matters most for complicated, semantic things like mechanisms, and dextrous tasks like building. Our models learn some mechanisms (buckets, flint and steel, crops, fishing) and not others (levers, ender pearls, animal taming). We are excited to see what capabilities emerge from further scale-ups.

#Basics

We begin simply: if we command a goal a little bit in front of the model, does it care about the goal frame at all? Will it stay there? We find that the Pan models walk to the target position and oscillate around it for the remainder of the rollout, seeming to try to line up their view with the goal frame perfectly. We call this a "goal dance", and it is a useful indicator that the model thinks it has achieved the goal. STEVE and the VLA are less consistent on these simple tasks, tending to wander off rather than stay on the goal, though they do accomplish them sometimes.

In Minecraft, building a tower underneath you is almost as basic as walking forward, and all of the models are able to do it. Our models again demonstrate more faithful goal following: whereas STEVE and the VLA tower up way beyond the commanded image, the Pan models frequently stop at the commanded height.

What about simple but unusual goals, like looking at a target made out of wool, or swimming through a pool towards a gold panel? This is easier to do, but less common in the training distribution. Here, the difference between model families is larger: Pan models are much more goal directed.

#Building

We test building ability in various settings, varying how much of the final structure we show in the goal frame, what materials are used, and how much is built at the start of the eval. At this level of difficulty, STEVE and the VLA are almost entirely incapable of producing sensible actions. Within our model family, we find a significant difference from scale: Pan-4B scores 34.7%95% CI[32.0%, 37.4%], Pan-2B 26.9%95% CI[24.4%, 29.4%], and Pan-200M 13.9%95% CI[12.1%, 15.8%].

We find that our models have difficulty recovering from mistakes. They will sometimes break wood or stone with their bare hands, but with only 30 seconds they can only do this so many times (these models don't have access to hotbar switching). Another failure mode is that the models sometimes wander around, seeming to look for the structure already built elsewhere in the world. It would be interesting to train larger models with much longer context windows, to see if they eventually give up looking and start building the structure themselves.

#Mechanisms

To play Minecraft competently, you must know about dozens of complex and specific mechanisms. Here we find significant returns to scale. On a diverse set of 30 environments testing many different mechanisms, Pan-4B scores 24.4%95% CI[22.3%, 26.5%], compared with 15.1%95% CI[13.5%, 16.9%] for the next-best model, Pan-2B—a margin of 9.3%95% CI[6.5%, 12.1%]. We find that Pan-4B significantly outperforms all other models tested.

Some notable successes:

And failures:

#Combat

We command Pan to fight by putting it in a narrow hallway, with a monster or two in between it and a gold panel. Pan-2B and Pan-4B have learned an effective general-purpose strategy for fighting monsters, first hitting, then backing up and waiting, then going forward and hitting again. This allows Pan to kill creepers and zombies effectively, even in tight quarters. Pan does not use a shield well, and often fails to kill skeletons.

#Difficult Environments

We construct some environments to be out-of-distribution by including substantial hand-made structures.

The following environment is called gauntlet. Pan has never seen this construction before, but still moves effectively, and the largest model sometimes completes it (note that grading on these envs is continuous; Pan is the only model to complete them):

#Reward Hacking

We've found some interesting instances of reward hacking, where the model finds a way to reproduce the goal image that wasn't what we intended. In one, rather than riding a horse over some fences to reach a gold panel, it sneaks around the fences, and jumps in order to reproduce the higher viewpoint it would have had were it on the horse. In another, rather than building a path from one side of a bridge to another, it falls through the gap but then manages to climb back up the other side of the ravine and reach the goal.

Note that unlike the reward hacking that occurs with online RL methods, this model was trained with purely offline data, and has never seen these environments before.

#Future work

The point of finding training objectives that work well on internet-scale data is that it enables scaling up the models, not just a little bit, but by many orders of magnitude. If the benefits to goal-conditioned pretraining continue to hold at larger scale, this could produce models that are superhuman at achieving goals in any kind of visual environment, digital or real.

Over the coming months, we plan to rapidly scale these models, and to train them on a much broader distribution of data, including many more video games, computer use, general real-world data, and robotics.

We're also excited to use these models as an initialization for online reinforcement learning. Like with language models, we expect that we'll be able to greatly improve performance via online RL, as long as we can start from an initialization that is at least moderately capable of a lot of things. (We're also designing hardware that will be uniquely good at RL in the real world.)

We're working on training fully general robotics models that can do almost anything in the physical world and operate autonomously for hours at a time. We think simple, scalable methods are what will get us there.

If this is the kind of work you're interested in, join us!

#Appendix

#Further reading

There is some beautiful math connecting goal-conditioned RL with traditional reward functions. Essentially, each goal corresponds to a different reward function, and the value function is replaced by the probability that a state sampled from the future of an agent's trajectory is the goal state.⁹9In goal-conditioned RL, we are interested in the successor measure for a policy: $\sigma_t(s, g) = \sum_{\delta t=1}^{\infty} \mathbb{P}(s_{t+\delta t} = g)$

#Why work on video game agents?

Video games are a useful testing ground for robotics. They are reproducible, open-ended, safe to run any model inside of, and can support very long-horizon goals (for example, collecting materials in order to build a large structure). They're also easier than robotics: if a method can't solve video games, we think it's unlikely to produce general robotics models.

#Model specs

We pretrained a family of goal-conditioned models from scratch on about 500k hours of diverse videos of Minecraft gameplay. Afterwards, we post-trained them on about 2k hours of contractor trajectories including both videos and action sequences.

For compute efficiency, these models were trained on videos with a resolution of 128x128p at 10 FPS. Although the low resolution poses a limitation for some activities such as crafting, it still allows for the emergence of some quite complex behaviors.

These models were trained with a context window of 300 frames, or about 30 seconds of gameplay. For simplicity, we only considered goals that occurred within the 300 frame context window, though we are excited about future work that could consider more distant goals. We use an efficient frame-representation scheme, and we estimate that training a larger model with a 1M token context window would correspond to about 7 hours of context, though long context was not the focus of this project.

As an action space, the model has access to 9 keys: WASD, space, left shift, right shift, left click, right click. In some of our experiments (labelled below), we also give the model access to the number keys 1-9 and a discrete representation of the scroll wheel (+1, -1, or 0), to allow it to switch between items in the hotbar. Lastly, the action space has two continuous dimensions representing mouse movements along the X and Y axes.

#Comparison details

VLA:

For this model, we tokenized the action space using separate sequence positions for each of the 9 discrete keys, and 256 quantile tokens for the two continuous mouse axes. It was post-trained with the same distribution of goals on the same action dataset as our pretrained models, and differs only in which set of pretrained weights are used at initialization, Gemma 4 or our goal-conditioned pretrained models.

STEVE-1:

This model is prompted with 16-frame goal videos instead of single goal images, so for each evaluation environment we load the goal world, run 16 frames of timesteps, and save the resulting goal video to command the model.

#Exploration results

Exploration is an important part of acting in an RL environment. We test Pan's ability to explore the world in pursuit of a particular tree, or arrangement of flowers, or pool of lava. This task has real-world applications: if I am a robot looking for a wrench in an unknown environment, how do I read context clues to guide my exploration?

The Pan models significantly outperform STEVE and the VLA (the delta between Pan-4B and STEVE is 20.8%95% CI[15.4%, 26.8%], Pan-4B and the VLA 20.3%95% CI[14.1%, 26.4%]), and are surprisingly consistent in some of these environments, suggesting that they are picking up on subtle clues that guide the exploration.

#Hotbar switching results

We trained a set of models that have access to the number keys and scrollwheel, allowing them to switch items in the hotbar. We find that they do learn to switch to the necessary tool for a task. However, they still struggle to switch back and forth between a block that they're building with and the necessary tool to break misplaced blocks.

#Why goal-conditioning?

Goal-conditioned pretraining is a self-supervised method for learning agentic behavior from a large, diverse dataset of videos or other observation-only datasets. It is similar to next-token prediction in that it doesn't require human annotations, is mathematically principled, and is completely general, being capable of accomplishing arbitrary tasks at inference time without re-training. Relative to learning a world model, goal-conditioning is similarly able to learn from large amounts of unlabelled data, but doesn't require a separate step to learn a policy for each task we want the agent to perform.¹¹11The two ideas could be combined: it may be possible to train a goal-conditioned agent inside of a world model, producing a task-agnostic policy while leveraging the cheap simulation enabled by world modeling. Future work could explore this direction.↩¹²12One example to help understand the difference between the two kinds of pretraining: learning a world model for a Rubik's cube doesn't teach you very much, because it's easy to predict what the cube will look like after you rotate it. Goal-conditioned training requires the model to be much smarter, because you have to know what moves to do in order to go from any state of the cube to any other state.↩

Goal-conditioned models are often more general than models trained to maximize a numerical score. For example, Dreamer 4¹⁰10Danijar Hafner, Wilson Yan, and Timothy Lillicrap, Training Agents Inside of Scalable World Models, 2025.↩ is trained on 20 different tasks, each with a manually specified reward function. This limits the range of applicability of the resulting agent to just those 20 tasks, and adding a new task would entail re-training on the new reward function. Our goal-conditioned agents, on the other hand, are trained once, and are able to achieve new goals that we come up with at inference time, without any training at all.¹³13There are ways of extending the idea of goal-conditioning further. For example, a language model could produce a text description of what has happened in a video so far, and this text could be used as the goal. More generally, any function of a prefix of observations can be used. The important thing is that the goal descriptions be inferrable from video alone, rather than needing to be annotated manually. Some GCRL algorithms are even more general and are able to accomplish linear combinations of goals, e.g. CRL.↩