World Models for Robots Explained: How Robots Learn by Predicting the Future, with Python Code

Before you catch a ball, you already know roughly where it will land. Before you push a full cup across a table, you can tell it might spill. You run a quick prediction of the world in your head and act on it. A robot that can do the same thing does not need to learn every lesson the hard way, on real hardware, one crash at a time.

That prediction is called a world model, and in 2026 it is one of the busiest ideas in robot learning. A large survey on world models for robot learning, with authors from Berkeley, Stanford, Oxford, ETH Zurich and others including Pieter Abbeel and Jitendra Malik, appeared this year. NVIDIA's DreamZero, a world model that outputs robot actions directly, beat the leading vision language action models on a public robot benchmark. Meta and NVIDIA both now release world models built for robots.

This post explains what a world model is, the three ways robots use one, how this year's biggest systems compare, and what still does not work. At the end, a short Python example shows the main payoff in numbers: a robot with a world model needs about 55 times less real practice than one without. If reinforcement learning is new to you, start with our Markov decision process explainer, which covers the states, actions and rewards used here.

World models at a glance

What it is	A learned model that predicts what happens next: given the current state and an action, it predicts the next state, and often the reward
Why robots use it	Real practice is slow, costly and can break hardware. Practice inside a model is fast, cheap and safe
Three main uses	Train a policy inside the model, plan by comparing predicted futures, or build the robot policy on top of a video world model
Landmark systems	DreamerV3 (Nature, 2025), DayDreamer (CoRL 2022), V-JEPA 2 (Meta, 2025), Cosmos Policy (NVIDIA and Stanford, 2026), DreamZero (NVIDIA, 2026)
Big open problems	Models that are confidently wrong, slow prediction, heavy compute, and no agreed way to measure them

What is a world model?

In reinforcement learning, a robot sees a state, takes an action, and the world moves to a new state and hands back a reward. Most robot learning treats the world as a black box. The robot tries something, sees what happens and adjusts. A world model opens the box: it is a network trained to predict the next state from the current state and action.

Once a robot has that, it can ask "what if" questions without moving. What if I push harder? What if I grasp the cup from the left? It can run thousands of these questions per second inside the model, then do only the best one for real.

World models come in a few forms:

• Small state models predict a handful of numbers, such as joint angles and velocities. They are cheap and work well for control.
• Latent models compress camera images into a short code and predict how that code changes. Dreamer and V-JEPA 2 work this way.
• Video world models predict whole future video frames. They are large and slow, but they can learn from the huge amount of video on the internet. Cosmos and DreamZero are built on models like these.

Why robots need world models

Robots learn slowly from real experience. Our overview of reinforcement learning beyond imitation shows how much effort labs spend to keep real robot practice short. Every real attempt takes seconds, needs a person to reset the scene, and wears out motors.

A world model changes the maths. The robot still needs real data, but it uses that data twice: once to learn how the world works, and then again and again inside the model to practise. DreamerV3, by Danijar Hafner and colleagues, published in Nature in 2025, learned more than 150 tasks with one fixed set of settings this way. It was the first method to collect diamonds in Minecraft from scratch, with no human data. DayDreamer, from Berkeley in 2022, applied the same idea to real robots: a four legged robot learned to roll off its back, stand up and walk in about one hour, with no simulator.

Three ways robots use world models

The 2026 survey sorts the field into a few uses. Three matter most for robots today.

1. Practise inside the model

The robot learns a world model from its real experience, then trains its policy on rollouts predicted by the model instead of real ones. Dreamer is the classic example. It starts each practice rollout from a real moment it remembers, predicts about 15 steps ahead, and trains an actor and a critic on those predicted steps. Real data keeps the model honest. The model supplies most of the practice.

2. Plan by comparing predicted futures

Instead of training a policy at all, the robot can plan at every step. It proposes many action sequences, predicts the outcome of each with the world model, takes the first action of the best one, and repeats. This is model predictive control, the same loop our Python example uses.

Meta's V-JEPA 2, released in June 2025, does this at scale. It learned from over 1 million hours of internet video, then from less than 62 hours of unlabeled robot video from the DROID dataset. With that, it picked and placed objects on Franka arms in two labs where it had never seen any data, with no task specific training and no reward, only a picture of the goal.

3. World action models

The newest idea flips the order. Start from a large video world model that already predicts how scenes change, and train it to output robot actions next to its predicted video. NVIDIA calls these world action models.

• Cosmos Policy (NVIDIA and Stanford, January 2026) fine tunes the 2 billion parameter Cosmos-Predict2 video model on robot demonstrations, with no changes to its architecture. Actions, future images and value estimates all become extra frames in the video. It reached 98.5% average success on the LIBERO benchmark and 67.1% on RoboCasa with only 50 demonstrations per task. Using its own predictions to choose between candidate actions raised real world task completion by 12.5% on two hard tasks.
• DreamZero (NVIDIA, February 2026) is a 14 billion parameter model built on the Wan2.1 video model. It predicts future frames and actions together. On a mobile two armed robot it reached 62.2% average task progress on seen tasks against 27.4% for the best pretrained vision language action model, and 39.5% against 16.3% on tasks it had never seen. With 12 minutes of human video or 20 minutes of video from another robot, its results on new tasks rose by over 42%.

These models connect directly to our earlier post on robots learning from human video. A model that already predicts how hands and objects move in video has a head start on learning robot actions.

World action models vs vision language action models

The big debate of 2026 is which starting point is better for a robot foundation model.

	Vision language action models (VLAs)	World action models
Built on	A vision language model, trained on text and images	A video model, trained to predict how scenes change
Strength	Follows language instructions well, fast	Knows how objects move, generalizes to new scenes
Weakness	Weak sense of physics and motion	Slow and costly to run
Examples	π0.5, GR00T N1	DreamZero, Cosmos Policy

On RoboArena, a public benchmark where policies compete on real robots, DreamZero scored an Elo rating of 1750 in April 2026, ahead of π0.5 at 1622. The cost is speed. NVIDIA's own analysis puts world action models at about 590 to 800 milliseconds per chunk of actions, against about 190 milliseconds for π0.5. NVIDIA's June 2026 review expects the eventual winner to be a hybrid of the two.

A small Python example: practice with and without a world model

The example below shows the main payoff in plain NumPy. A cart sits on a track. The robot pushes it left or right and has to park it at position 1.0. The real physics include drag that grows with the square of the speed, which the robot does not know.

We compare two robots:

1. Trial and error only. A small policy improved by random search. Every update costs two full real episodes.
2. World model, then plan. The robot pushes the cart at random for 100 steps, fits a simple world model with least squares, then plans at every step with the cross entropy method: it predicts the outcome of 100 action plans in the model, keeps the best 10, refines its guess around them three times, and acts. After each real episode it refits the model on all the data so far.

The world model is deliberately too simple. It is linear, so it cannot represent the real drag. Planning again at every step corrects for most of that error.

import numpy as np

# A cart on a track. The robot pushes it (action between -1 and 1) and must
# park it at position 1.0. The robot does not know the physics: the cart has
# drag that grows with speed squared, which no simple model captures exactly.
DT, STEPS, GOAL = 0.1, 50, 1.0


def real_step(x, v, a):
    a = np.clip(a, -1, 1)
    v = v + DT * (2.0 * a - 0.8 * v * np.abs(v) - 0.3 * v)
    return x + DT * v, v


def cost(x, v):
    return (x - GOAL) ** 2 + 0.1 * v ** 2


def parked(x, v):
    return np.abs(x - GOAL) < 0.05 and np.abs(v) < 0.1


# the same 20 start positions are used to test both robots
TEST_STARTS = np.linspace(-1.0, 0.5, 20)


# ---------------------------------------- 1. no world model: trial and error
def run_policy(w, x0):
    x, v, total = x0, 0.0, 0.0
    for _ in range(STEPS):
        a = np.tanh(w @ np.array([x - GOAL, v, 1.0]))
        x, v = real_step(x, v, a)
        total += cost(x, v)
    return total, parked(x, v)


def model_free(rng, budget):
    """Random search on a small policy. Every trial costs real robot steps."""
    w, used, curve = np.zeros(3), 0, []
    while used < budget:
        d = rng.normal(0, 1, 3)
        x0 = rng.uniform(-1, 0.5)
        up, _ = run_policy(w + 0.3 * d, x0)
        down, _ = run_policy(w - 0.3 * d, x0)
        used += 2 * STEPS
        w -= 0.05 * (up - down) * d / (abs(up - down) + 1e-8)
        if used % 1000 == 0:
            ok = [run_policy(w, s)[1] for s in TEST_STARTS]
            curve.append((used, np.mean(ok)))
    return curve


# ---------------------------------------- 2. learn a world model, then plan
def fit_world_model(data):
    """Least squares: predict the change in (x, v) from (x, v, a, 1)."""
    X = np.array([[x, v, a, 1.0] for x, v, a, _, _ in data])
    Y = np.array([[x2 - x, v2 - v] for x, v, a, x2, v2 in data])
    return np.linalg.lstsq(X, Y, rcond=None)[0]


def imagine(W, x, v, plans):
    """Roll many action plans forward inside the world model, at once."""
    x = np.full(len(plans), x)
    v = np.full(len(plans), v)
    total = np.zeros(len(plans))
    for t in range(plans.shape[1]):
        d = np.stack([x, v, plans[:, t], np.ones_like(x)], 1) @ W
        x, v = x + d[:, 0], v + d[:, 1]
        total += cost(x, v)
    return total


def plan(W, x, v, rng, mean):
    """Cross entropy method: imagine 100 plans, keep the best 10,
    refine the guess around them, and repeat."""
    std = np.full(15, 0.6)
    for _ in range(3):
        plans = np.clip(mean + std * rng.normal(size=(100, 15)), -1, 1)
        best = plans[np.argsort(imagine(W, x, v, plans))[:10]]
        mean, std = best.mean(0), best.std(0) + 0.05
    return mean


def mpc_episode(W, x0, rng, data=None):
    """Plan in imagination, take one real step, then plan again."""
    x, v, mean = x0, 0.0, np.zeros(15)
    for _ in range(STEPS):
        mean = plan(W, x, v, rng, mean)
        a = mean[0]
        x2, v2 = real_step(x, v, a)
        if data is not None:
            data.append((x, v, a, x2, v2))
        x, v = x2, v2
        mean = np.append(mean[1:], 0.0)  # reuse the rest of the plan
    return parked(x, v)


def model_based(rng, budget):
    data, used, curve = [], 0, []
    x, v = rng.uniform(-1, 0.5), 0.0
    for _ in range(100):  # 100 steps of random pushing to start
        a = rng.uniform(-1, 1)
        x2, v2 = real_step(x, v, a)
        data.append((x, v, a, x2, v2))
        x, v = x2, v2
    used = 100
    while used < budget:
        W = fit_world_model(data)
        test = [mpc_episode(W, s, rng) for s in TEST_STARTS]
        curve.append((used, np.mean(test)))
        # practise once on the real cart and keep the new data
        mpc_episode(W, rng.uniform(-1, 0.5), rng, data)
        used += STEPS
    return curve


def steps_to_90(curve):
    return next((used for used, ok in curve if ok >= 0.9), None)


mb_runs, mf_runs = [], []
for seed in range(5):
    rng = np.random.default_rng(seed)
    mb_runs.append(model_based(rng, 400))
    mf_runs.append(model_free(rng, 40_000))

print("Real robot steps until the cart parks in 90% of 20 test starts:")
for name, runs in [("World model + planning", mb_runs),
                   ("Trial and error only", mf_runs)]:
    s = [steps_to_90(c) for c in runs]
    runs_text = ", ".join(f"{x:,}" for x in s)
    print(f"  {name:<24} {runs_text}   (mean {np.mean(s):,.0f})")
print("World model + planning, success after each practice episode (seed 0):")
print("  " + "  ".join(f"{used}:{ok:.0%}" for used, ok in mb_runs[0]))

Running it prints (about a minute on a laptop):

Real robot steps until the cart parks in 90% of 20 test starts:
  World model + planning   100, 100, 100, 150, 100   (mean 110)
  Trial and error only     4,000, 4,000, 8,000, 6,000, 8,000   (mean 6,000)
World model + planning, success after each practice episode (seed 0):
  100:100%  150:90%  200:85%  250:100%  300:85%  350:100%

The robot with a world model parked the cart in 90% of test runs after about 110 real steps on average. Trial and error needed about 6,000, roughly 55 times more. On a real robot, where each step costs time and wear, that gap decides whether a skill takes minutes or days to learn.

The plot shows the cost too. The world model robot's success moves between about 88% and 97% from episode to episode. Its model is wrong about the drag, and it never gets fully reliable. Trial and error, once it gets there, hits 100% and stays there, since it learned from the real physics directly. That trade is the whole story of world models in small form: far less real practice, in exchange for living with a model that is never quite right.

What still does not work

The survey lists several open problems. Four stand out.

Models that are confidently wrong. A planner searches for the best plan inside the model, so it is drawn to the places where the model is most wrong in its favour. Real robots then fail in ways the model said were safe. Collecting fresh real data, as our example does after every episode, is the usual fix.

Speed. Predicting video is slow. DreamZero needed a 38 times speedup to run at about 7 control steps per second. Many tasks need far faster reactions.

Compute. Large video world models need far more training compute than vision language action models. NVIDIA estimates the full video pretraining stack at several times the cost of an efficient VLA recipe.

Measuring them. A world model can make video that looks right but has the wrong physics. There is still no agreed test for whether a world model is good enough to trust for robot control.

Frequently asked questions

Closing thoughts

For most of the history of robot learning, robots learned like someone touching a hot stove: by doing it and feeling the result. World models let them learn more like a chess player, by thinking a few moves ahead. The idea is decades old. What changed in 2026 is scale: video models trained on internet video now know enough about how the world moves to give robots a real head start.

If you run the example, try making the world model even simpler, or adding noise to the real cart, and watch how planning copes. We would like to hear what you find.

References

• B. Hou, G. Li and others, including P. Abbeel and J. Malik, World Model for Robot Learning: A Comprehensive Survey, 2026. arXiv:2605.00080
• D. Hafner, J. Pasukonis, J. Ba and T. Lillicrap, Mastering Diverse Control Tasks through World Models (DreamerV3), Nature, 2025. arXiv:2301.04104
• P. Wu, A. Escontrela, D. Hafner, K. Goldberg and P. Abbeel, DayDreamer: World Models for Physical Robot Learning, CoRL 2022. arXiv:2206.14176
• M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes and others, V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning, Meta, 2025. arXiv:2506.09985
• M. J. Kim, Y. Gao, T. Y. Lin and others, including P. Liang, S. Song and C. Finn, Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning, 2026. arXiv:2601.16163
• NVIDIA, World Action Models are Zero-shot Policies (DreamZero), 2026. arXiv:2602.15922
• M. Reuss, Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models, NVIDIA Technical Blog, June 2026. developer.nvidia.com