Beyond Imitation: How Reinforcement Learning Is Improving Robot Policies in 2026

A robot policy trained by imitation can only be as good as the people it copied. If the operators were careful, the robot is careful. If half of them were tired and rushed, the robot learns a blend of careful and rushed, and nobody tells it which half to trust.

For most of the last few years, the answer was to collect more and cleaner demonstrations. In 2026 a different answer took over. Labs now start with imitation learning, put the robot to work, and then improve it with reinforcement learning on the robot's own attempts. Since late 2025, Physical Intelligence, Stanford, Skild AI with Carnegie Mellon, and a group from Tsinghua and partners have all published results showing the same thing: a policy that starts from demonstrations and then learns from its own practice beats the demonstrations it started from.

This post covers why plain imitation hits a ceiling, the main recipes labs are using to push past it, the results from this year, and a short Python example you can run to watch the core idea work. If you are new to the topic, my overview of imitation learning, the DAgger explainer and the post on robots learning from human video set the scene.

Why imitation hits a ceiling

Behavior cloning learns to predict what the demonstrator did. That has three limits that more data alone does not fix.

• It copies the average, not the best. With mixed quality data, the policy lands somewhere in the middle. It has no signal telling it that one grasp worked and another one slipped.
• It never sees its own mistakes. This is the covariate shift problem from the DAgger post. The policy drifts into states the demonstrator never visited and has no idea how to recover.
• It cannot get faster than people. A robot that copies a teleoperator inherits their pace, including the pauses. For a factory or a café, speed is the whole point.

Reinforcement learning answers all three with one idea: score the robot's attempts, and shift the policy toward the attempts that scored well. The hard part has always been doing that on a real robot without breaking it, and without needing millions of trials.

The recipe most labs now follow

Across this year's papers, the training plan looks remarkably similar:

1. Pretrain a large vision language action model on many robots and tasks.
2. Fine tune with imitation on demonstrations of the target task. This gives a policy that works some of the time.
3. Improve with reinforcement learning using rewards from real attempts, simulation, or a learned reward model, while keeping an imitation loss in the mix so the policy does not forget what it already knew.

Step 3 is where the methods differ. The table below shows the four approaches I cover in this post.

Method	Where the RL happens	Main trick
π*0.6 with Recap (Physical Intelligence)	On real robots doing real work	Condition the policy on an advantage score, and add expert corrections
RL-Co (Tsinghua and partners)	In simulation, anchored to real demos	Mix simulation RL with a supervised loss on real data
FlowDPG (Skild AI, CMU, Penn)	Offline on demos, then online on the robot	A policy gradient for flow matching policies without backpropagating through the whole sampler
Real-Time EXPO-FT (Stanford)	Online on the robot, ten minutes of data	A small, fast edit policy that corrects the big model's actions

π*0.6 and Recap: learning from work

The result that pulled this idea into the spotlight came from Physical Intelligence in November 2025, and it shaped much of what followed in 2026. Their model π*0.6 is trained with a method they call Recap, short for RL with Experience and Corrections via Advantage conditioned Policies.

Recap mixes three kinds of data:

• Demonstrations to teach the basic skill.
• Corrections, where an expert takes over the moment the robot goes wrong and shows the way back. If this sounds like DAgger, it is the same instinct: label the states the robot actually reaches.
• Autonomous experience, where the robot works on its own and every attempt is scored by its outcome.

The clever part is credit assignment. A failure at the end of a task might come from a bad grasp much earlier. Recap trains a value function that estimates how promising each moment is. The change in that value after an action, called the advantage, says whether the action helped. The policy is trained on all of the data with the advantage as an extra input, and at run time it is asked for high advantage actions. That lets it do better than the average of its own training data.

The numbers they reported:

• Throughput more than doubled on some of the hardest tasks, and failure rates fell by half or more.
• The robot made espresso drinks from 5:30 in the morning until 11:30 at night.
• It folded 50 laundry items it had never seen in a new home, and assembled and labelled 59 chocolate packaging boxes in a real factory, running for hours without stopping.

RL-Co: practice in simulation, stay honest with real data

Real robot practice is slow and wears out hardware. Simulation is cheap but never quite right. RL-Co, from Liangzhi Shi, Chao Yu, Yu Wang and colleagues at Tsinghua, Harbin Institute of Technology, Peking University, CMU and others, tries to get the best of both.

Stage one fine tunes the policy on a mix of 20 to 50 real teleoperated demonstrations per task and 1,000 generated demonstrations in simulation. Stage two runs reinforcement learning in simulation, with a supervised loss on the real demonstrations kept in the objective. That anchor stops the policy from drifting into habits that only work inside the simulator.

On four tasks with a Franka Panda arm, average real world success came out like this:

Model	Real data only	Imitation on real and simulated data	RL-Co
OpenVLA	16.5%	40.0%	64.0%
π0.5	26.7%	45.9%	66.2%

One detail stuck with me. On the drawer opening task with OpenVLA, imitation on mixed data scored 0 percent, and RL-Co reached 35 percent. Imitation had nothing to copy that worked; practice found something.

FlowDPG: reinforcement learning for flow matching policies

Many modern robot policies, including π0.5 and GR00T, generate actions with flow matching or diffusion. The policy starts from noise and refines it over several steps into an action. That makes them great at copying varied demonstrations, and awkward for reinforcement learning. A standard actor critic update has to push a gradient back through every refinement step, which is slow and unstable.

FlowDPG, from Kexin Shi, Deepak Pathak, Shikhar Bahl and colleagues at Skild AI, Carnegie Mellon and the University of Pennsylvania, accepted at CoRL 2026, sidesteps that. It estimates the final clean action in one pass, asks a learned critic which way the action should move, and folds that nudge into the training target for the flow. A behavior cloning term stays in the loss as an anchor. Running the policy afterwards is unchanged.

The real robot results are the most convincing part:

• AirPods assembly with two Franka arms over 8 stages: 92 percent end to end success after online fine tuning, against 64 percent for the behavior cloning starting point.
• Scrambled egg cooking with two YAM arms over 14 stages: the rubric score rose from 64.2 to 85.8 percent from offline training alone.
• Recovering from disturbances: online training raised recovery rates by 6 to 18 points across four kinds of pushes and interruptions.

Real-Time EXPO-FT: fixing slow big models

Large vision language action models are slow. By the time an action comes out, the world has moved on, which ruins tasks like catching or balancing. In September, Perry Dong, Kuo-Han Hung, Dorsa Sadigh and Chelsea Finn at Stanford proposed Real-Time EXPO-FT.

The big pretrained model still proposes a chunk of actions. A small edit policy, fast enough to use the very latest camera frame, adjusts each action just before it runs, and that small policy is the part trained with reinforcement learning. On four dynamic real world tasks (passing objects, balancing a ball, table soccer kicks and picking moving objects) average performance rose from 42 percent to 97 percent, using at most 10 minutes of online robot data and no human help during training.

Imitation still matters: lessons from the REAL-I challenge

Reinforcement learning does not replace good demonstrations, and a report from the first REAL-I challenge at ICRA 2026 in Vienna is a useful reality check. Teams trained policies on a fixed set of demonstrations and deployed them on a shared bimanual humanoid, with the final round run on site. NUS-CLEAR won with a score of 71.0.

Their lessons apply to anyone fine tuning a robot policy, with or without RL:

• A short round of training on about 100 trajectories recorded at the venue beat deploying the earlier checkpoint as is. Lighting and calibration changes mattered.
• Fine tuning on a small dataset can erase skills the model had before, the same forgetting problem RL-Co guards against.
• Dropping whole bad trajectories can throw away useful recovery moves. Filtering short segments was safer.
• Lower action error on held out data did not always mean better behavior on the robot. The only test that counts is the robot.

A small Python example: weighting demonstrations by advantage

The core trick in Recap, and in a family of older methods like advantage weighted regression, fits in a few lines. Give each demonstration a weight based on how much better it did than usual for that situation, then run ordinary behavior cloning with those weights.

The toy task: a robot sees a target at position x and moves its gripper there, so the best action is a = x. We have 1,000 demonstrations. 300 come from a careful operator who hits the target. 700 come from a rushed operator who stops 40 percent short. Plain cloning learns the blend. Advantage weighting learns to trust the careful ones.

import numpy as np

rng = np.random.default_rng(0)

# ------------------------------------------------------------ the task
# A robot sees a target at position x and must move its gripper there.
# The best action is a = x. Reward is the negative squared miss.


def reward(x, a):
    return -(a - x) ** 2


# ------------------------------------------------------------ the demonstrations
# 300 demos from a careful operator, 700 from a rushed one who stops short.
n_good, n_rushed = 300, 700
x = rng.uniform(-1, 1, n_good + n_rushed)
a = np.concatenate([
    x[:n_good] + rng.normal(0, 0.05, n_good),              # careful: on target
    0.6 * x[n_good:] + rng.normal(0, 0.05, n_rushed),      # rushed: stops 40% short
])
r = reward(x, a)


def fit_line(x, y, w=None):
    """Weighted least squares fit of y = k * x + b."""
    w = np.ones_like(x) if w is None else w
    X = np.stack([x, np.ones_like(x)], axis=1)
    W = np.sqrt(w)[:, None]
    k, b = np.linalg.lstsq(X * W, y * W[:, 0], rcond=None)[0]
    return k, b


def evaluate(k, b, n=10_000):
    xt = rng.uniform(-1, 1, n)
    return reward(xt, k * xt + b).mean()


# ------------------------------------------------------------ 1. plain behavior cloning
k_bc, b_bc = fit_line(x, a)

# ------------------------------------------------------------ 2. advantage weighted cloning
# Value: the reward we expect at this state, fitted on x squared.
X_v = np.stack([x ** 2, np.ones_like(x)], axis=1)
v = X_v @ np.linalg.lstsq(X_v, r, rcond=None)[0]
advantage = r - v                          # better or worse than usual here?
weights = np.exp(advantage / 0.01)         # trust better than usual actions more
weights /= weights.mean()
k_aw, b_aw = fit_line(x, a, weights)

print(f"plain behavior cloning     slope {k_bc:.2f}   mean reward {evaluate(k_bc, b_bc):.5f}")
print(f"advantage weighted cloning slope {k_aw:.2f}   mean reward {evaluate(k_aw, b_aw):.5f}")
print(f"perfect policy             slope 1.00   mean reward {evaluate(1.0, 0.0):.5f}")

Running it prints:

plain behavior cloning     slope 0.73   mean reward -0.02393
advantage weighted cloning slope 1.00   mean reward -0.00004
perfect policy             slope 1.00   mean reward 0.00000

Plain cloning learns a slope of 0.73, a mix of the two operators, so it misses most targets. The weighted version recovers a slope of 1.00, almost exactly the careful operator, from the very same data. Nothing was relabelled and nobody marked which operator was which. The reward and the value estimate did that work.

The number 0.01 controls how sharply the weights favour good actions. When I set it to 0.05, the slope came out at 0.91: better than plain cloning, not as sharp. Real systems tune this carefully, and Recap sidesteps the choice by feeding the advantage to the policy as an input rather than as a weight.

What this means for robot learning

The old picture was a pipeline that ended when the demonstrations ran out. The new picture has no fixed end. A robot is trained by imitation, starts working, and keeps improving from what happens on the job, with people stepping in to correct it when it goes wrong.

There are open problems. Rewards are hard to define for messy real tasks, which is why FlowDPG uses a learned reward model that scores progress through stages. Real robot practice is still slow and costly, which is why RL-Co moves most of it into simulation and Real-Time EXPO-FT keeps it under ten minutes. And every method here keeps an imitation term in the loss, which tells you how easily pure reinforcement learning drifts.

For my own imitation learning course, this connects the old papers to the new ones. DAgger asked the expert to label the states the learner visits. Recap asks the expert to step in at those same moments, and then lets the robot learn from its own outcomes on top. The question imitation learning started with has not changed: how do you teach a policy about the situations it creates for itself?

Frequently asked questions

Closing thoughts

The idea that a robot policy is finished once training ends is fading. In 2026 the best results came from robots that kept learning after deployment, from their own attempts and from people stepping in at the right moments. Imitation learning gets the robot to the starting line. Practice takes it the rest of the way.

If you try the Python example, change the share of rushed demonstrations or the 0.01 setting and see when the weighting stops being able to find the careful operator. I would like to hear what you find.

References

• Physical Intelligence, π*0.6: a VLA that learns from experience, November 2025. pi.website
• L. Shi, S. Chen and others, Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models, 2026. arXiv:2602.12628
• K. Shi, J. Shi, P. Hebbar, Z. Zhao, T. Amarnath, Y. Su, S. Bahl and D. Pathak, FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulation, CoRL 2026. arXiv:2606.22303
• P. Dong, K. Hung, D. Sadigh and C. Finn, Reinforcement Learning for Real-Time Vision-Language-Action Policies, 2026. arXiv:2609.18207
• J. Wang, J. Chen, D. Liu and others, How to Better Train VLAs: Lessons Learned From the REAL-I Challenge at ICRA 2026, 2026. arXiv:2609.13679
• X. B. Peng, A. Kumar, G. Zhang and S. Levine, Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning, 2019. arXiv:1910.00177