Agentic Robotics Explained: When AI Agents Drive Robot Arms, with Python Code

For the last two years, teaching a robot a new task meant collecting demonstrations and training a policy. A person would drive the arm through the task hundreds of times, and a vision language action (VLA) model would learn to copy it. This month a different idea is taking over the robotics papers: skip the training, and let a general AI model run the robot the way it runs a coding task.

On November 6, 2026, a team including Vladlen Koltun and Matthias Müller posted AMP1, a framework that lets a general model control robot arms directly. With no demonstrations and no task training, it completed 104 of 108 real trials across 12 tasks. The best VLA policy they tested managed 30. The same week, a group including Ken Goldberg published a 28,000 run study of how such agents learn, and an open model team showed that open weight models can do this too.

This post explains what agentic robotics is, how AMP1 works, what the new results show, and where these agents still fail. A short Python example shows the idea that makes them work: an agent that checks its own work can turn a 60% grasp into a 97% success, but only if its checking is good.

Agentic robotics at a glance

The idea	A general AI model controls the robot by looking, writing code, acting and checking, with no demonstrations
In the news	AMP1 (November 6, 2026), uni-robot-agent (November 1), the Goldberg agentic robotics study (November 6), the GPT-6-Astra study (October 15)
The headline number	AMP1 completed 104 of 108 real trials, against 49 for FLUX 3 Action and 30 for π0.5
In simulation	85.6% zero shot on RoboLab-120, against 49.2% for the best VLA and 42.9% for the best world action model
The catch	About 280 seconds per success, 211 of them spent thinking
Open models	Qwen3.8-Flash-Next scored 31.39 on RoboDojo, above GPT-6-Astra's published 28.97
Our Python result	One try: 60%. Act, check and retry: 97% with a good checker, 86% with a weaker one

What is an agentic robot policy?

A VLA policy is a single network. A camera image and an instruction go in, motor commands come out, many times a second. It learns from demonstrations, as we covered in our imitation learning and diffusion policy guides. It is fast, but it can only do what its training data showed it.

An agentic manipulation policy (AMP) works more like a programmer at a robot. A general model, the kind used for chat and coding, gets tools: cameras, a list of objects in the scene, a Python interpreter and a safe way to send the arm to positions. It plans a step, writes code for it, runs it, looks at the camera to see what happened, and decides what to do next. Nothing about the task is trained in. Everything comes from what the model already knows about the world and about code.

The difference is a bit like a trained typist against a writer. The typist is fast on what they have practised. The writer is slow, but can handle a request they have never seen.

How AMP1 works

AMP1 is a reference framework, the code that sits between the model and the robot. It runs unchanged on single arm, two armed and mobile robots. The model gets:

• What it sees: three cameras (two shoulder views and a wrist camera) with colour, depth and calibration, plus joint positions and gripper state. Images enter the model's context only when its own code chooses to show them.
• A scene model: a perception tool that segments and tracks objects, which the model can list, filter, sort and correct.
• A Python interpreter that keeps its state. The model writes its own skills, like a grasp routine, on top of the robot's software kit. The basic framework gives it no grasp, push or place functions at all.
• Safe motion: the model sends timed waypoints for the gripper. The framework previews them, checks reach and collisions, and then runs them.
• Recovery tools: functions to read the arm's state and clear controller errors. Adding them raised real world success from 94 to 104 out of 108.

In simulation the model made about 35 calls per episode with full sensing, and about 54 when depth and calibration were taken away. With no skills given, the agents worked out their own strategies: searching for hidden objects, using one object as a tool on another, and even calibrating their own cameras. A version with a memory that carries over between episodes cut failures on two armed RoboDojo tasks by 4.6 times.

The results: agents vs VLA and WAM policies

The authors compared AMP1 with leading VLA policies and world action models (WAMs), which are policies built on top of a video world model.

	AMP1 agent	Best VLA	Best WAM
RoboLab-120 simulation, zero shot	85.6%	49.2%	42.9%
12 real tasks, 108 trials	104 / 108	30 / 108 (π0.5)	49 / 108 (FLUX 3 Action)

The real tasks include nesting cups, putting all tools except the hammer in a bin, pouring blue cubes onto a plate, and using a spatula to push four cubes into a row. AMP1 set the best result on the CaP-Gym benchmark.

The model matters a lot. Thirteen models were tested on RoboLab-120. Gemini 4 Argon reached 85.6%, Claude Opus 5.5 82.8%, Claude Haiku 5.5 51.6% and the older Claude Opus 4.5 35.2%. The real robot trials used Opus 5.5. Seeing matters too: when images were replaced with text descriptions, Opus 5.5 dropped from 82.8% to 36.8%.

Open models and the GPT-6-Astra study

Two other papers fill in the picture.

Open models can do it. On November 1, a group including Pieter Abbeel, Zhiyuan Liu and Ning Ding released uni-robot-agent, an open framework for robot agents that can be trained further. Out of the box, the open model Qwen3.8-Flash-Next scored 31.39 on RoboDojo with no privileged inputs, higher than the published GPT-6-Astra result of 28.97 and many VLA and WAM baselines. It held up on 19 real tasks. The authors link good robot use to four general skills that future models should train together.

A big model is competent, but not yet practical. In October a large SenseNova team asked "Has GPT-6-Astra Achieved Embodied Intelligence?" They tested it on 12 benchmarks covering table top arms, humanoids, mobile robots and dexterous hands, and on 8 real robot setups. Their answer: it has largely crossed the line of embodied competence, but not of practical embodied intelligence. Pairing it as a slow "System 2" planner with a fast "System 1" controller such as π0.5 worked well, but cost a lot. They propose a new kind of benchmark with a world clock that keeps running while the agent thinks.

How agents learn from failure. The Goldberg study ran 28,000 coding agent episodes over 9 tasks and 631 setups, varying the model, the skills it was given, the feedback it got and the order of tasks. Their main finding is that the hard part is rarely getting enough information. It is turning that information into the right fix. The agents that did well could tell why a try failed, reason about the geometry of the scene, and rewrite only the part of their code that was wrong.

What still goes wrong

• Waiting. In real trials AMP1 averaged 280 seconds per success, 211 of them thinking and 69 moving. The slowest task took 601 seconds. The authors say plainly that agentic policies are not yet practical for this reason.
• Nothing can move. The arm cannot change a motion halfway through, so the tests used still scenes. A moving object or a slipping grip is out of reach for now. This is the latency problem from our last post, but measured in minutes instead of milliseconds.
• Bumps. Most collisions were with the table, bin walls or the robot itself, from small calibration errors and simple collision shapes.
• Thinking it is done. On LIBERO-Goal the agent often believed it had finished when the checker disagreed.
• Weak sensing hurts weak models most. Claude Haiku 5.5 fell to 10.7% with plain colour images and no depth or calibration.

Two of these, the long thinking time and misjudging its own result, are what our Python example looks at.

A Python example: why checking matters

An agent wins by looking at what happened and trying again. We can see how much that is worth, and what it depends on, with a small simulation that needs no robot or model.

A robot has to put a cube in a bowl, and one try works 60% of the time. We compare three ways of running it:

• policy: a fast learned policy that tries once in 6 seconds and stops
• agent: thinks for 20 seconds, moves for 6, looks at the camera, and tries again if it sees a failure, within a 300 second limit
• agent + learns: the same, but every failure it spots makes the next try better, the way AMP1's agents rewrite their grasp code

The agent's check is not perfect. With some chance it misreads the camera: it may think a failure worked and stop, or think a success failed and pick the cube up again.

import numpy as np

# A robot must put a cube in a bowl. One try at grasp and place works
# 60% of the time. We compare three ways of running it:
#   policy : a fast learned policy that tries once and stops
#   agent  : a model that thinks, acts, then looks at the camera to check
#            the result, and tries again if it sees a failure
#   agent + learns : the same, but each failure it spots makes the next
#            try better (it fixes its grasp, as the agents in AMP1 do)
P_TRY = 0.60            # chance one try works
THINK, MOVE = 20, 6     # seconds of reasoning and of motion per try
BUDGET = 300            # seconds before we call the episode a failure


def episode(mode, check_acc, rng):
    """check_acc: chance the agent judges a try correctly from the camera."""
    if mode == "policy":
        return rng.random() < P_TRY, MOVE
    t, p = 0, P_TRY
    while t + THINK + MOVE <= BUDGET:
        t += THINK + MOVE
        worked = rng.random() < p
        sees_success = worked if rng.random() < check_acc else not worked
        if sees_success:           # the agent stops when it thinks it is done
            return worked, t
        # it thinks it failed (rightly or not), so it picks the cube up again
        if mode == "agent + learns":
            p = 1 - (1 - p) * 0.6  # each spotted failure fixes 40% of what was wrong
    return False, t


rng = np.random.default_rng(0)
N = 20000
print(f"{'mode':>16} {'check':>6} {'success':>8} {'mean time':>10}")
for mode in ["policy", "agent", "agent + learns"]:
    for acc in ([1.0] if mode == "policy" else [0.95, 0.80]):
        r = [episode(mode, acc, rng) for _ in range(N)]
        s = np.mean([w for w, _ in r])
        t = np.mean([tt for _, tt in r])
        print(f"{mode:>16} {acc:>6.0%} {s:>8.1%} {t:>9.0f}s")

Running it prints:

            mode  check  success  mean time
          policy   100%    59.7%         6s
           agent    95%    96.7%        44s
           agent    80%    85.9%        46s
  agent + learns    95%    97.4%        40s
  agent + learns    80%    89.5%        43s

Checking turns 60% into 97%. With a check that is right 95% of the time, the agent succeeds 96.7% of the time, against 59.7% for one try. That is the same kind of jump AMP1 shows over VLA policies.

It costs time. The agent needs about 44 seconds on average against 6 for the policy, and most of that is thinking. Real agents have the same split, only larger.

The check is everything. The chart sweeps how often the agent judges its own result correctly. At 50%, a coin flip, the agent does no better than a single try. Every point of checking accuracy after that turns into success. This matches the LIBERO-Goal failures and the Goldberg finding: reading the evidence correctly is the hard part.

Learning from failure helps most when checking is weak. At 50% checking, the agent that improves after each spotted failure reaches 72% instead of 60%. At 95% the two are almost the same, since the agent rarely needs a third try.

This is a toy. Real agents fail in many more ways than one grasp, and their checks are not independent coin flips. But it shows why the new papers spend so much effort on perception, verification and memory, and less on the motion itself.

Agents or learned policies: which one to use

	Learned policy (VLA, WAM)	Agentic policy
Speed	Tens of milliseconds per action	Minutes per task
New tasks	Needs demonstrations or training	Works zero shot from an instruction
Moving objects	Can react in real time	Not yet
Success on the tested tasks	Lower	Much higher
Cost	One GPU on or near the robot	Many large model calls per task

For a fixed task that runs thousands of times, such as sorting parcels, a trained policy is still the right tool. For a new or rare task, such as "put the tools except the hammer away", an agent can do it today without any data collection. The most likely future is the one the GPT-6-Astra study tested: an agent that plans and checks, handing fast motion to a learned controller.

Frequently asked questions

Closing thoughts

A year ago the question was how much robot data a policy needs. This month several groups showed that, for many table top tasks, the answer can be none, if the model is strong enough and the framework gives it good eyes, safe motion and a way to recover.

The open problems are clear: speed, moving scenes and knowing when a job is really done. Each one is a good research topic, and the link to our latency post is direct. The robot that wins will think like an agent and move like a policy.

If you run the example, try a lower success rate for one try, or a tighter time limit, and watch where the agent stops being worth the wait. We would like to hear what you find.

References

• C. Chen, O. Bagherieh and others, including M. Müller and V. Koltun, On Agentic Manipulation Policies, November 2026. arXiv:2611.09892
• K. Chen, X. Zhu and others, including K. Goldberg, A Controlled Study of Models, Skills, Feedback, and Curriculum in Agentic Robotics, November 2026. arXiv:2611.09030
• J. Yuan, Y. Zuo and others, including P. Abbeel and N. Ding, Democratizing Robot-Use Agents: A Unified Harness with Open Models, November 2026. arXiv:2611.01560
• SenseNova Joint Research Team, Has GPT-6-Astra Achieved Embodied Intelligence?, October 2026. arXiv:2610.22471
• ArXivSignals, robotics papers of the week, November 2026. arxivsignals.io