VLA Inference Latency Explained: Why Robot Brains Lag and How to Fix It, with Python Code

A robot arm reaches for a cup on a moving tray. Its camera takes a picture, a large vision language action (VLA) model studies it, and a chunk of motor commands comes back. By then the tray has moved on. The arm closes on the spot where the cup used to be.

This is the latency problem, and this week it got a careful measurement. VLALagBench, posted on November 2, 2026 by researchers including Christos Kozyrakis and Jason Clemons, found that only 100 milliseconds of model delay can nearly halve task success when the target is moving. A few days earlier, Leap-VLA showed a way to run a slow language model and a fast action model side by side, so that the robot keeps reacting while the big model thinks.

This post explains where the delay comes from, what the new results show, and the three tricks robots use to live with it. A short Python example puts numbers on it: at 100 ms of delay, a gripper that waits for its model grasps a moving object 34% of the time, and one that plans for the delay grasps it 85% of the time.

VLA latency at a glance

The problem	A big VLA model takes tens to hundreds of milliseconds to answer, but a robot needs a new command every 20 to 50 ms
In the news	VLALagBench (November 2, 2026) and Leap-VLA (October 31, 2026)
The headline number	100 ms of delay cut π0.5's success on a moving set down task from 71.5% to 40.5%
Fix 1: async execution	Ask for the next chunk early so the robot never stands still
Fix 2: plan for the delay	Freeze what will run anyway and plan the rest from where the robot will be (real time chunking)
Fix 3: split fast and slow	A slow model for meaning, a fast model for motion (Leap-VLA)
Our Python result at 100 ms	Sync 34%, async 38%, async plus delay aware planning 85%

Where the delay comes from

A VLA model is a vision language model with an action head on top. Models like π0.5 and GR00T N1.7 take a camera image and an instruction such as "put the cup on the tray" and return motor commands. The language model part has billions of parameters, and running it takes time, even on a good GPU.

The robot's controller runs much faster. Leap-VLA's authors put it simply: what the scene means changes about once a second, but reacting to it needs commands at roughly 20 to 50 Hz. That is one command every 20 to 50 ms. A model that takes 100 ms to answer cannot keep up one action at a time.

The usual answer is action chunking. Instead of one action, the model returns a short sequence, say ten actions covering 0.2 seconds, and the robot plays them back while nothing else happens. Most modern policies do this, including the diffusion policies we covered earlier. Chunking hides the latency, but it creates a new question: what does the robot do while the next chunk is being computed?

Latency is more than the model's own time. The full loop includes reading the camera, sending the image, waiting in a queue if the GPU serves several robots, running the model, and sending the actions back. Hugging Face measured about 100 ms for a small ACT policy on a 2021 laptop, which is about three camera frames at 30 frames per second, and large VLAs are slower.

Sync vs async execution

There are two ways to run a chunked policy.

Sync (sequential) execution. The robot plays its chunk to the end, sends a new image, and waits. When the next chunk arrives it starts moving again. This is simple, but the robot stops dead for the whole latency every chunk, and the new chunk was planned from an image that is now old.

Async execution. The robot sends a new image before its chunk runs out, so the next chunk arrives just in time. Hugging Face added this to LeRobot in July 2025: a client on the robot keeps a queue of actions and asks the server for more when the queue drops below a set fraction. With SmolVLA they reported tasks finishing about twice as fast with similar success, mostly by removing the pauses.

Async fixes the pauses but not the age of the plan. The new chunk was still computed from an image taken one latency ago, and it plans as if the robot were standing where it stood then. On a still object that barely matters. On a moving one it is the whole problem.

What VLALagBench found

VLALagBench is the first careful study of how much this costs. The authors tested three open VLAs, π0.5, GR00T N1.7 and MolmoAct2, on four LIBERO tasks: releasing, setting down, grasping and opening things. They added a controlled delay and moved the target at set speeds, and ran 200 trials for every setting. They checked the results in Isaac Sim and on a real SO-101 arm.

• Moving targets suffer most. With π0.5 running sync and the target moving at 0.1 m/s, set down success fell from 71.5% at no delay to 40.5% at 100 ms. Grasping fell less, from 86.5% to 74.0%.
• Async helps a lot when pauses are the problem. On the release task at 200 ms, a lazy async mode reached 81.5% against 48.5% for sync. The plans were just as old in both cases. The difference came from not stopping.
• Async does not fix everything. At 200 ms, neither async mode brought set down or grasping back to their no delay success.
• Success can be predicted. A simple model fitted on a few calibration runs predicted success within 5.7 points (π0.5) and 7.8 points (MolmoAct2) on average. GR00T N1.7 was harder to fit.
• That prediction picks settings. For a robot sharing an H100 over 5G, the model chose async execution with a short horizon and predicted 84.5% success. The robot measured 82.0%.

The useful lesson is that there is no single best setting. The right chunk length and execution mode depend on how fast the world moves, how long the model takes and how busy the server is.

Three fixes that work in 2026

1. Async execution

Overlap thinking with moving. It is the cheapest fix and it removes the stop and go motion. LeRobot's async inference and the lazy and continuous modes in VLALagBench are examples.

2. Plan for when the action lands: real time chunking

Kevin Black, Manuel Galliker and Sergey Levine at Physical Intelligence proposed real time chunking (RTC) in 2025. While one chunk runs, the policy plans the next one. The actions that will run before the new chunk can arrive are already decided, so RTC freezes them and asks the model to fill in only the rest, joining smoothly onto the frozen part. It works with any diffusion or flow based policy and needs no retraining. They tested it on 12 fast simulated tasks and 6 real two armed tasks, including lighting a match, and found it kept success high even with large delays.

3. Split the slow brain from the fast hands: Leap-VLA

Leap-VLA, posted on October 31, 2026, runs the big vision language model rarely and keeps its output in a cache. A small flow matching action model runs at a high rate, using that cache plus a fresh image from a compact camera encoder. The action model is told how old the cache is, so it can make allowances for stale meaning.

On LIBERO-Spatial, success rose from 61.4% to 95.5% while the language model refreshed only once every 2 seconds. Across four LIBERO suites it averaged 95.2% with a 1 second refresh, against 96.3% for a version that ran both models at the same rate. A small fix at chunk boundaries let it replan at every control step with 97.1% success and no added delay.

All three ideas share one rule: the plan must be for the moment the action runs, not the moment the picture was taken.

A Python example: grasping a moving object

To see the effect without a GPU, we wrote a small simulation. An object rides a wobbly conveyor at about 0.1 m/s, the same speed VLALagBench used. A gripper runs at 50 Hz and gets chunks of 10 actions from a "policy" that sees the object's position and a noisy estimate of its speed. At a random moment the gripper closes, and it succeeds if it is within 1 cm of the object.

We compare three ways of running the same policy:

• sync: play the chunk, then stand still while the next one is computed
• async: ask early so the next chunk lands just in time, but plan from the time of the picture
• async + delay aware: ask early, and plan for the steps when the actions will run, the idea behind RTC

import numpy as np

# A gripper must close on an object riding a wobbly conveyor.
# The control loop runs at 50 Hz. Every call to the policy returns a chunk
# of 10 actions (0.2 s), but the call takes LATENCY seconds to come back.
# The policy sees the object's position and a noisy velocity estimate,
# and plans where the object will be for each step of its chunk.
DT, H, T_END = 0.02, 10, 3.0          # 50 Hz control, 10 step chunks, 3 s
TOL = 0.01                            # success = within 1 cm when it closes
STEPS = int(T_END / DT)


def object_path(rng):
    # speed about 0.1 m/s, with a slowly drifting wobble on top
    v = rng.choice([-1, 1]) * rng.uniform(0.08, 0.12)
    acc = np.zeros(STEPS + 50)
    for i in range(1, len(acc)):          # smooth random acceleration
        acc[i] = 0.9 * acc[i - 1] + rng.normal(0, 0.12)
    vel = v + np.cumsum(acc) * DT
    return np.cumsum(vel) * DT, vel


def plan(x, vel, rng, t_obs, t_start):
    # predict where the object will be at each step of the chunk
    v_est = vel[t_obs] + rng.normal(0, 0.01)
    steps = (t_start - t_obs) + np.arange(1, H + 1)
    return x[t_obs] + v_est * steps * DT


def run(latency, mode, rng):
    x, vel = object_path(rng)
    d = int(round(latency / DT))          # latency in control steps
    grasp = rng.integers(50, STEPS)        # the moment the gripper closes
    g, queue, t, pending = 0.0, [], 0, None
    while t <= grasp:
        if mode == "sync" and not queue:
            t_obs = t
            t += d                         # robot stands still while it waits
            queue = list(plan(x, vel, rng, t_obs, t_obs))
            if t > grasp:
                break
        if mode != "sync" and pending is None and len(queue) <= d:
            # ask for the next chunk early, so it lands as this one runs out
            start = t + len(queue) if mode == "async + delay aware" else t
            pending = (t + max(d, 1) * (d > 0), plan(x, vel, rng, t, start))
        if pending is not None and t >= pending[0]:
            queue, pending = list(pending[1]), None
        if queue:
            g = queue.pop(0)
        t += 1
    return abs(g - x[grasp]) < TOL


rng = np.random.default_rng(0)
modes = ["sync", "async", "async + delay aware"]
print(f"{'latency':>8} " + " ".join(f"{m:>20}" for m in modes))
for ms in [0, 50, 100, 200]:
    rates = [np.mean([run(ms / 1000, m, rng) for _ in range(2000)]) for m in modes]
    print(f"{ms:>6} ms " + " ".join(f"{r:>19.0%} " for r in rates))

Running it prints:

 latency                 sync                async  async + delay aware
     0 ms                 95%                  95%                  95% 
    50 ms                 84%                  89%                  92% 
   100 ms                 34%                  38%                  85% 
   200 ms                 15%                  20%                  64%

Up to about 40 ms, nothing much happens. The object moves only a few millimetres in that time, which fits inside the 1 cm tolerance.

Between 50 and 100 ms, sync falls off a cliff. Success drops from 84% to 34%, close to the halving VLALagBench saw on real policies.

Async only helps a little in our example. It removes the pauses, but every chunk is still aimed at where the object was. In our task the stale aim is the main error, so async gains only a few points. In VLALagBench's release task the pauses mattered more, and async gained over 30 points. Which effect dominates depends on the task.

Planning for the delay holds up. The same policy, told that its actions will start one latency later, keeps 85% at 100 ms and 64% at 200 ms. What remains is the honest cost of guessing further into the future with a noisy speed estimate.

This is a toy. A real VLA does not output a clean speed estimate, and RTC does its planning inside the diffusion process rather than with a formula. But the structure of the problem is the same.

What to check on your own robot

• Time the whole loop. Measure from camera exposure to motor command, not just the forward pass. Network hops and GPU queues often add more than the model.
• Test with moving things. A policy that looks fine on still objects can fall apart when the target drifts. Add a slow conveyor or a moving tray to your tests.
• Match chunk length to latency. Chunks shorter than the latency force pauses. Chunks much longer react slowly.
• Run async, then plan for the delay. Async removes the pauses for free. RTC style freezing or conditioning on delay fixes the stale aim.
• Split slow and fast when you can. If the scene's meaning changes slowly, a Leap-VLA style split lets a cheap model handle the motion.

If you are new to how these policies are trained, our guides on imitation learning, world models and reinforcement learning after imitation cover the background.

Frequently asked questions

Closing thoughts

For a few years the race in robot learning was about bigger models trained on more data. Those models are now good enough that their speed has become the limit. A model that reads the scene perfectly but answers 200 ms late will still drop the cup.

The good news is that the fixes are cheap. Run the model in parallel with the motion, and make sure every plan is aimed at the moment it will run. VLALagBench gives a way to measure how much latency a task can take, and RTC and Leap-VLA show two ways to stay inside that budget.

If you run the example, try making the object change speed more often, or giving the policy a better speed estimate, and watch how the gap between the three lines changes. We would like to hear what you find.

References

• Q. Wen, B. Xu and others, VLALagBench: Understanding When and Why VLA Inference Latency Matters, November 2026. arXiv:2611.03103
• L. Yang and others, Leap-VLA: High-Rate Robot Control from a Low-Rate VLM Backbone, October 2026. arXiv:2611.01011
• K. Black, M. Y. Galliker and S. Levine, Real-Time Execution of Action Chunking Flow Policies, 2025. arXiv:2506.07339
• Hugging Face, Asynchronous Robot Inference: Decoupling Action Prediction and Execution, July 2025. huggingface.co
• ArXivSignals, robotics papers of the week, November 2026. arxivsignals.io