DAgger Explained: How Dataset Aggregation Fixes Behavior Cloning, with Python Code

Picture a new driver who learned only by watching an instructor. The instructor always kept the car in the middle of the lane, so the student copied that well. Then one morning the student drifts half a metre to the left. Nobody ever showed them what to do from there. They guess, the guess is a little wrong, and the next moment is worse than the last.

That small story is the main problem with behavior cloning, the simplest kind of imitation learning. In 2011 Stephane Ross, Geoffrey Gordon and Drew Bagnell published a short fix for it called DAgger, short for Dataset Aggregation. It is one of the papers I am studying this semester in my imitation learning course, and fifteen years later it still shows up in robot learning work on cars, drones and arms.

This post walks through the paper: what goes wrong with plain copying, how DAgger repairs it, the guarantee behind it, what the experiments showed, and a small Python car you can run to watch both methods side by side. If imitation learning is new to you, start with my overview of imitation learning and come back.

The paper at a glance

Title	A Reduction of Imitation Learning and Structured Prediction to No Regret Online Learning
Authors	Stephane Ross, Geoffrey J. Gordon, J. Andrew Bagnell (Carnegie Mellon University)
Venue	AISTATS 2011, JMLR Workshop and Conference Proceedings, volume 15
Link	arXiv:1011.0686
Main idea	Let the learner drive, ask the expert what it should have done in the states the learner reached, add those labels to the data, and train again

Behavior cloning in one paragraph

Behavior cloning treats driving, flying or grasping as plain supervised learning. You record an expert, collect pairs of what the robot saw and what the expert did, and train a model to map one to the other. It is simple, it needs no reward function, and with enough good data it works surprisingly often. Many modern robot policies, including ACT and Diffusion Policy, are behavior cloning at heart.

The catch sits in a single assumption. Supervised learning expects the test data to come from the same distribution as the training data. A robot policy breaks that rule the moment it acts, since its own actions decide which situations it sees next.

Covariate shift: why copying drifts

The expert’s demonstrations only cover the states the expert visits. A good expert stays in a narrow band of safe, tidy states. The learner is never perfect, so after a few small mistakes it reaches states the expert never visited. In those states the training data says nothing useful, the learner’s mistakes grow, and the states move further from anything it has seen. This mismatch between training and test states is called covariate shift.

Ross and Bagnell put a number on it in an earlier paper, and the DAgger paper restates it as Theorem 2.1. Let T be the number of steps in a task, and let ε be the learner’s error rate on states drawn from the expert’s distribution. Then the cost of the cloned policy π can be as bad as

The cost grows with the square of the horizon. Double the length of the task and the worst case gets four times worse. The paper notes this bound is tight: there are problems where behavior cloning really does that badly.

A quick way to feel the T² is to think about when the first mistake happens. With a 1 percent error rate, a 100 step task has a good chance of one mistake somewhere. After that mistake the learner is off the expert’s path, so every remaining step can go wrong too. Early mistakes are the expensive ones, and the longer the task, the more steps they can spoil.

How DAgger works, step by step

The fix is easy to state. If the learner fails in the states it visits, collect training data in the states it visits.

1. Start with an empty dataset D and any first policy.
2. Run the current policy in the world for T steps. In early rounds you can let the expert take control with probability β, so the robot does not wreck itself.
3. For every state the robot visited, ask the expert which action it would have taken there. The expert labels; it does not need to drive.
4. Add these labelled states to D. Nothing is thrown away, which is where the name Dataset Aggregation comes from.
5. Train a new policy on all of D.
6. Repeat for N rounds and keep the policy that scores best on validation.

Step 3 is the clever part. The expert gets asked about awkward situations, the car half a metre off centre or the drone too close to a tree, and those answers are exactly the recovery behavior the original demonstrations never contained.

Choosing β

β controls how often the expert drives during data collection in round i. The paper suggests β to start at 1 and decay, for example β equal to p raised to i minus 1, with p set to 0.5 or 0.9. The only formal requirement is that the average of the β values goes to zero over the rounds. The authors report that the simplest choice, the expert drives in round one and never again, often performs best in practice. The code below uses that choice.

Why it works: the no regret view

The paper’s title promises a reduction to no regret online learning, and that is the theory behind DAgger.

In online learning, a learner picks a model each round, then sees a new loss and pays it. A no regret algorithm is one whose average loss over N rounds approaches the loss of the best single model in hindsight. Follow the Leader, which retrains on all data seen so far, is one such algorithm when the loss is strongly convex.

DAgger is Follow the Leader in disguise. Each round, the loss is the learner’s error on the states its latest policy visited. Training on the union of all those datasets is exactly retraining on every past loss. So DAgger inherits the no regret guarantee, and with it a bound on how the final policy performs on its own state distribution. Theorem 3.2 states that after enough rounds (on the order of uT) there is a policy in the sequence with

Here ε_N is the best training loss achievable on the aggregated data, and u bounds how much one wrong action can raise the cost to go. When the task forgives small mistakes, u is small and the cost grows linearly in T, not quadratically. Theorem 3.4 gives the same shape with a high probability statement when each round collects only a finite number of trajectories.

What the experiments showed

The paper tests DAgger on two games and one sequence labelling problem. All three compare it against plain supervised training, the authors’ earlier SMILe method and SEARN.

Task	Setup	Result
Super Tux Kart, a 3D racing game	Steering from screen images, linear controller, about 1,000 labelled frames per round, 20 rounds	Behavior cloning did not get better with more data. SMILe still fell off the track about twice per lap after 20 rounds. DAgger stopped falling after 15 rounds
Super Mario Bros.	4 binary actions learned with linear SVMs, 5,000 labels per round, 20 rounds	DAgger covered about 3,000 units per stage (2,980 to 3,030 depending on β), out of roughly 4,200 to 4,300 for a full stage, ahead of SMILe, SEARN and plain supervised training
Handwriting recognition	Label each character of a word using the previous predicted character	85.5 percent character accuracy against 83.6 percent for the supervised version and 82 percent with no structure

The racing result is the one people remember. More expert laps did not help the copy at all, while a handful of rounds where the learner drove and the expert corrected fixed the problem.

Python example: behavior cloning vs DAgger on a lane keeping car

Here is a small experiment you can run on a laptop with only NumPy. A car drives along a winding road one metre either side of the centre line. The expert is a simple controller that follows the bend and steers back toward the centre. Both learners use the same model, a k nearest neighbours lookup that copies the expert’s action in the five most similar recorded situations. The only difference is where the training data comes from.

The code

import numpy as np

# ---------------------------------------------------------------- the road and the car
DT, SPEED, ROAD_HALF_WIDTH = 0.1, 1.0, 1.0


def curvature(s):
    """How sharply the road bends at distance s along it."""
    return 0.35 * np.sin(0.21 * s) + 0.25 * np.sin(0.07 * s + 1.0)


def step(state, steer, rng, noise=0.05):
    """Move the car one time step. state = (s, y, theta):
    s distance along the road, y offset from the centre line, theta heading error."""
    s, y, th = state
    steer = np.clip(steer + rng.normal(0, noise), -1.5, 1.5)   # a little actuator noise
    k = curvature(s)
    s += DT * SPEED * np.cos(th)
    y += DT * SPEED * np.sin(th)
    th += DT * SPEED * (steer - k)
    return np.array([s, y, th])


def observe(state):
    """What the policy sees: offset, heading error and the bend just ahead."""
    s, y, th = state
    return np.array([y, th, curvature(s), curvature(s + 1.0)])


def expert(state):
    """The teacher: follow the bend and steer back toward the centre."""
    s, y, th = state
    return curvature(s) - 1.5 * y - 2.0 * th


# ---------------------------------------------------------------- the learner
class NearestNeighbourPolicy:
    """Copies the action of the closest recorded situation (k nearest neighbours)."""

    def __init__(self, k=5):
        self.k = k

    def fit(self, obs, actions):
        self.obs, self.actions = np.asarray(obs), np.asarray(actions)
        self.scale = self.obs.std(axis=0) + 1e-6
        return self

    def __call__(self, state):
        d = np.linalg.norm((self.obs - observe(state)) / self.scale, axis=1)
        nearest = np.argpartition(d, self.k)[: self.k]
        return self.actions[nearest].mean()


# ---------------------------------------------------------------- rollouts
def rollout(policy, steps, rng, start=0.0, beta=0.0, label=False):
    """Drive for `steps` steps. With probability beta the expert drives each step.
    Returns visited observations, the expert's labels for them, and the distance
    covered before leaving the road (or the full distance)."""
    state = np.array([start, 0.0, 0.0])
    obs, labels = [], []
    for t in range(steps):
        if label:
            obs.append(observe(state))
            labels.append(expert(state))
        act = expert(state) if rng.random() < beta else policy(state)
        state = step(state, act, rng)
        if abs(state[1]) > ROAD_HALF_WIDTH:
            return obs, labels, t + 1, True
    return obs, labels, steps, False


def evaluate(policy, steps, rng, laps=20):
    crashes, survived = 0, []
    for lap in range(laps):
        _, _, n, crashed = rollout(policy, steps, rng, start=rng.uniform(0, 200))
        crashes += crashed
        survived.append(n)
    return crashes / laps, np.mean(survived) / steps


def behaviour_cloning(n_laps, steps, rng):
    obs, act = [], []
    for _ in range(n_laps):
        o, a, _, _ = rollout(expert, steps, rng, start=rng.uniform(0, 200), beta=1.0, label=True)
        obs += o
        act += a
    return NearestNeighbourPolicy().fit(obs, act)


def dagger(n_iters, steps, rng, history=None):
    obs, act = [], []
    policy = None
    for i in range(n_iters):
        beta = 1.0 if i == 0 else 0.0          # the paper's beta_i = I(i = 1)
        o, a, _, _ = rollout(policy, steps, rng, start=rng.uniform(0, 200), beta=beta, label=True)
        obs += o                               # every visited state, labelled by the expert
        act += a
        policy = NearestNeighbourPolicy().fit(obs, act)
        if history is not None:
            history.append(evaluate(policy, steps, rng))
    return policy


if __name__ == "__main__":
    STEPS = 600          # one lap: 60 seconds of driving
    ITERS = 10
    rng = np.random.default_rng(0)

    print("laps  BC crash rate  BC road kept   DAgger crash rate  DAgger road kept")
    hist = []
    dagger(ITERS, STEPS, np.random.default_rng(1), history=hist)
    for n in range(1, ITERS + 1):
        bc = behaviour_cloning(n, STEPS, np.random.default_rng(100 + n))
        bc_crash, bc_kept = evaluate(bc, STEPS, np.random.default_rng(7))
        dg_crash, dg_kept = hist[n - 1]
        print(f"{n:4d}  {bc_crash:13.0%}  {bc_kept:12.0%}   {dg_crash:17.0%}  {dg_kept:16.0%}")

The DAgger loop is the whole idea in six lines. In round one the expert drives. From round two the learner drives, and every state it reaches gets the expert’s label before it joins the dataset.

Reading the result

Each row trains both learners on the same number of 60 second laps, then tests each on 20 fresh laps that start at random points on the road.

laps  BC crash rate  BC road kept   DAgger crash rate  DAgger road kept
   1           100%           15%                100%               15%
   2           100%           15%                  0%              100%
   3           100%           15%                  0%              100%
   4           100%           13%                  0%              100%
   5           100%           25%                  0%              100%
   6           100%           26%                  0%              100%
   7           100%           28%                  0%              100%
   8           100%           18%                  0%              100%
   9           100%           20%                  0%              100%
  10            95%           41%                  0%              100%

After one lap the two methods are identical, since DAgger’s first round is pure expert data. From the second round on, DAgger never leaves the road. Behavior cloning with ten full laps of expert driving, 6,000 labelled steps, still leaves the road on 19 laps out of 20. DAgger used 5,241 labelled steps after ten rounds, fewer than behavior cloning, since its early rounds ended when the car left the road.

More expert data barely helps the copy. The expert never strays more than about 2 cm from the centre, so ten laps of it are ten laps of nearly the same tidy states. The data DAgger adds is the data that matters: the car 20 or 30 cm off line, with the expert saying how to get back.

The horizon chart shows the compounding effect directly. On 5 second drives the cloned policy fails 10 percent of the time. On 40 second drives it fails 88 percent of the time. The model did not change between those bars; only the number of steps over which its small mistakes could pile up.

One fair caveat: this is a toy. The expert is a clean controller with no noise in its labels, and k nearest neighbours has no ability to generalise outside the data it has stored, which makes covariate shift especially visible. A neural network would extrapolate a bit better, but the shape of the result holds on real systems, as the racing game experiment showed.

What DAgger asks of you in practice

DAgger trades a data problem for a people problem. The expert has to label states the learner chose, often awkward ones, round after round. That raises a few practical questions that later papers took on.

• Labelling cost. A program expert, like the controller above or a planner with access to the full map, can label millions of states for free. A human expert labelling video frames after the fact is slow and finds it hard to say what they would have done at a moment they did not drive.
• Safety during collection. Letting a half trained policy drive a real robot can damage it. HG DAgger (Kelly et al., 2019) lets a human take over only when the robot looks unsafe and learns from those takeovers. SafeDAgger (Zhang and Cho, 2017) trains a second network to predict when the learner is about to go wrong.
• Noise instead of queries. DART (Laskey et al., 2017) injects noise into the expert’s own demonstrations, so the expert shows recoveries from small disturbances without ever handing over control.
• Validation. The guarantee is for the best policy in the sequence, not the last one, so keep every round’s model and pick on held out laps.

DAgger in robots today

The best known robot follow up came from the same group. In 2013 Ross and colleagues trained a small quadrotor to fly through a forest from a single camera using DAgger, with a human pilot labelling the flights afterwards. That paper is a direct ancestor of the learned collision free flight used on drones today, and it is one reason I care about it for Veg, my SLAM based drone project.

The pattern has spread well beyond that. Self driving research uses DAgger style training in simulators, where a privileged planner labels every state the learning car visits. Large robot learning pipelines collect human corrections on top of autonomous runs, which is DAgger with a friendlier interface. Whenever you see "on policy data with expert labels", the idea traces back to this 2011 paper.

Frequently asked questions

Closing thoughts

DAgger fits on an index card: let the learner drive, ask the expert, keep all the data, retrain. Its lasting value is the diagnosis behind it. A policy has to be trained on the situations it creates for itself, and copying a perfect teacher does not give you those.

If you try the car example, change the expert’s gains or the road’s curvature and watch how long behavior cloning survives. Then try it on your own robot task. I would like to hear where DAgger worked for you and where labelling became the bottleneck.

References

• S. Ross, G. J. Gordon and J. A. Bagnell, A reduction of imitation learning and structured prediction to no regret online learning, AISTATS 2011. arXiv:1011.0686
• S. Ross and J. A. Bagnell, Efficient reductions for imitation learning, AISTATS 2010.
• S. Ross, N. Melik Barkhudarov, K. S. Shankar, A. Wendel, D. Dey, J. A. Bagnell and M. Hebert, Learning monocular reactive UAV control in cluttered natural environments, ICRA 2013.
• M. Kelly, C. Sidrane, K. Driggs Campbell and M. J. Kochenderfer, HG DAgger: interactive imitation learning with human experts, ICRA 2019.
• J. Zhang and K. Cho, Query efficient imitation learning for end to end simulated driving (SafeDAgger), AAAI 2017.
• M. Laskey, J. Lee, R. Fox, A. Dragan and K. Goldberg, DART: noise injection for robust imitation learning, CoRL 2017.