End to End Learning for Self-Driving Cars, Bojarski et al, 2016
What has changed?
“Our collected data is labeled with road type, weather condition, and the driver’s activity (staying in a lane, switching lanes, turning, and so forth).”
End to End Learning for Self-Driving Cars, Bojarski et al, 2016
What has changed?
How much has changed?
A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots, Giusti et al., 2016
Learning Monocular Reactive UAV Control in Cluttered Natural Environments, Ross et al, 2013
DAgger: Assumptions for theoretical guarantees
Strongly convex loss
No-regret online learner
(Ross & Gordon & Bagnell, 2011): DAgger, or Dataset Aggregation
• Imitation learning as interactive supervision
• Aggregate training data from expert with test data from execution
Imitation Learning via DAgger
Train/test data are not i.i.d.
If expected training error on aggr. dataset is \(\epsilon\)
Expected test error after T decisions is
\[
O(T\epsilon)
\]
Errors do not compound
Supervised Learning
Assumes train/test data are i.i.d.
If expected training error is \(\epsilon\)
Expected test error after T decisions
\[
T\epsilon
\]
Errors are independent
No-Regret Online Learners
Intuition: No matter what the distribution of input data, your online policy/classifier will do asymptotically as well as the best-in-hindsight policy/classifier.
\(\qquad \qquad \quad \big\uparrow\)
Policy has access to
data up to round N
\(\qquad \quad \big\uparrow\)
Policy has access to
data up to round i
No-regret:\(\lim_{N \to \infty} r_N = 0\)
Another way to say this: a no-regret online algorithm is one that outputs a sequence of policies \(\pi_1, \ldots, \pi_N\) such that the average loss with respect to the best-in-hindsight policy goes to 0 as \(N \to \infty\)
DAgger is a no-regret online learning algorithm
No-Regret Online Learners
Intuition: No matter what the distribution of input data, your online policy/classifier will do asymptotically as well as the best-in-hindsight policy/classifier.
Imitation via multi-modal generative models (Implicit Behavioral Cloning)
Implicit Behavioral Cloning (Florence et al, CoRL 2021)
Explicit policy: train a network to directly regress \(\hat{a} = F_\theta(o)\) (or a simple Gaussian/mixture head), a continuous function, so it cannot represent discontinuous or multi-valued action targets
Implicit policy: train an energy-based model \(E_\theta(o,a)\) with an InfoNCE-style contrastive loss (classify the demonstrated action against sampled negatives), then act via \(\hat{a} = \arg\min_a E_\theta(o,a)\), solved with derivative-free sampling/optimization at inference time
Composing energy landscapes lets implicit models fit sharp discontinuities and multimodal action distributions that explicit regression smooths over
Imitation via multi-modal generative models (Implicit Behavioral Cloning)
On a toy pixel-coordinate regression task, the implicit (EBM) model stays accurate even for query points outside the convex hull of the training examples, while the explicit (MSE) model’s error blows up there
On real-robot precision tasks (tight insertion, sorting, pushing) implicit policies trained with the same architecture/data as explicit baselines achieve substantially higher success
Same recipe generalizes to image observations and high-dimensional action spaces, without needing an explicit likelihood or partition function
Imitation via multi-modal generative models (Diffusion Policy)
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (Chi et al, RSS 2023)
Explicit policies (\(\hat a = F_\theta(o)\)): mode-average multimodal demos, e.g. “left or right” collapses to “straight through”
Implicit policies (\(\arg\min_a E_\theta(o,a)\)): multimodal, but the energy landscape is unstable/hard to train
Diffusion policy: iteratively denoises \(\hat a\) along \(\nabla E(a)\), same expressiveness, stable DDPM-style training
Imitation via multi-modal generative models (Diffusion Policy)
Conditions on a short history of observations \(O_t\), denoises a whole action sequence\(A_t\) (not a single action) — action chunking again reduces compounding error and captures temporally consistent multimodal behavior
Executes a receding horizon: run a few actions from the predicted sequence, then re-observe and re-plan
Appendix 1: Why do behavioral cloning errors accumulate quadratically?
Efficient reductions for imitation learning. Ross & Bagnell, AISTATS 2010.
Appendix 2: Types of Uncertainty & Query-Efficient Imitation
Let’s revisit the two main ideas from query-efficient imitation:
1. DropoutDAgger:
Keep an ensemble of learner policies, and only query the expert when they significantly disagree
2. SHIV, SafeDagger, MMD-IL:
(Roughly) Query expert only if input is too close to the decision boundary of the learner’s policy
Need to review a few concepts about different types of uncertainty.
