CSC2626 Imitation Learning for Robotics
Week 3: Offline/Batch Reinforcement Learning
Today’s agenda
• Reinforcement Learning Terminology
• Distribution Shift in Offline RL
• Offline RL with Policy Constraints
• Offline RL with Conservative Q-Estimates
Policy Optimization vs Value Function Estimation
Credit: John Schulman
On-policy vs Off-policy Methods
On-policy RL methods: improve the policy that acts on the environment using data collected from that same policy
Off-policy RL methods: improve the policy that acts on the environment using data collected from any policy
Batch (Offline) vs Online Methods
Online RL methods: Can collect data over multiple rounds. Data distribution changes over time.
Batch/Offline RL methods: Can collect data only once from any policy. Data distribution is stationary.
Batch (Offline) vs Online Methods
\(\mathbf{s} \in \mathcal{S}\) – discrete or continuous state
\(\mathbf{a} \in \mathcal{A}\) – discrete or continuous action
\(\tau = \{s_0, a_0, s_1, a_1, \ldots, s_T, a_T\}\) - trajectory
\(\underbrace{\pi(s_0, a_0, \ldots, s_T, a_T)}_{\pi(\tau)} = p(s_0) \prod_{t=0}^{T} \pi(a_t | s_t) p(s_{t+1} | s_t, a_t)\)
\(d_t^{\pi}(s_t)\) – state marginal of \(\pi(\tau)\) at \(t\)
\(d^{\pi}(s) = \frac{1}{1-\gamma} \sum_{t=0}^{T} \gamma^t d_t^{\pi}(s_t) \quad \text{visitation frequency}\)
\(Q^{\pi}(s_t, a_t) = r(s_t, a_t) + \gamma \mathbb{E}_{s_{t+1} \sim p(s_{t+1} | s_t, a_t), a_{t+1} \sim \pi(a_{t+1} | s_{t+1})} \left[Q^{\pi}(s_{t+1}, a_{t+1})\right]\)
\(V^{\pi}(s_t) = \mathbb{E}_{a_t \sim \pi(a_t | s_t)} \left[Q^{\pi}(s_t, a_t)\right]\)
\(A^{\pi}(s_t, a_t) = Q^{\pi}(s_t, a_t) - V^{\pi}(s_t) \quad \text{(advantage function)}\)
On-policy actor-critic with function approximation
- update \(Q_\phi\) to decrease \(E_{s \sim d^{\pi_\theta}(s), a \sim \pi_\theta(a|s)} \left[\left(Q_\phi(s,a) - (r(s,a) + \gamma E_{\pi_\theta}[Q_\phi(s',a')])\right)^2\right]\)
- update \(\pi_\theta\) to increase \(E_{s \sim d^{\pi_\theta}(s), a \sim \pi_\theta(a|s)} [Q_\phi(s,a)]\)
Off-policy actor-critic with function approximation
- update \(Q_\phi\) to decrease \(E_{s \sim d^{\pi_\beta}(s), a \sim \pi_\beta(a|s)} \left[\left(Q_\phi(s,a) - (r(s,a) + \gamma E_{\pi_\theta}[Q_\phi(s',a')])\right)^2\right]\)
- update \(\pi_\theta\) to increase \(E_{s \sim d^{\pi_\beta}(s), a \sim \pi_\theta(a|s)} [Q_\phi(s,a)]\)
Q-Learning/Fitted Q-Iteration with function approximation (off-policy)
- update \(Q_\phi\) to decrease \(E_{s \sim d^{\pi_\beta}(s), a \sim \pi_\beta(a|s)} \left[\left(Q_\phi(s,a) - (r(s,a) + \gamma E_{\pi_\theta}[Q_\phi(s',a')])\right)^2\right]\)
- choose \(\pi\) according to: \(\pi(a_t | s_t) = \begin{cases} 1 & \text{if } a_t = \arg\max_{a_t} Q_\phi(s_t, a_t) \\ 0 & \text{otherwise} \end{cases}\)
QT-Opt (roughly: continuous-action Q-Learning)
Policy evaluation: you want a value function close to the true one, but we use the Bellman equation as a consistency check / error.
This paper shows that low Bellman error can be a poor indicator of how accurate the value function actually is.
- Error cancellation: the same value function appears on both sides of the Bellman equation, so errors can cancel between them
- With a finite dataset, infinitely many suboptimal value functions satisfy it exactly, so zero Bellman error is compatible with arbitrarily large value error
Logistic Q-Learning
The usual squared Bellman error used for policy evaluation lacks strong theoretical justification
Proposes a convex loss function for policy evaluation, derived from regularized linear programming, as a theoretically principled alternative
Yields QREPS: an algorithm combining ideas from Relative Entropy Policy Search with a Q-function, enabling an efficient, exact model-free implementation
Provides an error-propagation analysis connecting the quality of each update to final policy performance
When should we prefer offline RL over behavioral cloning (BC)?
Given the same expert demonstrations, is BC always the right choice?
Claim: policies trained on sufficiently noisy/suboptimal data can outperform BC trained on expert data, especially on long-horizon tasks
Offline RL has the biggest edge when rewards are sparse and the data is noisy/suboptimal
On critical/bottleneck states, there’s no clear winner
Takeaway: the best algorithm depends on the structure of the problem and data, not just on data quality
Today’s agenda
• Reinforcement Learning Terminology
• Distribution Shift in Offline RL
• Offline RL with Policy Constraints
• Offline RL with Conservative Q-Estimates
Today’s agenda
• Reinforcement Learning Terminology
• Distribution Shift in Offline RL
• Offline RL with Policy Constraints
• Offline RL with Conservative Q-Estimates
Addressing Distribution Shift via Pessimism
Different Types of Policy Constraint Methods
Note: KL divergence is not symmetric
How should we evaluate offline RL methods
Evaluating Offline RL
Evaluating Offline RL
Standardized Benchmark for Offline RL
Value Function Regularization for Offline RL
Learning Lower-Bounded Q-values
A Tighter Lower Bound
Practical CQL Algorithm
Today’s agenda
• Reinforcement Learning Terminology
• Distribution Shift in Offline RL
• Offline RL with Policy Constraints
• Offline RL with Conservative Q-Estimates
The need for safe exploration in RL for robotics
When applying RL to robotics we need to guarantee that the algorithm will not visit unsafe states very often during learning.
The need for safe exploration in RL for robotics
When applying RL to robotics we need to guarantee that the algorithm will not visit unsafe states very often during learning.
The need for safe exploration in RL for robotics
When applying RL to robotics we need to guarantee that the algorithm will not visit unsafe states very often during learning.
Our proposed method:
The learned policy should be safe at each iteration, not just when optimization has converged
Results: fewer accidents
Constrained Policy Optimization, https://arxiv.org/abs/1705.10528, Achiam, Held, Tamam, Abbeel
Results: task value vs safety
