Main Hypothesis
Shared autonomy can improve human performance without any assumptions about:
dynamics,
the human’s policy,
the nature of the goal.
CSC2626 Imitation Learning for Robotics
Week 10: Shared Autonomy and Human-in-the-Loop Learning
Shared Autonomy via Deep Reinforcement Learning
Siddharth Reddy, Anca Dragan, Sergey Levine UC Berkeley
Presented by Ioan Andrei Bârsan on February 22, 2019
iab@cs.toronto.edu
Key Question
How can a robot collaborating with a human infer the human’s goals with as few assumptions as possible?
Motivation
• Hard: Actuating a robot with many DoF and/or unfamiliar dynamics.
• Hard: Specifying a goal formally (e.g., coordinates).
• Easy: Demonstrating the goal indirectly.
• …let the machine figure out what I want!
Image source: “Multihierarchical Interactive Task Planning. Application to Mobile Robotics” Galindo et al., 2008
Motivation: Unknown Dynamics are Hard for Humans
It can get even worse than Lunar Lander…
https://www.foddy.net/Athletics.html or Google “qwop”
Formulation
\[ \underbrace{R_{\text{feedback}}(s, a, s')}_{\text{unknown, but observed}} \]
Needs
virtual
“user”!
• The authors introduce three variants of their method:
Known goal space, known user policy.
Known goal space, unknown user policy.
Unknown goal space, unknown user policy.
The Method (Continued)
The Method (Continued)
Input to RL Agent
Real-World Experiments
Goal: Land drone on pad facing a certain way.
Pilot: Human, knows target orientation.
Copilot: Our Agent, knows where pad is, but not target orientation.
Real-World Results
Important observation: Only n = 4 humans in drone study.
Experimental Results: Assumptions
Higher alpha means we take any action. α = 1.0 means we ignore the pilot.
Experimented in virtual environment.
Thanks for your attention!
Q&A, if time permits it.
Project website: https://sites.google.com/view/deep-assist
Video of computer-assisted human piloting the lander.
Teleoperation
Noisy, insufficient degrees of freedom, tedious
Image credit: Javdani RSS2015 talk
Shared Autonomy
User Input
+
Autonomous Assistance
=
Achieve Goal
Shared Autonomy
User Input
+
Autonomous Assistance
=
Achieve Goal
Shared Autonomy
Predict goal Assist for single goal
[Dragan and Srinivasa 13]
[Kofman et al. 05]
[Kragic et al. 05]
[Yu et al. 05]
[McMullen et al. 14]
…
+
Autonomous Assistance
=
Achieve Goal
Shared Autonomy
Predict goal Assist for single goal
[Dragan and Srinivasa 13]
[Kofman et al. 05]
[Kragic et al. 05]
[Yu et al. 05]
[McMullen et al. 14]
…
Predict goal distribution Assist for distribution
[Hauser 13]
This work!
+
Autonomous Assistance
=
Achieve Goal
Method
- System dynamics: \(x’ = T(x, a)\)
Method
System dynamics: \(x' = T(x, a)\)
User (MDP) as \((X, U, T, C_g^{\text{usr}})\)
- User policy: \(\pi_g^{\text{usr}}(x) = p(u|x, g)\)
- MaxEnt IOC: \(C_g^{\text{usr}} : X \times U \to \mathcal{R}\)
Method
System dynamics: \(x' = T(x, a)\)
User (MDP) as \((X, U, T, C_g^{\text{usr}})\)
- User policy: \(\pi_g^{\text{usr}}(x) = p(u|x, g)\)
- MaxEnt IOC: \(C_g^{\text{usr}} : X \times U \to \mathcal{R}\)
System (POMDP) as \((S, A, T, C^{\text{rob}}, U, \Omega)\)
- Uncertainty over user’s goal
- System state: \(s = X \times G\)
- Observation: user inputs \(U\)
- Observation model \(\Omega\)
\(p(g|\xi^{0 \to t}) = \frac{p(\xi^{0 \to t}|g)p(g)}{\sum_{g'} p(\xi^{0 \to t}|g')p(g')}\)
- Cost function \(C^{\text{rob}} : S \times A \times U \to \mathcal{R}\)
Hindsight Optimization
MDP solution:
\[V^{\pi^r}(s) = \mathbb{E}\left[\sum_t C^r(s_t, u_t, a_t) \mid s_0 = s\right]\]
\[V^*(s) = \min_{\pi^r} V^{\pi^r}(s)\]
POMDP solution:
\[V^{\pi^r}(b) = \mathbb{E}\left[\sum_t C^r(s_t, u_t, a_t) \mid b_0 = b\right]\]
\[V^*(b) = \min_{\pi^r} V^{\pi^r}(b)\]
HOP approximation:
\[V^{\text{HS}}(b) = \mathbb{E}_b\left[\min_{\pi^r} V^{\pi^r}(s)\right]\]
\[= \mathbb{E}_g[V_g(x)]\]
Deterministic
problem for
each future
Results (video)
Results
Compare with method that predicts one goal, the proposed method has:
Faster execution time
Fewer user inputs
User Study
Recap: Forward Kinematics (FK)
Forward Kinematics
A common robotic skeleton is a tree of rigid bones
The relative Euler angles of all the joints determine the end-effector
- End-effectors? A tool that’s connected to the end of a robot arm
![]()
![]()
![]()
materials from https://github.com/alecjacobson/computer-graphics-csc418
Recap: Inverse Kinematics (IK)
Inverse Kinematics
We formulate the inverse kinematics function as: \(\Theta = IK(p)\) , which can be easily written in an analytic form for a simple tree skeleton.
Pose contains velocity?
Hard to find feasible state space?
In reality, IK is often treated as an optimization problem \[ \chi_p(\Theta) = \| p_g - FK(\Theta) \|_2 \]
Imitation Learning
Self-collision
Imitation Learning
Discontinuities
Imitation Learning
Goal mistracking
Imitation Learning
Unpredictable behaviors 
Relaxed IK
Self-collision loss
Common approach: very slow
Relaxed IK:
Approximate how imminent the robot is to a collision state
Using simulated data to train a network to predict the distances between links
\[ \text{col}(\Theta) = \sum_{i,j} b \cdot \exp\left( -\frac{\text{dis}(l_i, l_j)^2}{2c^2} \right) \]
![]()
Relaxed IK
Pros
- Much faster and smoother performance
- combining neural network and traditional robotics
- Data driven, less human-engineering
- novel singularity metric
- Easy to deploy
- sim2Real
