PosterPDF Available
Teleoperation for Autonomous Driving Failures
Markus Hofbauer, Christopher Kuhn and Eckehard Steinbach
Chair of Media Technology
Technical University of Munich, Germany
e-mail: markus.hofbauer@tum.de, christopher.kuhn@tum.de,
eckehard.steinbach@tum.de
Motivation
Unavoidable failures in autonomous driving (AD) in challenging scenarios
Need for automatic detection and prediction of such scenarios
Teleoperation of autonomous vehicle to resolve failures
Need for low delay data transmission under every network condition
Concept
Figure 1: Schematic of teleoperation workflow
Car in AD mode encounters problematic scene and predicts its own failure
Transmission of teleoperation request and adaptive streaming of
necessary sensor information
Human teleoperator resolves situation and car resumes AD mode
Automatic and Predictive Detection
Machine-Learning-based analysis of entire AD system and subtasks
Predict failure of AD system based on input and previous performance
using introspection [1]:
Input AD system Success/
failure Labels
Data
Train introspection
model
Estimate uncertainty of individual tasks such as object detection via
Monte Carlo dropout [2]:
Input Trained neural
network
ndropout
masks
Sample s1
Sample sn
Uncertainty:
V ar[s1, ..., sn]
. . .
Estimate uncertainty using variance of deep ensembles [3]:
Input ninitializations nneural
networks
Prediction p1
Prediction pn
Uncertainty:
V ar[p1, ..., pn]
. . .
Simulated driving (CARLA simulator) to generate training data
BMW fleet data as real-life training data
Figure 2: Sample BMW camera image with highlighted objects
Driving Simulator
Figure 3: Sample CARLA scene with depth and segmentation
Open source autonomous driving simulator CARLA [4]
Deployment of autonomous driving models for data collection
Manual vehicle control through network emulator
Figure 4: Teleoperation workflow with CARLA driving simulator
Teleoperation Workstation
Figure 5: Teleoperation workstation with CARLA driving simulator
Teleoperation scenario evaluation in CARLA driving simulator
Visualization of multiple sensor information
Control command input via gaming wheel and pedals
Driver performance evaluation with predefined scenarios
Real-world teleoperation evaluation with
Custom model car
BMW teleoperation vehicle
Roadmap
Generation of a simulated autonomous driving failure data set
Setup of custom model car for real-work teleoperation
Adaptive streaming of sensor information
Deployment to real BMW teleoperation car
References
[1] S. Daftry et al., ”Introspective perception: Learning to predict failures in vision systems”,
IEEE IROS 2016.
[2] Y. Gal and Z. Ghahramani, ”Dropout as bayesian approximation: Representing model
uncertainty in deep learning”, International Conference on Machine Learning 2016.
[3] B. Lakshminarayanan, ”Simple and scalable predictive uncertainty estimation using deep
ensembles”, Conference on Neural Information Processing Systems 2017.
[4] A. Dosovitskiy et al., ”CARLA: An Open Urban Driving Simulator“, Conference on Robot
Learning 2017.
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
As robots aspire for long-term autonomous operations in complex dynamic environments, the ability to reliably take mission-critical decisions in ambiguous situations becomes critical. This motivates the need to build systems that have situational awareness to assess how qualified they are at that moment to make a decision. We call this self-evaluating capability as introspection. In this paper, we take a small step in this direction and propose a generic framework for introspective behavior in perception systems. Our goal is to learn a model to reliably predict failures in a given system, with respect to a task, directly from input sensor data. We present this in the context of vision-based autonomous MAV flight in outdoor natural environments, and show that it effectively handles uncertain situations.
Dropout as bayesian approximation: Representing model uncertainty in deep learning
  • Y Gal
  • Z Ghahramani
Y. Gal and Z. Ghahramani, "Dropout as bayesian approximation: Representing model uncertainty in deep learning", International Conference on Machine Learning 2016.
Simple and scalable predictive uncertainty estimation using deep ensembles
  • B Lakshminarayanan
B. Lakshminarayanan, "Simple and scalable predictive uncertainty estimation using deep ensembles", Conference on Neural Information Processing Systems 2017.
CARLA: An Open Urban Driving Simulator
  • A Dosovitskiy
A. Dosovitskiy et al., "CARLA: An Open Urban Driving Simulator", Conference on Robot Learning 2017.