Content uploaded by Markus Hofbauer
Author content
All content in this area was uploaded by Markus Hofbauer on May 11, 2020
Content may be subject to copyright.
Teleoperation for Autonomous Driving Failures
Markus Hofbauer, Christopher Kuhn and Eckehard Steinbach
Chair of Media Technology
Technical University of Munich, Germany
e-mail: markus.hofbauer@tum.de, christopher.kuhn@tum.de,
eckehard.steinbach@tum.de
Motivation
•Unavoidable failures in autonomous driving (AD) in challenging scenarios
•Need for automatic detection and prediction of such scenarios
•Teleoperation of autonomous vehicle to resolve failures
•Need for low delay data transmission under every network condition
Concept
Figure 1: Schematic of teleoperation workflow
•Car in AD mode encounters problematic scene and predicts its own failure
•Transmission of teleoperation request and adaptive streaming of
necessary sensor information
•Human teleoperator resolves situation and car resumes AD mode
Automatic and Predictive Detection
•Machine-Learning-based analysis of entire AD system and subtasks
–Predict failure of AD system based on input and previous performance
using introspection [1]:
Input AD system Success/
failure Labels
Data
Train introspection
model
–Estimate uncertainty of individual tasks such as object detection via
Monte Carlo dropout [2]:
Input Trained neural
network
ndropout
masks
Sample s1
Sample sn
Uncertainty:
V ar[s1, ..., sn]
. . .
–Estimate uncertainty using variance of deep ensembles [3]:
Input ninitializations nneural
networks
Prediction p1
Prediction pn
Uncertainty:
V ar[p1, ..., pn]
. . .
•Simulated driving (CARLA simulator) to generate training data
•BMW fleet data as real-life training data
Figure 2: Sample BMW camera image with highlighted objects
Driving Simulator
Figure 3: Sample CARLA scene with depth and segmentation
•Open source autonomous driving simulator CARLA [4]
•Deployment of autonomous driving models for data collection
•Manual vehicle control through network emulator
Figure 4: Teleoperation workflow with CARLA driving simulator
Teleoperation Workstation
Figure 5: Teleoperation workstation with CARLA driving simulator
•Teleoperation scenario evaluation in CARLA driving simulator
–Visualization of multiple sensor information
–Control command input via gaming wheel and pedals
–Driver performance evaluation with predefined scenarios
•Real-world teleoperation evaluation with
–Custom model car
–BMW teleoperation vehicle
Roadmap
•Generation of a simulated autonomous driving failure data set
•Setup of custom model car for real-work teleoperation
•Adaptive streaming of sensor information
•Deployment to real BMW teleoperation car
References
[1] S. Daftry et al., ”Introspective perception: Learning to predict failures in vision systems”,
IEEE IROS 2016.
[2] Y. Gal and Z. Ghahramani, ”Dropout as bayesian approximation: Representing model
uncertainty in deep learning”, International Conference on Machine Learning 2016.
[3] B. Lakshminarayanan, ”Simple and scalable predictive uncertainty estimation using deep
ensembles”, Conference on Neural Information Processing Systems 2017.
[4] A. Dosovitskiy et al., ”CARLA: An Open Urban Driving Simulator“, Conference on Robot
Learning 2017.
