Conference PaperPDF Available

Time-of-Flight Cameras for Parking Assistance: A Feasibility Study

Authors:
Time-of-Flight Cameras for Parking Assistance: A Feasibility Study
Josef Steinbaeck1, Norbert Druml1, Allan Tengg2, Christian Steger3, and Bernhard Hillbrand2
1Infineon Technologies AG, Graz, Austria, {josef.steinbaeck, norbert.druml}@infineon.com
2Virtual Vehicle Research Center, Graz, Austria,{allan.tengg, bernhard.hillbrand}@v2c2.at
3Graz University of Technology, Graz, Austria, steger@tugraz.at
Parking assistance is one of the most demanded assisted driving functionalities of
today. In contrast to 2D cameras, time-of-flight (ToF) sensors provide real 3D
data which can be used to inform the driver about the vehicle’s surroundings.
State-of-the-art ToF cameras are compact, inexpensive and capable of providing
high frame-rate 3D data with minimal computational overhead.
In this paper, we evaluate the feasibility of ToF cameras used as perception
sensors in parking assistance applications. ToF cameras are highly capable for
parking assistance applications, but for outdoor usage, an enhanced illumination
unit is necessary.
1. Introduction
Starting with May 2018, a new regulation in the USA and Canada became effective,
making it mandatory for car manufacturers to install a rear-view camera into every new
vehicle. The law was enforced to counteract the high number of accidents that occur while
vehicles are backing up. A publication by the US National Highway Traffic Safety
Administration reports an estimation of 292 total annual back-over fatalities within the USA
[1]. Reasons include overlooking and occlusion of obstacles due to the limited field-of-view,
particularly to the vehicle’s back.
Modern technology is capable of providing additional information to the driver in order to
support maneuvers at low speed. Rear-view cameras have already been integrated into many
upper-class vehicles during the last decades. Multiple cameras around the ego vehicle can be
used to assist the parking process with a top-down bird’s view perspective of the close
environment. Ultrasonic sensors have been used as distance sensors for parking assistance
since a long time already. Many modern cars come with these sensors, since they are
comparably small, cheap and easy to integrate into the vehicle’s chassis. However, ultrasonic
sensors are not precise enough to achieve a high resolution.
ToF cameras provide resolutions of over 100k pixels, millimeter precision and frame rates
of more than 100 frames per second (FPS). In contrast to ultrasonic sensors, ToF data can be
utilized to create an accurate 3D model of the close surroundings. In this paper, we evaluate
the feasibility of a 3D ToF camera for parking assistance. We mounted one ToF camera on a
passenger vehicle and inspected the data quality in different environments. Additionally, we
designed parking assistance software which visualizes the 3D points to the driver. Considering
the results of this real-world use case, we point out the advantages and disadvantages of ToF
cameras for parking assistance. To sum up, the contributions of this paper are:
Evaluation of ToF data for parking assistance.
Exploration of different mounting options on a passenger vehicle.
Visualization of the 3D data to the vehicle’s dashboard.
2. Related Work
ToF cameras are used in various applications like augmented reality, face-tracking and 3D
localization. Prominent approaches in the automotive context include driver/interior
monitoring and hand-gesture recognition [2], [3]. Scheunert et al. already presented an
approach where a PMD camera is used to detect the free space of a parking slot in 2007 [4].
However, the used camera with a resolution of 16x64 pixels is by far inferior to a modern ToF
camera. Gallo et al. show an approach where a ToF camera is used to detect curbs and ramps
in order to perform safe parking [5]. Ringbeck et al. mount a ToF camera on a car with a
powerful light source (8W optical power) and achieve a range of up to 35 m [6].
A vision-based system, utilizing multiple fish-eye cameras around the vehicle to realize
automatic parking is shown by Wang et al. [7]. The authors use inverse perspective mapping
of four fish-eye images to provide a bird’s eye view of the vehicle’s close surrounding.
Compact, inexpensive and high-resolution ToF cameras are relatively new to the industry.
To the best of our knowledge, there is no work available to the scientific community,
evaluating a state-of-the-art ToF camera for parking assistance. This paper fills this gap by
presenting an approach to utilize a modern (2018) ToF camera to obtain 3D data of the ego
vehicle’s environment.
3. A Time-of-Flight Camera for Parking Assistance
We mounted a ToF camera onto a passenger vehicle in order to use it with a parking
assistance system. The ToF data is used to visualize depth information of the environment to
the vehicle’s dashboard screen.
3.1 Time-of-Flight Principle
The prevalent way to realize indirect ToF cameras is illuminate the scene with
modulated infrared light and utilize photonic mixing device (PMD) pixels to detect the
reflections. Each pixel measures a value that indicates the correlation between the received
signal and a reference signal. The so called four-phase algorithm is used to determine the
distance and amplitude for every pixel [8]. Four measurements with different phase-shifted
versions of the transmitted signal as reference signal are used to calculate the phase difference
and the amplitude for each pixel. The distance of a ToF pixel can be easily determined using
the phase difference, the speed of light and the modulation frequency.
The pixels integrated in the selected ToF camera implement a suppression of background
illumination (SBI) circuitry [6]. This circuit prevents the pixels from saturation when exposed
to an unmodulated light source with a spectral component in the same range as the ToF
working frequency (e.g., sunlight). However, the pixels still experience noise from ambient
light. Thus, in applications with bright ambient light, the illumination power has to be selected
accordingly.
3.2 Evaluation Setup
We use the CamBoard pico monstar as ToF camera. The camera comes with the
IRS1125C REAL3 3D Image Sensor developed by Infineon and pmdtechnologies. The image
sensor has a resolution of 352×287 pixels, a field-of-view of 100°×85° and provides up to 60
frames per second. The illumination is performed by four vertical-cavity surface-emitting
lasers (VCSEL) at a wavelength of 850 nm.
We mounted the ToF camera on a Ford Mondeo passenger vehicle, facing backwards,
while considering the following requirements/trade-offs:
The camera shall capture the edges of the vehicle in order to determine the boundaries.
The ground area captured bythe field-of-view shall be maximized.
The mounting position shall be as far to the back of the vehicle as possible.
3.3 Visualization of Time-of-Flight 3D Data
We designed and implemented software to visualize the obtained 3D data on the
dashboard screen of the vehicle. The data has to be presented in a simple enough way to even
allow an untrained driver to interpret the data without effort.
The software uses the OpenGL API for rendering the 3D point cloud on the screen. There
are several toolkits (i.e., GLUT or Open Inventor) available that simplify the visualization of
3D data. However, as the desired target platform is an automotive on-board computer, the
native OpenGL API was chosen to keep the software as portable as possible. This low level
API allows defining several properties of the scenery like the field-of-view of the camera, the
viewing distance as well as light sources.
In that virtual 3D world, the data from the ToF camera is rendered as a bunch of cubes.
Performance of both, the visualization software and the graphic hardware, is an important
issue at that point, since about 100k points have to be drawn for every frame. By adding a 3D
model of the ego vehicle, the viewer gets a better understanding of the scene. This requires an
exact calibration and registration to preserve the scale of the voxels in relation to the ego
vehicle. In the last step, the OpenGL’s virtual camera is placed in the 3D world to view the
scene from any perspective.
4. Results
The setup, as already presented in Section 3.2, was used to perform measurements in
different parking environments. The first evaluation took place in an indoor parking lot
without any sunlight present. The ToF camera provided an almost flawless amplitude and
distance image. The close surroundings including the obstacles (parked car, walls) were
clearly visible in the distance image. We performed another evaluation of the ToF data in an
outdoor parking lot in bright sunlight. As expected, this scene resulted in a noisier amplitude
image. Since low amplitude values correspond to a lower confidence, some pixels within the
distance image were discarded.
In presence of bright sunlight, the camBoard pico monstar showed reduced performance.
The main reason for this is that the camera’s illumination unit is not optimized for outdoor,
long-range applications and thus, produces insufficient laser output power. Yet, there already
exist optimized outdoor cameras that integrate the same PMD ToF sensor chip but with
stronger illumination units which achieve a working range of 35m in adverse/bright ambient
light situations.1
Two different options to visualize the ToF data are shown in Fig. 1. The visualization is
streamed to the vehicle’s dashboard screen. Especially the top-down view of the point cloud,
after removing the drivable area, is highly capable of supporting the driver in parking tasks.
This perspective enables the driver to estimate distances around the car intuitively.
1O3M 3D sensor for mobile applications (https://www.ifm.com/at/en)
Fig. 1. Visualization of ToF data during a parking task, viewed from different perspectives.
5. Conclusion
ToF cameras provide precise, high frame-rate 3D data of the environment. This data can
be used to obtain a detailed 3D model of the surroundings. We mounted a ToF camera on a
passenger vehicle and implemented software, capable of streaming the point cloud of the
surrounding environment to the dashboard and visualizing it to the driver. Considering the
results of this work, we conclude that ToF cameras are feasible to be used in parking
assistance systems. A promising scenario is the combination of a top-down surround-view
video stream with the 3D data from multiple ToF sensors. Augmenting the top-down 2D color
image with the 3D points from the ToF cameras could provide the driver with detailed and
easily-interpretable visual information about the car’s surroundings.
Acknowledgements
The authors of this paper would like to thank the Electronic Component Systems for European
Leadership Joint Undertaking for funding the IoSense: Flexible FE/BE Sensor Pilot Line for
the Internet of Everything project with the number 692480.
References
[1] R. Austin, “Fatalities and injuries in motor vehicle backing crashes,” National Highway
Traffic Safety Administration, Tech. Rep., 2008.
[2] D. Demirdjian and C. Varri, “Driver pose estimation with 3d time-of-flight sensor,” in
2009 IEEE Workshop on Computational Intelligence in Vehicles and Vehicular
Systems, March 2009.
[3] P. Molchanov, S. Gupta, K. Kim, and K. Pulli, “Multi-sensor system for driver’s hand-
gesture recognition,” in 2015 11th IEEE International Conference and Workshops on
Automatic Face and Gesture Recognition (FG), May2015.
[4] U. Scheunert, B. Fardi, N. Mattern, G. Wanielik, and N. Keppeler, “Free space
determination for parking slots using a 3D PMD sensor,” 2007 IEEE Intelligent
Vehicles Symposium, 2007.
[5] O. Gallo, R. Manduchi, and A. Rafii, “Robust curb and ramp detection for safe parking
using the Canesta TOF camera,” 2008 IEEE Computer Society Conference on Computer
Vision and Pattern Recognition Work-shops, CVPR Workshops, 2008.
[6] T. Ringbeck, T. Möller, and B. Hagebeuker, “Multidimensional measurement by using
3-D PMD sensors,” Advances In Radio Science, 2007.
[7] C. Wang, H. Zhang, M. Yang, X. Wang, L. Ye, and C. Guo, “Automatic parking based
on a bird’s eye view vision system,” Advances in Mechanical Engineering, 2014.
[8] R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE Journal of
Quantum Electronics, March 2001.
... A comparative study has been presented and it is shown that these ToF devices only perform well up to a maximum of 5-7 m and are too sensitive for outdoor usage, especially in bright areas [4,5,6]. Baeck et al. [15] showed the use of ToF for outdoor parking assistance includes additional lighting units to visualize depth details. Light field (or plenoptic) camera [16][17][18][19] design records the intensity and the direction in which light rays pass in space. ...
Article
This paper proposes a new technique to achieve a person's depth from a single image by considering the target foreground and background scene variations in extreme weather conditions within 40 meters range. For this purpose series of images are captured on each person at successive intervals. The height, distance, foreground, and background features are extracted using an object detection deep learning framework. The obtained features are then subsequently trained by the Gradient Booster Regressor to predict the depth information. Furthermore, the algorithm is tested on various images and is validated with ground truth depth data. The findings presented in this paper attest to the reliability of the methodology used for depth estimation.
Article
A novel amplitude-modulated continuous wave (AMCW) time-of-flight (ToF) scanning sensor based on parallel-phase demodulation is proposed and demonstrated. Contrast to the conventional AMCW ToF sensor which utilizes serial-phase demodulation, the proposed sensor processes each different phase shift demodulation independently in parallel manner, so that the proposed sensor can maintain high demodulation contrast and short total integration time. Based on these characteristics of parallel-phase demodulation, the proposed sensor platform can be implemented with 2D laser scanning system with a single photodetector, maintaining a moderate frame rate, high optical SNR, and very high pixel resolution. In the validating demonstration, the proposed AMCW ToF scanning sensor shows extremely low measured distance noise per reference distance of 0.056% at 1.50 m with only 30 mW illumination power. Additionally, FHD scale (1920 × 1080) highly precise 3D depth map with 800 nsec integration time can be achieved using the proposed AMCW ToF scanning sensor.
Article
The mobile mapping system (MMS) could become the foundation of digital twins and 3D modeling, and is widely applicable in a variety of fields, such as infrastructure management, intelligent transportation systems, and smart cities. However, data collected by MMS is extensive and complex, making data processing difficult. We present a novel method for segmenting urban assets (specifically in this case study traffic signs) with a lower-cost Azure Kinect and automatic data processing workflows. First, it was necessary to verify the reliability of this approach using the Time of Flight (ToF) camera from Azure Kinect to detect road signs outdoors. Using the data generated by the ToF camera, we then extracted the Region of Interest (ROI) quickly and efficiently. After transforming the ROI to the RGB image, we obtained the traffic sign area through a hybrid color-shape based method. In addition, we calculated the distance between the traffic sign and Azure Kinect based on the depth image. The Coefficient of Variation cv averaged 1.1%. It is thus evident that Azure Kinect is reliable for outdoor traffic sign segmentation. Our algorithm has been compared with deep learning algorithms. According to our analysis, our algorithm has an accuracy of 0.8216, while the accuracy of deep learning is 0.7466, which indicates that our solution is more flexible and cost-effective.
Article
Full-text available
This paper aims at realizing an automatic parking method through a bird's eye view vision system. With this method, vehicles can make robust and real-time detection and recognition of parking spaces. During parking process, the omnidirectional information of the environment can be obtained by using four on-board fisheye cameras around the vehicle, which are the main part of the bird's eye view vision system. In order to achieve this purpose, a polynomial fisheye distortion model is firstly used for camera calibration. An image mosaicking method based on the Levenberg-Marquardt algorithm is used to combine four individual images from fisheye cameras into one omnidirectional bird's eye view image. Secondly, features of the parking spaces are extracted with a Radon transform based method. Finally, double circular trajectory planning and a preview control strategy are utilized to realize autonomous parking. Through experimental analysis, we can see that the proposed method can get effective and robust real-time results in both parking space recognition and automatic parking.
Article
Full-text available
Range sensors for assisted backup and parking have po-tential for saving human lives and for facilitating parking in challenging situations. However, important features such as curbs and ramps are difficult to detect using ultrasonic or microwave sensors. TOF imaging range sensors may be used successfully for this purpose. In this paper we present a study concerning the use of the Canesta TOF cam-era for recognition of curbs and ramps. Our approach is based on the detection of individual planar patches using CC-RANSAC, a modified version of the classic RANSAC robust regression algorithm. Whereas RANSAC uses the whole set of inliers to evaluate the fitness of a candidate plane, CC-RANSAC only considers the largest connected components of inliers. We provide experimental evidence that CC-RANSAC provides a more accurate estimation of the dominant plane than RANSAC with a smaller number of iterations.
Article
Full-text available
Optical Time-of-Flight measurement gives the possibility to enhance 2-D sensors by adding a third dimension using the PMD principle. Various applications in the automotive (e.g. pedestrian safety), industrial, robotics and multimedia fields require robust three-dimensional data (Schwarte et al., 2000). These applications, however, all have different requirements in terms of resolution, speed, distance and target characteristics. PMDTechnologies has developed 3-D sensors based on standard CMOS processes that can provide an optimized solution for a wide field of applications combined with high integration and cost-effective production. These sensors are realized in various layout formats from single pixel solutions for basic applications to low, middle and high resolution matrices for applications requiring more detailed data. Pixel pitches ranging from 10 micrometer up to a 300 micrometer or larger can be realized and give the opportunity to optimize the sensor chip depending on the application. One aspect of all optical sensors based on a time-of-flight principle is the necessity of handling background illumination. This can be achieved by various techniques, such as optical filters and active circuits on chip. The sensors' usage of the in-pixel so-called SBI-circuitry (suppression of background illumination) makes it even possible to overcome the effects of bright ambient light. This paper focuses on this technical requirement. In Sect. 2 we will roughly describe the basic operation principle of PMD sensors. The technical challenges related to the system characteristics of an active optical ranging technique are described in Sect. 3, technical solutions and measurement results are then presented in Sect. 4. We finish this work with an overview of actual PMD sensors and their key parameters (Sect. 5) and some concluding remarks in Sect. 6.
Article
Full-text available
The concept of a real-time range camera without moving parts is described, based on the time-of-flight (TOF) principle. It operates with modulated visible and near-infrared radiation, which is detected and demodulated simultaneously by a 2-D array of lock-in pixels employing the charge-coupled device principle. Each pixel individually measures the amplitude, offset and phase of the received radiation. The theoretical resolution limit of this TOF range camera is derived, which depends on the square root of the detected background radiation and the inverse of the modulation amplitude. Actual measurements of 3-D sequences acquired at 10 range images per second show excellent agreement between our theory and the observed results. A range resolution of a few centimeters over a range of 10 m, with an illumination power of a few hundreds of milliwatts is obtained in laboratory scenes for noncooperative, diffusely reflecting objects
Conference Paper
We propose a novel multi-sensor system for accurate and power-efficient dynamic car-driver hand-gesture recognition, using a short-range radar, a color camera, and a depth camera, which together make the system robust against variable lighting conditions. We present a procedure to jointly calibrate the radar and depth sensors. We employ convolutional deep neural networks to fuse data from multiple sensors and to classify the gestures. Our algorithm accurately recognizes 10 different gestures acquired indoors and outdoors in a car during the day and at night. It consumes significantly less power than purely vision-based systems.
Conference Paper
In this paper we tackle the problem of vision-based driver pose estimation, i.e. estimating the location and orientation of the driver's limbs, including arms, hands, head and torso from visual information. In our approach, visual information consists of depth and intensity images provided by an infrared time-of-flight (TOF) camera. For pose estimation we propose a variant of the articulated ICP algorithm, a 3D model fitting approach, which is able to incorporate the uncertainty in visual observation and model data in the pose estimation framework.
Conference Paper
We present an approach for parking slot detection using a 3D Range camera of PMD type. This sensor allows referring to a large number of spatial point measurements detailed representing cuts of the observed scene. The focus of this paper is on the feature extraction out of the PMD data as well as the fusion of the features defining the free space of a parking slot. The feature extraction includes reliable and exact curb detection and robust obstacle detection. The approach for the optimal feature extraction is based on the usage of an occupancy grid in combination with a feature conform definition of detection channels.
Fatalities and injuries in motor vehicle backing crashes
  • R Austin
R. Austin, "Fatalities and injuries in motor vehicle backing crashes," National Highway Traffic Safety Administration, Tech. Rep., 2008.