Conference Paper

The driver fatigue monitoring system based on face recognition technology

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

This paper uses different algorithms, which are called AdaBoost algorithm and the difference between infrared frames algorithm, to locate the precise position of the eyes in different light environment of driving. We identify the eye's status by extracting the characteristic parameters of eyes and detect fatigue based on the method of PERCLOS. At the same time, tfurther test the driver's fatigue, we use the Local Binary Patter (LBP) algorithm to detect the yawning as an aided detection. The results of the experiment show that algorithm ensures the accuracy of the system and it can achieve the requirement of non contact type, different lighting conditions and real-time detection.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... A few years ago, vision-based methods were proposed to detect fatigue by capturing specific features with one or more cameras and image processing. In these methods, a driver monitor system captures the face of the driver and extracts features such as blinking, yawning, and the head movement of the driver [4][5][6][7][8]. Further, a driver assistance system captures the road image, labels the lane line to determine the lane departure, and decides the fatigue state [7,8]. ...
... After the system detects the landmarks from Dlib, blink detection can be performed by checking if EAR exceeds the threshold. In addition to blink detection, Luo et al. proposed PERCLOS as an indicator of fatigue [4]. The definition of PERCLOS is ...
Article
Full-text available
In this paper, a vision-based physiological signal measurement system is proposed to instantly measure driver fatigue. A remote photoplethysmography (rPPG) signal is a type of physiological signal measured by a camera without any contact device, and it also retains the characteristics of the PPG, which is useful to evaluate fatigue. To solve the inconvenience caused by the traditional contact-based physiological fatigue detection system and to improve the accuracy, the system measures both the motional and physiological information by using one image sensor. In a practical application, the environmental noise would affect the measured signal, and therefore, we performed a preprocessing step on the signal to extract a clear signal. The experiment was designed in collaboration with Taipei Medical University, and a questionnaire-based method was used to define fatigue. The questionnaire that could directly react to the feeling of the subject was treated as our ground truth. The evaluated correlation was 0.89 and the root mean square error was 0.65 for ten-fold cross-validation on the dataset. The trend of driver fatigue could be evaluated without a contact device by the proposed system. This advantage ensures the safety of the driver and reliability of the system.
... They used principal component analysis and Adaboost to classify eye data and finally used PERCLOS for driver fatigue detection. Luo et al. [8] used the AdaBoost algorithm to find the exact position of the eyes in different driving conditions under different lights. They used the characteristic parameters of PERCLOS and eyes for fatigue detection. ...
... Fig. 7 shows Multi-task ConNN model. Depending on the width of the mesh, the filter size was determined as first (12,15,20) and second (6,8,10) in three convolution layers, respectively. Two experiments were conducted to select network structures including convolution layer capacity, nonlinear activation function and pooling method. ...
Article
Full-text available
Changes and progresses in information technologies have played an important role in the development of intelligent vehicle systems in recent years. Driver fatigue is an important factor in vehicle accidents. For this reason, traffic accidents involving driver fatigue and driver carelessness have been followed by researchers. In this article, a Multi-tasking Convulational Neural Network (ConNN * ) model is proposed to detect driver drowsiness/fatigue. Eye and mouth characteristics are utilized for driver’s behavior model. Changes to these characteristics are used to monitor driver fatigue. With the proposed Multi-task ConNN model, unlike the studies in the literature, both mouth and eye information are classified into a single model at the same time. Driver fatigue is determined by calculating eyes’ closure duration/Percentage of eye closure (PERCLOS) and yawning frequency/frequency of mouth (FOM). In this study, the fatigue degree of the driver is divided into 3 classes. The proposed model achieved 98.81% fatigue detection on YawdDD and NthuDDD dataset. The success of the model is presented comparatively. * Convulational Neural Network has been abbreviated as ConNN, not CNN or CoNN, CNN has been used as the abbreviation of Celluar Neural Network and CoNN has been used as the abbreviation of Cooperative neural networks in the leterature as a long time. </fn
... Among the found approaches the goal of detecting driver's drowsiness by using minimal equipment without interfering with vehicle's equipment can only be achieved via the computer vision-based models running on a mobile phone. However, these approaches depend on the lighting conditions [7], [17], [18], which are far from perfect in a moving car. Besides, drowsiness detection is especially important at night, when lighting conditions are the worst (not only the light is low, but there are also significant lighting changes due to lights of oncoming vehicles and road lights). ...
Conference Paper
Detection of the drivers drowsy state is still an actual task since it is a reason for a significant number of traffic accidents. The carried out literature review showed that a significant number of approaches rely on special equipment for driver state identification. At the same time, efficient operation of computer vision-based techniques heavily depends on the lighting conditions, which are usually not good in a moving car. The paper presents a research effort aiming at using speed recordings to identify the drivers state. For this purpose, the speed recordings are analyzed as a time series, and its characteristics are used as features for the classification task. The results show that the suggested approach is viable and promising.
... Driver tracking system that has been successfully tested in real time and in different light conditions is presented in [6]. The system estimates whether it is the day or night based on the brightness of the camera's input image. ...
... Liu et al. offered a robust method to detect the profile face of the driver by a modified version of skin segmentation [3]. Moreover, the drowsiness state of the driver can be detected using the multi-features of face202122, such as facial expressions [23]. Discrete cosine transform (DCT) and modified census transform (MCT) models are introduced in [24]. ...
Conference Paper
Some novel vision-based dazzling avoidance systems, such as the ShadeVision system, can intelligently cast a shadow on driver's eye region to protect the eyes from the dazzling effect that are caused by the strong external light sources. However, during this protective state (eyes covered by shadow), the system still needs to detect the driver's face to realize further monitoring, e.g., the intention of the driver as well as the dazzling effect from another light source. The protective shadow will make it difficult to apply the state-of-the-art algorithms to detect the driver's face, as the eye information is of great importance to vision-based face detection. This paper presents a series of robust algorithms for the face detection with protective shadow cast on driver's eyes, among which the Partially Masked Training, the Consecutive Sub-block Training and the Overlapped Sub-block Training are proposed for the first time for this kind of task. The on-road experimental results verify the effectiveness of the designed algorithms. Keywords—face detection; eyes occluded by shadow; overlapped training; dazzling avoidance system
... Matsuo et al. developed an algorithm which can robustly rious sunlight environment [6]. Moreover, the driver drowsiness state can be detected based on multi-features of face [7][8], such as face expression [9 highly depends on the face and eye information [10][11]. However, all these do not provide a robust method to detect the profile face of the driver, which is critical in the vision-based dazzling avoidance system. ...
Conference Paper
This paper presents robust methods of detecting the profile face and estimating the rotation head in the ShadeVision system [1], which is a novel vision-based driving assistance system aiming to avoid the dazzling effect caused by strong external light. As the detection rate of dazzling effect highly relies on the brightness of the profile face [2], new algorithms need to be developed to realize the robust profile face detection in driving scenario. When there are multiple strong external light sources, the system needs to sight is focusing in order to determine the light source that is dangerous to the driver [1]. To this end the precise rotation angle of the driver is of great importance. The effectiveness of the developed algorithms is verified by the laboratory and field tests.
... The methods of detecting driver fatigue include physiological signal detection [1,2], vehicle behavioral characteristics detection [3] and computer vision based detection [4][5][6]. In these driving fatigue evaluation methods, PERCLOS (Percentage of Eyelid Closure over the Pupil Over Time) based on computer vision is recognized as the most effective, automotive and real-time one [7], which was first proposed in the technical forum of the Federal Highway Administration in April 1999. ...
Article
Full-text available
Driver fatigue is a popular problem which has attracted people’s views. Many research departments are researching driver fatigue detection in order to improve the traffic safety. This paper presented a driver fatigue detection system based on DM3730. The system calculated the inter-image difference between frames captured by near-infrared light irradiation, which included identification of the eyes by Otsu adaptive threshold segmentation method and prediction the orientation of the eye in nearby images by Kalman filter. Then the system determined the state of fatigue by improved PERCLOS (Percentage of Eyelid Closure over the Pupil)method. Experimental results show that the system has the advantages of small size and low power consumption. Meanwhile it meets the requirements of all-weather, real-time monitoring. The system can be extended to automobile and other production processes which the fatigue monitoring is required.
... Matsuo et al. developed an algorithm which can robustly rious sunlight environment [6]. Moreover, the driver drowsiness state can be detected based on multi-features of face [7][8], such as face expression [9 highly depends on the face and eye information [10][11]. However, all these do not provide a robust method to detect the profile face of the driver, which is critical in the vision-based dazzling avoidance system. ...
Conference Paper
This article presents a novel method of detecting the dazzling effect in a new vision based driving assistance system (ShadeVision) which aims to avoid the dazzling effect caused by strong external light sources. Different from existing systems which only detect the dazzling effect at night, our proposed system can also identify the dazzling effect during the day, particularly in dawn and dusk. To that end, the eye region of the driver and other regions surrounding the eyes, the interior of the vehicle, as well as the background light outside the window are all captured by a high speed camera with 1280×500 resolution and a frame rate of 1000 Hz mounted in the vehicle. Afterwards all these data are synthesized and analyzed to determine the dazzling effect. Moreover, the details of dazzling detection method for nighttime and daytime driving are presented explicitly. The final tests verify the effectiveness of our proposed approaches.
Chapter
Today, human drivers remain the main cause of traffic accidents. While not always being the main accident reasons, drowsiness and fatigue are often indirect reasons for accidents. As a result, it is still actual to classify the driver’s state (drowsy or awake) while driving. The most popular approaches to driver’s state classification today are the application of special medical equipment (which gives the highest results) or machine vision techniques. However, the former are nearly impossible for mass implementation due to the complexity and high costs, and the latter suffer from continuously varying lighting, which is the case for moving vehicles especially in dark conditions when the identification of the driver’s state is even more important. Previous research has shown the fundamental possibility of identification of the driver’s state based on an analysis of vehicle speed. However, its results were not very high. In this work, we are searching for a way to increase the quality of the classification by taking into account the driving context. The results show that (a) the classification of the vehicle’s driving context can be performed based on the analysis of its speed with sufficiently high accuracy, and (b) that the preliminary classification of the vehicle’s driving context can significantly increase the accuracy of the classification of the driver’s state based on the analysis of the vehicle speed.
Article
Full-text available
The literature on the topic has shown a strong correlation between the degree of precision of face localization and the face recognition performance. Hence, there is a need for precise facial feature detectors, as well as objective measures for their evaluation and comparison. In this paper, we will present significant improvements to a previous method for precise eye center localization, by integrating a module for mouth localization. The technique is based on Support Vector Machines trained on optimally chosen Haar wavelet coefficients. The method has been tested on several public databases; the results are reported and compared according to a standard error measure. The tests show that the algorithm achieves high precision of localization.
Conference Paper
Full-text available
This paper describes a machine learning approach for visual object detection which is capable of processing images extremely rapidly and achieving high detection rates. This work is distinguished by three key contributions. The first is the introduction of a new image representation called the "integral image" which allows the features used by our detector to be computed very quickly. The second is a learning algorithm, based on AdaBoost, which selects a small number of critical visual features from a larger set and yields extremely efficient classifiers. The third contribution is a method for combining increasingly more complex classifiers in a "cascade" which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions. The cascade can be viewed as an object specific focus-of-attention mechanism which unlike previous approaches provides statistical guarantees that discarded regions are unlikely to contain the object of interest. In the domain of face detection the system yields detection rates comparable to the best previous systems. Used in real-time applications, the detector runs at 15 frames per second without resorting to image differencing or skin color detection.
Conference Paper
This paper is presented research on face detection for fatigue driving detection. Due to the complicated situation when driving, conventional methods that applied in detecting driver's face take a large amount of time and are at low accuracy rate. For this reason, an ellipse with self-learning radius is taken to instead of 2d Gaussian Distribution Model, and some other promoted methods are also used in this paper such as Integral Projection, Barycenter Adjustment method, etc. All of these achieve the purpose of positioning face accurately and quickly in various complicated cases such as complex illumination, small angles of side face, etc.
Article
Drowsiness among commercial vehicle drivers has been identified as the number one safety concern of commercial fleets at trucking summit meetings. Over the past 10 years, the U.S. Department of Transportation's National Highway Traffic Safety Administration (NHTSA) and its research partners have sought to quantify the loss of alertness among commercial vehicle drivers. This work led to the development of the world's first unobtrusive and valid sensor of loss of alertness, and has been the benchmark for continuing international study. Replicated experiments have shown that the most valid measure of loss of alertness among drivers is the percentage of eyelid closure over the pupil over time (Perclos). Formerly pioneered by Dr. Walter Wierwille at the Virginia Polytechnical Institute and State University, using a manual observation technique(6), Perclos is now monitored in real time using machine vision technology in the vehicle. In order to estimate the highway safety benefit based on the effectiveness of the system, a Field Operational Test (FOT) is underway with long haul and express (i.e., overnight) fleet operations. This paper discusses the field test methodology, as well as the questions each analysis seeks to answer. A summary of the status of the project, the results to date, and a vision of future work for the deployment of this technology will be provided.
Article
This paper presents a method for automatic real time face and mouth recognition using radial basis function neural networks (RBFNN). The proposed method uses the motion information to localize the face region, and the face region is processed in YCrCb color space to determine the locations of the eyes. The center of the mouth is determined relative to the locations of the eyes. Facial and mouth features are extracted using multiscale morphological erosion and dilation operations, respectively. The facial features are extracted relative to the locations of the eyes, and mouth features are extracted relative to the locations of the eyes and mouth. The facial and mouth features are given as input to radial basis function neural networks. The RBFNN is used to recognize a person in video sequences using face and mouth modalities. The evidence from face and mouth modalities are combined using a weighting rule, and the result is used for identification and authentication. The performance of the system using facial and mouth features is evaluated in real time in the laboratory environment, and the system achieves a recognition rate (RR) of 99.0% and an equal error rate (EER) of about 0.73% for 50 subjects. The performance of the system is also evaluated for XM2VTS database, and the system achieves a recognition rate (RR) of 100% an equal error rate (EER) of about 0.25% for 50 subjects.
Conference Paper
Fatigue affects a driver's ability to control his or her cars safely. This paper proposes a novel real-time face tracking algorithm for detection of driver fatigue, based on tracking eyes, mouth and pose of a driver. The input face images are preprocessed to gray level images by using generalized Laplacian. Two eyes and mouth are labeled in binary image. According to pose of the driver, the system forms either isosceles feature triangle or right feature triangle to acquire potential face region. During the tracking of the video sequence, the confusing triangle is removed effectively. Experimental results are presented to demonstrate the good performance of the proposed face tracking algorithm, and about 98% success rate is achieved.
Article
This paper presents a novel and efficient facial image representation based on local binary pattern (LBP) texture features. The face image is divided into several regions from which the LBP feature distributions are extracted and concatenated into an enhanced feature vector to be used as a face descriptor. The performance of the proposed method is assessed in the face recognition problem under different challenges. Other applications and several extensions are also discussed.
Article
A cascade face detector uses a sequence of node classifiers to distinguish faces from non-faces. This paper presents a new approach to design node classifiers in the cascade detector. Previous methods used machine learning algorithms that simultaneously select features and form ensemble classifiers. We argue that if these two parts are decoupled, we have the freedom to design a classifier that explicitly addresses the difficulties caused by the asymmetric learning goal. There are three contributions in this paper. The first is a categorization of asymmetries in the learning goal, and why they make face detection hard. The second is the Forward Feature Selection (FFS) algorithm and a fast pre- omputing strategy for AdaBoost. FFS and the fast AdaBoost can reduce the training time by approximately 100 and 50 times, in comparison to a naive implementation of the AdaBoost feature selection method. The last contribution is Linear Asymmetric Classifier (LAC), a classifier that explicitly handles the asymmetric learning goal as a well-defined constrained optimization problem. We demonstrated experimentally that LAC results in improved ensemble classifier performance.
Conference Paper
Fatigue driving is an important reason of traffic accidents. Yawning is an evidence of driver fatigue. This paper proposes to locate and track a driver's mouth movement using a CCD camera to study on monitoring and recognizing a driver's yawning. Firstly detecting drivers' faces uses Gravity-Center template, then detecting drivers' left and right mouth corners by grey projection, and extracting texture features of drivers' mouth corners (left and right) using Gabor wavelets. Finally LDA is applied to classify feature vectors to detect yawning. The method is tested on 400 images from twenty videos. In contrast, yawning is also detected by the ratio of mouth height and width. The experiment results show that Gabor coefficients are more powerful than geometric features to detect yawning and the average recognition rate is 95% which has more than 20% improvement.
Conference Paper
By using the fusion of contextual, visual and non-visual features, a model based on Dempster-Shafer (D-S) evidence theory is proposed to obtain a reliable driver's fatigue recognition. Firstly, an overall model structure is set up with respect to the selected features and key symptoms of driver's fatigue. Secondly, a set of heuristic knowledge rules are used to determine the basic probability assignment; and a modified evidence combination is adopted to combine multiple pieces of evidence including consistent and conflicting ones. Thirdly, decision policy based on the basic probability assignment is applied to fatigue recognition. At last, an example is given to illustrate the proposed fatigue recognition model.
Conference Paper
This paper examines the feasibility of a semantic-level driver activity analysis system. Several new considerations are made to construct the hierarchy of driver activity. Driver activity is represented and recognized at multiple levels: individual body-part pose/gesture at the low level, single body-part action at the middle level, and the driver interaction with the vehicle at the high level. Driving is represented in terms of the interactions among driver, vehicle, and surround, and driver activity is recognized by a rule-based decision tree. Our system works with a single color camera data, and it can be easily expanded to incorporate multimodal sensor data.
Conference Paper
We present a non-intrusive prototype computer vision system for real-time monitoring driver's vigilance. It is based on a hardware system, for real time acquisition of driver's images using an active IR illuminator, and their software implementation for monitoring some visual behaviours that characterize a driver's level of vigilance. These are the eyelid movements and the pose face. The system has been tested with different sequences recorded on night and day driving conditions in a motorway and with different users. We show some experimental results and some conclusions about the performance of the system.
Article
This paper describes a real-time online prototype driver-fatigue monitor. It uses remotely located charge-coupled-device cameras equipped with active infrared illuminators to acquire video images of the driver. Various visual cues that typically characterize the level of alertness of a person are extracted in real time and systematically combined to infer the fatigue level of the driver. The visual cues employed characterize eyelid movement, gaze movement, head movement, and facial expression. A probabilistic model is developed to model human fatigue and to predict fatigue based on the visual cues obtained. The simultaneous use of multiple visual cues and their systematic combination yields a much more robust and accurate fatigue characterization than using a single visual cue. This system was validated under real-life fatigue conditions with human subjects of different ethnic backgrounds, genders, and ages; with/without glasses; and under different illumination conditions. It was found to be reasonably robust, reliable, and accurate in fatigue characterization.
Drive Fatigue Monitoring and Identification Methods
  • Song Zhumei
  • Wang Hailin
  • Liu Hanhui