International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol 16 No 16 (2022) Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car https://doi.org/10.3991/ijim.v16i16.33075 Abdelfettah Soultana(), Faouzia Benabbou, Nawal Sael, Sara Ouahabi Laboratory of Modeling and Information Technology Faculty of Sciences Ben M’SIK, University Hassan II, Casablanca, Morocco soultana.abdelfettah@gmail.com Abstract—In recent years, a significant increase in road accidents worldwide has been observed. This can partly be due to either driver distraction or fatigue. Therefore, a reliable alerting system that can detect the driver’s inattention including fatigue, sleep, and distraction is necessarily required to prevent any potential accidents. The aim of this paper is to conduct a systematic review of literature (SLR) on monitoring driver inattention. In particular, the present study deals with different aspects of prior studies such as the sensors used; the types of data, the feature engineering techniques, the machine-learning techniques applied and their performance along with, the dataset used, etc. anotherFour approaches can be depicted from literature according to indicators they are based on: phys- iological, physical, driver performance and hybrid approach. We will focus on these different approaches in order to answer different questions, starting with the type of indicators used in the case of distraction or fatigue detection, the different datasets employed, the feature extraction techniques and the machine learning models applied. Furthermore, the study examines the practicality and reliability of each of the four approaches, as well as possible future prospects in the area, and highlights new challenges in the field of driver inattention detection with both forms of fatigue and distraction. Keywords—driver inattention, driver distraction, driver fatigue, driver drowsiness, artificial intelligence, machine-learning, deep-learning 1 Introduction The problem of road crashes has become a major global concern. According to World Health statistics (WHO), approximately 1.35 million people die each year because of traffic accidents. In addition, 20 to 50 million individuals are affected by non-fatal injuries, with many becoming crippled as a result [1, 2, 3]. Road traffic injuries would become the fifth leading cause of mortality by 2030, according to a World Health Organization (WHO) global status report on road safety [4]. The factors that affect the risk of traffic accidents are classified into three categories: human, road environment, and vehicle condition [5]. The main causes of these accidents are distracted driving, 160 http://www.i-jim.org https://doi.org/10.3991/ijim.v16i16.33075 mailto:soultana.abdelfettah@gmail.com Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car speeding, drunk driving, reckless driving, running red lights and stop signs, fatigue, weather conditions, road conditions, vehicle defects and so on. The NHTSA reports that in 2018 only, 2800 lives were lost and more than 400,000 people were injured due to distracted driving [6]. According to a research conducted by the American Automobile Association (AAA) Foundation for Traffic Safety, approximately 328,000 drowsy driv- ing accidents occur each year [7]. Distraction impairs driver performance and is a sig- nificant cause of traffic accidents [8, 9]. According to [10], more than 90% of recorded traffic crashes are caused by human error and issues with visual information collection. Furthermore, drivers may be found responsible for 90% of critical traffic circumstances [11, 12]. According to [13, 14], human factors still play a role in 93% of accidents, with perception mistakes accounting for the largest number of errors. Accidents on the road result in not only deaths and disability, but also significant financial losses for victims and their families [15, 16]. Road traffic crashes cost most countries 3% of their gross national product (GNP) per year [17]. Driver inattention refers to any condition or event that causes the driver to lose paying attention to the activity (or activities) most critical for safe driving [18]. This inattention may be due to a deterioration in alertness (fatigue and drowsiness) or it may occur when the driver is engaged in a secondary activity (distracted driver). Driver fatigue and distraction have the same effects: impaired driving skills, longer reaction times, and an increased probability of being involved in a collision [7]. Fatigue reduces a driver’s ability to drive safely by impairing their focus, awareness, attentiveness, and decision-making skills. Continuous fatigue has been shown to cause depreciation in performance comparable to those caused by alcohol [19, 20]. When driving, these symptoms increase the possibility of drivers missing road signs or exits, drifting into other lanes, or leading to an accident [21]. Secondary activities such as eating, drinking, taking something, or listening the radio, as well as the use of cell phones and other tech- nologies, can all induce distraction [22]. Secondary tasks that divert drivers’ attention away from the road ahead [23, 24], obstruct visual scan, or increase cognitive workload can be all extremely dangerous. The driver inattention due to distraction or fatigue is known to be the primary reason for many accidents, according to previous reports. All of the statistics and numbers are alarming, and they require the scientific community’s attention in order to pro- pose smart systems to prevent and avoid accidents. Embedded in current generation of vehicles these systems can be used as a preventive system to monitors the driver’s state and alerts him/her in real time if he/she is distracted or fatigued. For that, various sensors are used to collect data which, thanks to Machine Learning and Deep learning techniques, are analyzed to perform classification or prediction tasks. The advances made in the design of intelligent cars would not have been possible without the help of new technologies such as Internet of things (IoT), Big Data and machine learning algorithms. This later has proven to be one of the powerful methods of data exploration which provide techniques, and tools to efficiently analyze, and interpret the data pro- vided from different sensors. Inattention detection systems can be divided into two main categories: driver perfor- mance-based systems and driver behavior-based systems. The first group uses vehicle indicators such as speed, steering angle, deviation [25, 26, 27], and the second group exploits driver indicators such as physiological and physical signals. The most essential iJIM ‒ Vol. 16, No. 16, 2022 161 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car indicators are eyes and facial movements [28, 29], heart rate variability [30, 31], and brain activity [32]. Due to the significant impact of inattention (fatigue/distraction) on driver performance and the great risks involved this paper provides a survey of studies published between 2014 and 2021 that addresses the context of the driver inattention due to the fatigue or distraction. These latter are rarely addressed together, that is why we are interested to analyze their differences and similarities and their interaction with each other. This study aims to expose the different approaches for driver inattention detection and analyze the general process of these systems from data acquisition, pre- processing to classification, prediction or clustering tasks. According to the type of sensors used to collect relevant information on the driver, many approaches have been proposed and different AI algorithms were used to analyze and process this data. Each approach has some challenges that we will highlight and suggest some directions for future researches. The following is the structure of the paper: Section 2 discusses research methodol- ogy, while Section 3 presents driver inattention detection. The review results are pre- sented in Section 4. Section 5 has a discussion and some prospects. Section 6 presents the review scope’s limitations. Finally, section 7 brings the paper to a close. 2 Research methodology 2.1 Research questions This systematic review aims to recognize the different approaches used for driver monitoring, and to identify the different artificial intelligence algorithms for driver dis- traction and fatigue detection. We aim to recognize what we know in the field of car monitoring systems based on the driver context [33], and to identify challenges to over- come. The comprehensive review of full-text research articles is designed to address the following research questions: RQ1: What are the different approaches used to detect driver inattention (fatigue/ distraction)? The objective of this question is to show the different approaches adopted in the literature to detect driver inattention by specifying in each approach the component(s) on which the measurements were made. This question also highlights the advantages and disadvantages of each technique. RQ2: What are the indicators of driver fatigue? This question deals with the charac- teristics that are used to detect that the driver is tired or drowsy. RQ3: What are the indicators of driver distraction? This question aims to present all the actions performed by the driver that make him/her distracted. RQ4: Which sources are used to collect inattention information and which dataset are used by different approaches? This question focuses on data collection from sensors and datasets used in machine learning models. RQ5: what are feature extraction techniques used in different approaches? This question identifies the feature extraction techniques most often used in fatigue and distraction detection. 162 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car RQ6: What are the algorithms deployed for the detection of inattention? This ques- tion aims to present the algorithms and models used to detect inattention while driving and their performance. 2.2 Search method The population of the systematic review consists of research articles (published between 2014 and 2021) related to the detection of driver inattention. As shown in Table 1, the Boolean “OR” was used to combine alternate terms in each part, while the Boolean “AND” was used to join the three major parts. Table 1. Results based on the source search Search Source String NB. of Papers Search 1 Springer-link Driver AND (Inattention OR Drowsiness OR Fatigue OR Distraction) AND Detection 217 Search 2 IEEE Driver AND (Inattention OR Drowsiness OR Fatigue OR Distraction) AND Detection 256 Search 3 MDPI Driver AND (Inattention OR Drowsiness OR Fatigue OR Distraction AND Detection 22 Search 4 HINDAWI Driver AND (Inattention OR Drowsiness OR Fatigue OR Distraction) AND Detection 286 Search 5 Science direct Driver AND (Inattention OR Drowsiness OR Fatigue OR Distraction) AND Detection 155 Publications found from electronic databases 936 Backward and forward method 112 Total papers found 1038 The articles are not all identified by the 5 searches, as the forward procedure con- siders the references cited on the used articles, and the backward procedure searches for new studies that cited the previously selected article, to identify new publications. A simple methodology, presented in Figure 1, summarizes the steps of the systematic literature review (SLR), as well as the filters used in each part of the SLR. The criteria used in the filtering were: exclusion by repeated articles, title reading, abstract read- ing, full article accessibility, full reading, and quality analysis. The final number was 52 articles. We conducted our initial study on search engines such as IEEE explore, Science-direct, Springer, HINDAWI, and MDPI to extract information relevant to the detection of inattention while driving. The initial search procedure resulted in 1038 research articles. From these, we selected 324 articles based on the title relevant to our study. The abstracts of the selected articles were reviewed, resulting in 100 additional research articles. These articles are then reviewed in depth, and 52 of them are selected for our primary study. The entire selection process is illustrated in Figure 1. iJIM ‒ Vol. 16, No. 16, 2022 163 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car Fig. 1. Steps used during the systematic literature review 2.3 Quality assessment Quality assessment of relevant articles is crucial in systematic literature reviews [34]. Its purpose is to assist with the inclusion/exclusion procedure in order to identify the original studies’ relevance and rigor [35]. According to [34], there is no standard definition for determining the quality of primary research. The majority of this eval- uation, however, is based on a quality check-list that examines each research paper separately. In this regard, the authors designed a checklist based on a series of four questions: QA1: The study was presented at a recognized conference, symposium, or work- shop, or it was published in a peer-reviewed journal. For conferences, work- shops and symposia: (+1.5), for journals: (+2) QA2: The method relies on sensors to collect data about inattention (fatigue and dis-traction). There are two possible answers: “yes: +1”, “no: 0” QA3: The methods utilized are described in detail. There are three possible answers: “yes: +1”, “no: 0” and “partially: 0.5” QA4: The approach is developed on the basis of AI algorithms. There are two pos- sible answers: “yes: +1”, “no: 0”. There are 52 articles in total, of which 28 studies address the issue of driver fatigue and 24 studies address the issue of distracted driving. There are 15 articles presented at international conferences and 37 papers published in international journals. There are 17 papers in the first quartile (46%) and 14 in the second (37%) quartiles, as well as 3 papers in the third quartile (8%), and three papers that have not yet been assigned to a quartile (8%). 164 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car 3 Driver inattention detection In this section, we have carried out two state of the art studies distinguishing between inattention caused by states of fatigue or distraction since in both cases the detection is done in a specific way. 3.1 Driver fatigue detection Fatigue is a term that refers to a general feeling of tiredness and/or a lack of energy. Several approaches have been proposed to highlight symptoms of driver fatigue. The latter can lead to states of sleepiness, drowsiness, yawning, tilting the head forward and so forth. Determining the state of the driver through external signs is a great challenge for researchers. In this section we will present an overview of selected papers dealing with driver fatigue based on some facial features, such as eyes closure, mouth open- ness, head angle, etc. [36] proposed a system for monitoring driver fatigue through yawning detection based on Support Vector Machine (SVM). Mouth state detection is based on Circular Hough transform (CHT). The system here assumes that the driver is in a yawning state if a significant number of consecutive frames found where the mouth is wide open found. The system performance accuracy was up to 98%. [37] Presented an eye blink tracking algorithm that employs eye feature points to assess whether the eye is open or closed as well as an alarm if the driver feels drowsy. The Viola Jones Cascade classifier was operated to recognize eyes and the Harris Corner Detector was applied on the input eye image to obtain three landmarks’ points. The suggested technique reached an accu- racy of 94%. [38] Advocated drowsiness detection system using Linear Discriminant Analysis (LDA) classifier using Heart Rate Variability (HRV) signals obtained from an electrocardiogram (ECG). One of the limitations of the study is the reference signal used for the drowsiness episodes detector and the complexity to identify precisely the beginning and the end of drowsiness episodes. [39] Proposed a real-time fatigue detec- tion system that achieves an accuracy of 98% using Viola and Jones algorithm. Based on the driver’s eye closure time (PERCLOS) and yawning behavior, the given system tracks the driver’s eyes to detect when they are closed or half-open for more than 5 seconds, or when the driver yawns with closed or half-open eyes. In this situation, the system immediately concludes that the driver is fatigued, and an alarm is generated to alert them. The infrared camera works well at nighttime but the performance decreases in daytime. [40] Presented a method for driver fatigue detection based on eye state recognition (PERCLOS and the blink frequency parameter). The face detection from an infrared video was conducted with AdaBoost algorithm and a Convolution Neural Network (CNN) which helped classify the eye state. The adopted method achieved an accuracy of 98.7%. This can also work in condition of wearing glasses. [41] Proposed a method for detecting driver fatigue based on EMG and ECG data acquired in real time by a smartphone and non-contact sensors in a driver’s seat cushion. The model of driver fatigue is built based on Principal Components Analysis (PCA) which reached an accu- racy of to 91%. [42] put forward a method to identify the driver fatigue symptoms using CNN and transfer learning technique (AlexNet). The three features, namely furrowing iJIM ‒ Vol. 16, No. 16, 2022 165 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car the brow, narrowing the eyes and Yawning were exploited. The best results were obtained for the developed detector of yawning with an accuracy of 98.28%. [43] Advanced a non-intrusive system that tracks eye movements to detect driver fatigue. The simula- tion driving experiment and the D-Lab surveillance system collected the driver’s eye movement features, and a fatigue driving model based on the fuzzy K-nearest neighbor (FKNN) algorithm was constructed to detect driver weariness which reached an accu- racy of 89%. A method for identifying fatigue status using the spatial–temporal feature of the driver’s eyes was proposed in [44], to extract the eye region from infrared videos, the authors used a deep cascaded multi-task framework. The classification achieved a 95.83% by combining convolutional layers with long and short-term memory (LSTM) units, which are capable of learning spatial representations and modeling temporal dynamics. [45] Proposed a driver fatigue detection system based on transfer learn- ing AlexNet model that performed an accuracy of 90%. The accuracy of seven EEG channels was measured using the EEG sensor, and the most accurate one was chosen. The results show that the channels FP1 and T3 are the most effective channels able to indicate the drive fatigue state. [46] Recommended a driver fatigue detection algorithm based on multi-facial features and two-stream network models. The algorithm consists of four parts: A multi-task cascaded CNN (MTCNNs) to locate the mouth and eye, a partial facial image to extract the static features, a partial facial optical flow to extract dynamic characteristics, and combining both static and dynamic features to make the classification. An accuracy of 97.06% was reached on the NTHU-DDD dataset. [47] Presented a driver monitoring algorithm using video captured from the camera located in the vehicle to detect whether the driver’s head is present in each frame of the video. The algorithm is based on the face and eye detectors pre-trained according to Viola and Jones algorithm. The algorithm examines three cases, namely whether the driver’s eyes are closed while driving, or whether the driver looks sideways, and whether the driver’s head has dropped for a long time. The detection of the proposed algorithm reached an accuracy of 80±3%. [48] Proposed a drowsiness monitoring method based on steering wheel status. A driving simulator was adapted to collect eleven parameters related to the steering wheel, where four parameters having significant correlations with driver sta- tus (Ellipse, Amp_D2_Theta, NMRHOLD, and SW_Range_2). A Multilevel Ordered Logit (MOL) model, SVM model and back propagation Neural Network model were built based on the selected parameters. [49] Developed a fatigue detection algorithm based on facial expression. Firstly, Multi Block Local Binary patterns (MB-LBP) and Adaboost classifier are trained to detect face key point (24 facial features). Secondly, the fuzzy inference system is utilized to detect the driver’s fatigue state (normal, slight fatigue, severe fatigue) based on the unit time (PERCLOS) and yawning frequency. An accuracy of 96.5% was achieved. 3.2 Driver distraction detection In the literature, there are several definitions of driver distraction as a type of inattention. Driver distraction is defined as follows: “a diversion of attention from activities essential for safe driving to a competing activity” [18]. The driver is distracted when they give insufficient attention to their surrounding environment 166 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car (road, other drivers, etc.) by engaging in a secondary task when driving. Usually that involves conversation with the passengers, receiving or sending a message, answering the phone or making a call, using in-vehicle information systems, such as navigation system and radio, smoking, eating or drinking something, applying makeup and so forth. The studies on distraction detection are diverse and different systems are proposed to detect the secondary tasks responsible for making the driver distracted and losing their concentration. [50] Proposed a new head pose descriptor as an indicator of visual focus of attention, resulting from the fusion of four of the most relevant orientation-based head descriptors, namely steerable filters, histogram of oriented gradients (HOG), Haar features, and an adapted version of the speeded up robust feature (SURF) descrip- tor. Based on visible spectrum camera data, the SVM classifier was used to assess head pose variations with an accuracy of 97.5% for pitch and 98.2% for yaw. [51] Experimented two transfer learning models Inception ResNet and Mobile Net to detect driver distraction. This study was conducted on a small dataset (4000 images) where images were captured using USB webcam and the accuracy achieved by MobileNet was 94.1%. [52] Focused on ECG signal processing aspect with the aim of predicting driver distraction such as an active engagement on a phone conversation between the driver and the passenger. They used a Wavelet Packet Transform (WPT) to localize the impact of distracting elements. Due to high dimensionality of the WPT generated space, they applied LDA for feature space dimensionality reduction which reached an average prediction accuracy of 88.45%. [53] Presented a transfer learning model VGG- 16 based system that not only detects the distracted driver but also identifies the cause of distraction. They used a viz dataset with 17308 images including 10 classified activ- ities, namely safe driving, talking on mobile phones using right or left hand, texting on mobile phones using right or left hand, adjusting radio, eating or drinking, hair and makeup, reaching behind and talking to passengers. The method achieved an accuracy of 96.31%. A thinned version of VGG-16 with just 15M parameters was also proposed and achieved an accuracy of 95.54%. [54] On the other hand, employed a kinect camera installed inside a vehicle to track drivers and detect distracted driving. Seven different distracting tasks was studied: normal driving, checking the left, right, and rear-view mirrors, answering the phone, texting with one or both hands, and using in-vehicle video devices. The seven tasks reached an accuracy of 80% using a Feed Forward Neu- ral Network (FFNN). [55] used a distraction detection system based on the spectrogram and MEL Cepstrum representation of GSR signals and a CNN model to quantify the influence of secondary tasks such as calling and texting on the driver. The suggested method identifies distraction with 93.28% accuracy. [56] Elaborated a reliable deep learning-based solution (a genetically weighted ensemble of CNN) that recognizes dis- tracted driving postures which achieved an accuracy of 90%. They also presented a new publicly available dataset for driver distraction with more distraction postures and extracted 14,478 frames using a single camera distributed over the following classes: safe driving phone right, phone left, text right, text left, adjusting radio, drinking, hair, reaching behind, and talking to passenger. [57] proposed a method to detect distracted drivers using their cell-phone. The Inception-V3 model was implemented on a dataset of 85,401 images, achieving an AUC of 0.891. [58] proposed a driving-related activity recognition system, which are the normal driving, rear mirror checking, right mirror checking, left mirror checking, using in-vehicle radio device, texting, and answering iJIM ‒ Vol. 16, No. 16, 2022 167 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car the mobile phone, based on the deep learning models AlexNet, GoogLeNet, and Res- Net50. The experimental images are collected using a low-cost camera, the raw RGB images are first processed with a GMM-based segmentation algorithm to extract the driver body from the background before training the models. The AlexNet obtained an average of 81.6% detection accuracy for the seven tasks, whereas the GoogLeNet and ResNet50 achieved 78.6% and 74.9% detection accuracy, respectively. [59] investi- gated the use of a deep learning approach to automatically recognize in a single image driving behavior such as normal driving, calling, playing mobile phone, driving with hands off the wheel, smoking and talking with passengers. The approach consists of two steps: (1) Employ multi-stream CNN to extract multi-scale features and (2) investi- gate different fusion strategies to combine the multi-scale information and generate the final decision for driving behavior recognition. The main disadvantage of this method is that the solution is not effective in video-based recognition, due to no consideration of motion information and the accuracy rate was of 86.6%. [60] proposed an algorithm for driver cell phone usage detection that is based on deep learning. The algorithm includes two steps: face detection and face tracking using Progressive Calibration Net- works (PCN). Then, we determine the calling detection area. The driver’s cell phone detection method is used to identify the candidate area. The algorithm performed well in simulations under different lighting conditions, with an accuracy of 96.56%. By recognizing and positioning the driver’s right hand and right ear, [61] proposed an image-based driver distraction recognition method. The framework location is made up of two modules: the first predicts the bounding boxes of the driver’s right hand and right ear from RGB pictures using YOLO calculation, and the second module takes the ROIs of the ear and hand as input and develops a multi-layer perceptron (MLP) to infer the driver’s status from the ROIs. The algorithm achieved an accuracy measure of 82% with a set of 106,677 frames extracted from recordings. [62] proposed a method based on CNN to recognize driver use of cell phone (cell-phones and hands), A multi-angle arrangement of cameras are used to improve the integrity of image acquisition and to ensure the detection accuracy of the target recognition in which an accuracy of 95.7% was achieved. [63] developed an automated supervised learning method called Drive- Net for driver distraction detection based on a combination of a CNN and a Random Forest (RF). The methods were validated on a publicly available database of images acquired under controlled conditions. This database contained about 22425 images, the accuracy rate was of 95%. 4 Review results This section highlights the results of research questions related to the detection of driver inattention due to fatigue or distraction. In particular, we are interested in: 1) identifying the different approaches to inattention detection; 2) determining the indicators used for detection in the case of fatigue and distraction; 3) investigating the sources and datasets used to train the different models; 4) identifying the feature extraction techniques used and their impact on the models; and finally 5) reviewing the machine learning, deep learning and transfer learning techniques used to build the classifiers with the corresponding performances. 168 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car 4.1 Approaches for driver inattention detection (fatigue/distraction): RQ1 Based on state of the art we established four approaches used to identify driver inattention according to how the data are collected: 1) physical approach, 2) the physi- ological approach, 3) the driving performance approach, and 4) the hybrid approach. As shown in Table 3, the first approach is based on the driver facial measures. It occupies an important place in the literature because it can be exploited in real applications and it is low cost. It basically requires a camera for recording the driver. This approach allows to extract the symptoms of fatigue and identified the distraction task from the state of the head, mouth, eyes and hands. The second approach is based on the analysis of physiological signals such as EEG (brain activity), ECG (heart activity), EMG (muscle activity), respiratory rate, etc. In this approach, the measurement of driver fatigue and drowsiness is performed by attaching electronic devices such as sensors to the driv- er’s body. The third approach is based on driving performance, such as steering wheel movement, gear change, brake pedal and accelerator pedal deflection as indicators of driver fatigue. And finally, the hybrid approach that combines these approaches. Table 2. Details of approaches used to detect driver fatigue Approach Used Sensors Used Intrusive Component Paper ID Physical Camera No Face [75] Mouth [36] Eyes [37, 77, 83, 89] Eyes and mouth [49, 65, 67, 81] Eyes and head orientation [47] D-Lab system No Eyes [43] IR Camera No Eyes and mouth [46] Eyes [40, 44, 74] NIR Camera No Eyes and mouth [39] Eyes and mouth, brow [42] Physiological Cushion driver’s seat sensor No EMG signal, ECG signal [41] Doppler radar, smart bracelet No Respiration, heartbeat signals [64] ECG recorder Yes ECG signal [38] EEG sensor Yes EEG signal [45, 82] Driving performance Steering wheel sensor No Steering wheel [48, 88] Multi-channel camera No Lane position [85] Hybrid Camera, EEG, ECG sensors Yes EEG signal, ECG signal, eyes [86] EEG sensor, gyroscope, camera Yes Head orientation, eyes, EEG signal [84] iJIM ‒ Vol. 16, No. 16, 2022 169 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car Table 3. Details of approaches used to detect driver distraction Approach Used Sensors Used Intrusive Component Paper ID Physical Camera No Hands [57, 62] Hands, face [56, 79, 80] Hands, right ear [61] Head [50, 93] Head, hands, face [53, 54, 58, 59, 63, 66, 78] Right hand, right ear [76] IR Camera No Hands [60] USB webcam No Head [51] Physiological A wearable channel network No Brainwaves [92] ECG recorder Yes ECG signal [52] Wireless GSR wearable device No Hands, ears [55] Driving performance Lane Tracker, radar, accelerometer No Vehicle speed, lateral velocity, lateral displacement [90] STISIM simulator No acceleration, deceleration, lane position, steering angle, vehicle heading angle, speed [87] Controller Area Network (CAN) bus No Vehicle speed, Steering Wheel [91] In terms of detecting fatigue, 69% of research used a physical approach, 15% used a physiological approach, and two studies used a hybrid method. For the detection of distracted driving, 81% of the papers are based on the physical approach, 13% of the papers are based on the physiological approach. The Figure 2 summarizes the percent- age of use of the different approaches. We can see from Figure 2 that the physical approach dominates the other approaches for both fatigue and distraction detection, followed by the physiological approach, and the hybrid approach. We can explain that from the fact that the physical approach is non-intrusive, and data are easier to collect, but the physiological approach suffers from its intrusive nature, and needs more involvement from drivers which makes it not easy to implement and the sensors used are very expensive. The driving perfor- mance approach is interesting and is not enough explored in literature, although it is not intrusive. It is strongly influenced by the road condition, the type of car, as well as by the weather conditions that make drivers less confident and studies shown a high false alarm rate. However, the change in driving behavior and performance can be a strong sign of fatigue or inattention rather than irresponsible driver behavior or road conditions. 170 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car Fig. 2. Percentage of use of the different approaches The Table 4 sum up the advantages and disadvantages of each approach for deriver inattention detection. Table 4. Advantages and disadvantages of each approach Approach Advantages Disadvantages Physical approach Low cost Practicable Effective Non-intrusive Lighting conditions Wearing glasses Wearing sunglasses Feature extraction is difficult Distance from camera Physiological approach Accurate Represent the true internal state of the driver. They change in the very early stages of drowsiness Intrusive Costs of sensors are too high Sensitive to driver movement Signals susceptible to noise Driving performance approach Non-intrusive Robustness of the data acquisition Weather conditions Driver experience Individual differences Road geometry, Road quality Hybrid approach Reducing the probability of false alarm Execution time Costs of sensors These common methods of driver inattention detection have both advantages and disadvantages. That is why hybrid driver measures are expected to provide more reli- able solutions that will both accurately detect driver inattention and minimize the num- ber of false alarms However, portable sensors may offer an important opportunity to adopt the physiological approach in real application. On the other hand, the combina- tion of the physical approach and the driving performance approach could intuitively increase confidence in the detection of inattention, whereas the accuracy of the physio- logical approach could be used for the validation of the proposed systems or to employ it in a non-intrusive manner. iJIM ‒ Vol. 16, No. 16, 2022 171 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car 4.2 The indicators of driver fatigue: RQ2 Many indicators were used to detect driver fatigue depending on the approaches used. In this section we collect all indicators used from literature studied for driver fatigue detection. As we can see the most studied features are eye closure, yawning and blink behavior. The Table 5 shows clearly that eyes play an important role in fatigue detection using eye closure 25%, blinking behavior (17%), and pupil diameter/area (4%). The mouth was used for detecting yawning frequency (19%). At last 8% of papers was based on the EGG channels. Table 5. Different indicators used for driver fatigue detection Features Type of Approach Paper Id Respiration rate Physiological [64] Heartbeat [64] EEG channels [45, 82, 84, 86] Complexity of EMG/ECG [41] SampEn of ECG [41] HRV features [38] Blink behavior Physical [37, 40, 44, 81, 83, 84, 86, 89] Eye closure [39, 40, 42, 44, 46, 47, 49, 65, 67, 74, 77, 86] Pupil diameter/area [43, 83] Gaze zone [83] Driver face [47, 75] Head nodding [47, 84] Yawning frequency [36, 39, 42, 46, 49, 65, 67, 81, 84] Fixation durations [43] Furrowing the brow [42] Steering wheel parameters Driving performance [48, 88] Lane position [85] As one of the most important features of the face, eye movements can play an important role in expressing the physical and mental condition of the driver for long term driving. In particular, there are certain evaluation criteria such as the percentage of eye closure (PERCLOS), which is calculated by counting the number of frames in which there was no pupil detected, and dividing this by the total number of frames for a specific time interval. PERCLOS is a temporal ratio over a given time window. It is defined as the percentage of time in which at least 80% of the eyelid remains closed over the pupil. Accordingly, the Eye blink rate measures the blinking rate frequency of eye-blinks to detect fatigue and drowsiness. The normal blinking rate per minute is roughly 10, but when the driver is drowsy, the blinking rate decreases. AECS (average eye closure speed) is a means to measure the amount of time needed to fully close the eyes and to fully open them, drowsy person, here, will blink distinctly slower than the alert person. In second rang, Yawning is also an important facial feature to detect driver 172 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car fatigue. Yawning is the reflex act of opening your mouth wide and inhaling deeply due to fatigue or drowsiness. It can signify that a driver is about to fall asleep behind the wheel. By tracking the shape of the mouth and the position of the lip corners, methods can be used to detect yawning traits in the driver. Researchers extracted and examined the driver’s mouth features using image recognition technology to determine whether they were yawning. The number of yawns per unit time and the degree of mouth open- ing after each yawn were used to determine the driver’s fatigue level. 4.3 Indicators of driver distraction: RQ3 In distraction driver is involved in secondary tasks that can be visual, cognitive or manual. The term “manual distraction” refers to tasks that need the driver’s hands being taken away from the steering wheel., e.g., drinking, eating, using cellphone, adjusting vehicle devices like the GPS or radio, etc. The visual distraction is the type of distrac- tion that takes the driver’s eyes and focus off road, even for mere seconds, e.g., viewing text messages, watching a video, etc. At last, the cognitive distraction where the driv- er’s mind is not fully focused on the driving task, e.g., talking, thinking, daydreaming and so forth. The three types of distractions can occur separately or at the same time. Texting, for example, is a visual, physical, and cognitive distraction because it requires the driver’s vision, hands, and mind. However, not only the types of distractions, but also the frequency and duration of distracting acts affect the risk of a crash caused by distracted driving. As shown in Table 6, calling and texting are the most distraction studied in research papers. Studies have focused on the detection of the secondary task but few if any have investigated the level of risk induced by each type. All secondary tasks have not the same level of severity, and texting, looking behind or calling is more dangerous than smoking or talking. Table 6. Different indicators used for driver distraction detection Secondary Task Type of Distraction IDs Calling Manual, Cognitive [52, 53, 54, 55, 56, 58, 59, 60, 61, 63, 66, 76, 78, 79, 80, 92] Texting Manual, Visual, Cognitive [52, 53, 54, 55, 56, 57, 58, 59, 61, 62, 63, 66, 76, 78, 79, 92] Drinking/eating Manual [56, 61, 63, 66, 76, 78, 79] Smoking Manual [59, 80] Talking to passengers Visual [52, 56, 59, 63, 66, 78, 79] Reaching behind Visual [56, 63, 66, 78, 79] Adjusting radio Manual, Visual [53, 56, 63, 66, 78, 79, 90] Hair and Makeup Manual, Visual [56, 58, 63, 66, 78, 79] Touchscreen Manual, Visual [61, 76] Moving object Manual, Visual [76] Using on board CD Manual, Visual [90] Using navigation system Manual, Visual [90] Mirror checking Visual [53, 54, 58] Handsoff the wheel Manual [59] Head orientation Visual [50, 51, 76] iJIM ‒ Vol. 16, No. 16, 2022 173 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car 4.4 Data sources/dataset: RQ4 Data sources. Sensors play an important role to capture data from the driver while driving. These include cameras that can be used to monitor the driver’s facial expressions as well as other activities such as secondary tasks while driving. The position of the cameras is determined by the intended result. If the goal is to detect fatigue, the camera should focus on the driver’s facial signs; if the goal is to detect distraction, the camera should monitor all secondary tasks using the hands, eyes, and head, among other things. Various sorts of cameras are utilized, according to the Table 3, including simple cameras, infrared cameras (IR), and near-infrared cameras (NIR). However, one of the most prominent drawbacks of computer vision systems is lighting. The ability to maintain huge changes in light intensity, from bright sunlight to nighttime driving, is a difficult task. One solution for ensuring operability in low light conditions is to use an infrared camera as a sensor. Besides, sensors are used for physiological measurements of the driver in order to measure brain activity (EEG), heart activity (ECG), ocular activity (EOG), muscle activity (EMG) and tension and others. ECG signals capture vital information concern- ing a driver’s fatigue which including heart rate, heart rate variability, and respiration rate. Heart Rate (HR): When people transition goes from alert to sleepy, their HR (heart rate per minute) decreases [68, 69]. HR is influenced by emotions, mental activity, and physical exercise [70, 71]. HRV (Heart Rate Variability), also known as RRI, is the variation in the time gap between two consecutive heart beats. HRV can detect changes in the autonomous nervous system (ANS) activity caused by fatigue or stress [72, 73]. The respiratory rate is defined as the number of exhaled and inhaled breaths per minute (RR). These measurements are the most reliable ones. However, they have to be placed directly into the driver’s body which could make the driver feel uncomfortable since they limit its movements in the car. EEG activity is characterized by separating the frequencies into bands, referred as alpha (8–13 Hz), beta (13–30 Hz), delta (0.5–4 Hz), theta (4–8 Hz), and gamma (30–42 Hz) rhythms [20]. Alpha, beta, delta, theta, and gamma rhythms all drop after fatigue, with the beta and gamma rhythms decreasing considerably. Although the EEG sensor has 32 channels, and most studies used only a subset of them rather than all. For example, [45] worked on 7 channels, [65] on 8 channels, [66] on 2 channels only, and [67] on 16 channels, implying that a thorough investigation of all channels is required to determine the minimum necessary of channels that accurately detect driver fatigue and distraction, as well as cognitive distraction. Table 7. Studies based on the EEG signal ID N. of EEG Channels Channels [45] 7 FP1, FP2, T3, T4, O1, O2, Oz. [65] 8 C3, C4, P7, P8, O1, O2, FP1, FP2 [66] 2 Cz, T7 [67] 16 Fp1, Fp2, C3, C4, F3, F4, F7, F8, P3, P4, O1, O2, T3, T4, T5, T6 Datasets. In this section, we present various public datasets used in research studies for driver fatigue and distraction detection. Tables 8 and 9 present public datasets for 174 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car fatigue and distraction detection, respectively. More details are depicted for each dataset such as the size, the resolution, the number of classes, and the number of participants in the experiment, as well as the original EEG data for driver fatigue detection, which contains EEG data based on a 40-channel Neuroscan amplifier. Table 8. Public datasets for driver fatigue detection Dataset Type Size Resolution/ Amplifier N. of Classes N. Participants Paper ID TJPU-FDD Video infrared camera 500 video clips of 6 second 1920 × 1080 fatigue and normal 26 [44] NTHU-DDD Video infrared camera Nine and a half hours 640 × 480 drowsy and non-drowsy 36 [65, 74, 75] YawdDD Video RGB camera 322 video clips The videos last between 15–40 seconds 640 × 480 normal, talking or singing, and yawning 107 [46, 65] EEG data for driver fatigue EEG data 545.91 MB 40-channel Neuroscan amplifier fatigue and normal 12 [45] Table 9 shows the public datasets used for detecting driver distraction, including the type of data (images or videos), the number of classes (number of activities that dis- tract the driver), the dataset size, and the number of participants in the data collecting experiment. Table 9. Public datasets for driver distraction detection Dataset Type Size Resolution N. of Classes N. Participants Paper ID Distracted Driver Dataset Videos RGB camera 14478 images 1080 × 1920 10 44 [56] Seu-Driving Videos 17730 images 640 × 480 4 20 [59] Kaggle-Driving (State Farm) Videos 4662 images 640 × 480 10 – [59, 66] VIVA hand tracking Videos Microsoft Kinect device 2000 images 16 × 16 19 different dynamic hand gestures 8 [61, 76] We note that 37.5% of studies are based on public datasets, which is very important for research work to be compared in terms of performance and precision. Only limited amounts of annotated data for face feature recognition, and driver secondary task iden- tification are publicly available. Furthermore, these datasets are generally well illumi- nated scenes. These datasets cannot describe the real-world challenges on the road in some cases such as occlusions and difficult lighting. Hence there is a need for a large data set containing features on inattention indicators such as: several facial, physiologi- cal and vehicle indicators, with a variety of subjects, a variety of lighting, and different driver postures, and several head inclinations. iJIM ‒ Vol. 16, No. 16, 2022 175 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car Simulated environments are although used to reach more than feature and to be closer to the real world. The key benefits of employing simulators include experimental control, efficiency, cheap cost, safety, and data collecting convenience. One serious disadvantage of utilizing driving simulators is that the drivers are unaware of any dan- ger. Being conscious that you are in a simulated environment may cause you to behave differently than you would on the road. However, because a moving car might provide obstacles such as changes in illumination, background and vibration noise, as well as the usage of sunglasses, hats, and other accessories, the results may be drastically dif- ferent in real-world driving settings. 4.5 Feature extraction techniques: RQ5 The process of defining a set of features, or image characteristics that will most effectively or usefully represent the information needed for analysis and classification is known as feature extraction. To identify driver inattention, several techniques are uti- lized to extract valuable data from images/video or data generated by different sensors. To extract fatigue features based on the physical approach, the driver’s face must first be detected in a set of images of the dataset or frames from the video. In this review three algorithms are frequently used for the detection of the driver’s face and also for face alignment: MTCNN, viola and jones algorithm, DLIB and LBP. Viola-Jones was created for frontal faces; therefore, it detects them better than faces that are facing side- ways, upwards, or downwards. After locating the driver’s face, the next step is to detect and track the location of the eyes, mouth, nose and other facial features, various methods used for this tasks such as Viola-Jones that is used in [37, 39, 47, 77, 94, 95, 96], which is one of the most suc- cessful tools for object recognition based on the Haar cascade feature approach and is frequently used for facial feature extraction (location of eyes, nose and mouth on face) and face detection. This algorithm uses 4 main factors namely the Haar feature, Inte- gral Image, Cascade Classifier, and Adaboost machine-learning. Also, the frontal face detector provided by DLIB is used in [74] by extracting features from the histogram of oriented gradients (HOG), which are then passed through an SVM is used to estimate the location of 68 coordinates (x, y) that map the facial points on a person’s face. It can detect and describe important facial features such as: eyes, eyebrows, nose, mouth and jawline). MTCNNs or Multi-Task Cascaded Convolutional Neural Networks is a neural network which detects faces and facial landmarks on images is used in [44, 46, 65], is one of the most popular and accurate face recognition software available today. It’s made up of three neural networks coupled in a cascade: P-Net, R-Net, and O-Net. MTCNNs can detect five key points of the face: left and right corners of the mouth, nose, and left and right eyes. In terms of speed, DLIB seems to be the fastest algorithm, followed by Viola-jones classifier and MTCNNs. However, MTCNNs tend to be the most accurate algorithm. DLIB perform pretty well but have some issues identifying small faces. For Viola-Jones is real-time on the CPU, but there have been a lot of incorrect predictions. In spite of the high accuracy and real-time facial detection with MTCNNs, the training time may take longer. 176 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car Several techniques were used to extract features from images or video frames in both cases inattention and distraction. CNN models and CNN pre-trained models are the most models used to extract useful characteristics from an image automatically. Pretrained CNN architectures, include AlexNET [42, 45, 58], ResNet50 [58, 78], and VGG-16 [53, 79], as well as Xception, GoogleNet, Inception-V3 [78], and YOLOV3 [76, 80, 81]. In the physiological approach, a variety of signal feature extraction methods are utilized in practice, with WPT being the most commonly employed in the selected research. The time domain features of the studied channel EEG were extracted using the Wavelet Packet Transform (WPT) [52, 82]. The WPT transforms the time-amplitude representation of a signal into a time-frequency representation stored as a sequence of wavelet coefficients. In the driving performance approach, the Approximate Entropy (ApEn) was employed to extract valuable features from the steering wheel angle in a driving performance approach [88]. Approximate Entropy is a non-linear dynamic quantity that assesses the irregularity of time series data. Based on the studies that were selected, Figure 3 depicts the most feature extraction techniques used. Fig. 3. Algorithms used for features extraction As shown in Figure 3, the most common feature extraction approaches employed in selected research for driver fatigue detection are CNNs and viola-jones algorithm. CNN models, particularly its pre-trained models, are frequently utilized for feature extraction, which identifies distracted activities such as phone calls, eating, texting, and smoking. Yolo, AlexNet and so on are pre-trained models on huge data and are used to build models to solve a similar problem. Their use is particularly effective for gen- erating accurate models with a minimal quantity of data, when training a model from scratch is impractical. 4.6 IA algorithms for detection of inattention while driving: RQ6 The current section discusses several Artificial Intelligence techniques for improv- ing driver fatigue and distraction detection. These approaches can be classified as machine-learning, deep learning, transfer-learning, and others methods. Most of inves- tigations are based on supervised learning techniques that uses labeled data. iJIM ‒ Vol. 16, No. 16, 2022 177 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car Tables 10 and 11 present the different algorithms used for fatigue and distraction detection, the performance reached and the datasets used for training the model. The algorithms are classified as machine learning (ML), deep learning (DL) and transfer learning (TL). The third column presents the dataset used to build the models, and the others describe the paper Id and the performance of the models. We can remark that the literature is very rich in terms of IA algorithms deployed for driver inattention detection. We can also notice that CNN and SVM are the most used techniques in fatigue and distraction detection. The CNN based methods are known to be accurate in extract- ing useful features from images. Pre-trained models are increasingly used, and present attractive performances. Their power comes from the fact that these models are trained on large amounts of data and are used as a starting point to train other models on datasets with limited amount of data. The SVM and MTCNN algorithm are widely used as classi- fiers in driver inattention detection purpose, and the performance achieved is the highest. Table 10. Algorithms used for driver fatigue detection Algorithm Type Datasets Paper Id Accuracy SVM ML 6 videos (305 seconds) 3892 samples FDD 4320 samples 2536 samples [36] [48] [64] [83] [77] 98% 86.03% 91.2% 85% 96.5% ANN 6000 images, EEG data, 5000 instances – [84] [85] 93.91%; 90% Adaboost Caltech10k Web Faces, FDDB 2536 processed frames [49] [77] 96.5% 95.4% Fuzzy K-NN – [43] 89% Multiple Linear Regressions – [41] 91% Random Forest 2847 rows – [86] [87] 87.7%; 85.38% Extra Trees physionet EEG, simulated virtual driving driver. [82] 85.3% BPNN SWA [88] 88.02% CNN DL IRF [40] 95.81% CNN-LSTM 1042 images FDDB, VOT100 [81] [67] 91:7% 80.33% CNN-BILSTM 2208 images [89] 94% MTCNN NTHU-DDD YawdDD, NthuDDD [46] [65] 97.06% 98.81% MTCNN-LSTM TJPU-FDD [44] 95.83% ALEXNET TF 1280 images; The original EEG data for driver fatigue detection [42] [45] Misclassification rate is 5.5% 82.27% YOLO-V3 1042 images [81] 91.7% LDA other SDB, ADB, RDB [38] 96% 178 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car Machine learning techniques such as SVM, ANN, Adaboost, fuzzy K-NN, multi- ple linear regressions, Random forest, Extra-Trees, and BPNN are all used to identify driver fatigue with varying degrees of success. Deep learning models, such as CNN, LSTM, BILSTM, and MTCNN, are used to detect driver fatigue, as well as trans- fer learning, which is based on AlexNet and YOLO-V3. We also mention that based on physiological features, the algorithm LDA gives good results (96% accuracy) for detecting driver fatigue. Table 11. Algorithms used for driver fatigue distraction Algorithm Type Datasets Ids Accuracy SVM ML Pointing’04; 300 points of IVIS operating – [50] [90] [91] 97% 89.9% 95% FFNN – [54] 80% CNN DL 22,216 images; ILSVRVC2012; Distracted Driver Dataset; 85,401 images; 44000 images; 1764 samples; stateFarm [92] [51] [56] [57] [62] [55] [66] 96.56% 94.1% 90% 92.8% 95.7% 93.28% 99% multi-stream CNN Seu-Driving, Kaggle-Driving [59] 93.2% CNN-RF DL/ML 22425 images [63] 95% YOLO, MLP VIVA hand tracking; 106,677 frames, VIVA, hand tracking [61] [76] 82% 59% VGG-16 TF 17308 images;22,420 images [53] [79] 95.54% 82.5% ResNet50, Inception-V3 Xception 102250 images [78] 96.74% AlexNet, GoogLeNet ResNet50 33394 images [58] 91.4% Game Theory other – [93] 90.12% LDA 1765 samples [52] 95.51% Threshold based algorithm 65 samples (9 scenarios) [43] 98.6% As shown in Table 11, SVM provides hight accuracy in detecting driver distraction (97%), and we note that CNNs and the pre-trained models are the most commonly employed to identify various secondary tasks while driving and provide a good accu- racy (more than 90%). We also highlight the Threshold-based approach, which has 98.6% for accuracy rate for detecting driver distraction. The best accuracy obtained for each approach for the detection of inattention is shown in Table 12. For the physical approach, the MTCNN algorithm provides the highest accuracy for the fatigue detection and Threshold algorithm presents the highest accuracy for the distraction detection, both with an accuracy of more than 98%. In the physiological approache, we find that linear discriminant analysis (LDA) gives the best accuracy for both fatigue and distraction; and for driving performance approach SVM gives a good result, with 86.03% for wea- riness and 89.9% for distraction. The hybrid approach achieved an accuracy of 93.91% using ANN and no selected research focus on hybridization to identify distraction. iJIM ‒ Vol. 16, No. 16, 2022 179 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car Table 12. Best accuracy for every approach on both fatigue and distraction Approach Fatigue Distraction Physical MTCNN, 98.81% [20] Threshold based algorithm, 98.6% [43] Physiological LDA, 96% [5] LDA, 95.51% [38] Driving performance BPNN, 88.02% [90] SVM, 89.9% [39] Hybrid ANN, 93.91%, [21] – 5 Discussion In this SLR we highlight three alternative methods to identify inattention while driv- ing, whether it is fatigue or distraction. The physical approach is based on the analysis of images from different cameras that monitor the driver’s facial features for fatigue detection and driver secondary tasks performing while driving. The physiological approach uses measurements of the driver’s internal state, such as brain activity, heart rate, and respiratory rate. Therefore, it is necessary to take advantage of this approach by placing sensors on the steering wheel, in the seat belt, or in another area that does not interrupt the driver or even use wearable sensors such as bracelets. The driving performance approach is another method used for detecting driver inattention based on measurements such as steering wheel angle, lane tracking, and others. The hybrid approach it is an interesting track to dig because the physiological measurements bring clear results on the internal state of the driver and thus when detecting driver inatten- tion, a hybrid system reduces false alarms, increasing system dependability and ensur- ing system endurance even when one approach fails. Another result of this SLR is to propose a baseline of the different indicators used in the different approaches for driver inattention detection. The Figure 4 summarizes the different indicators used according to the type of approach employed. Fig. 4. Taxonomy of different approaches used for driver inattention detection system 180 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car Physical ways include yawning, blinking, closing one’s eyes, pupil diameter, gaze area, and head nodding. The physiological method includes things like shallow breath- ing, heart rate variability, and EEG channels. Driving performance approaches include lazy steering, incorrect gear shifts, and drifting in and out of lanes. For the fatigue detec- tion, we notice a wide use of the so-called percentage of eye closure (PERCLOS) which gives strong indications of fatigue and drowsiness. Combined with frequent yawning, we can attain a good indicator for the detection of driver fatigue. The indicators of driver distraction refer to determine the different secondary task. Manual distraction like using navigation system, drinking, eating, smoking, or visual distraction like head motions, talking to passengers, hear & makeup or cognitive distraction like day dream- ing. Nevertheless, cognitive distraction remains a difficult task to detect because the driver’s behavior shows that they are attentive but their reflection is not a few proposals attempt to tackle this area. Public databases exist and are used in many researches but sometimes the size is not large enough to apply deep learning models. In This case machine learning as SVM and pre-trained models could be better to use for classification. Another problem is the quality of the dataset, which includes balanced data, appropriate labeling and features, and a minimum of noise on the data. The public datasets make possible comparison between researches which will speed up study in this field. As a result, large-scale, high-quality databases of many measurements are required. To recognize driver inat- tention, several algorithms are used to extract features from images, physiological signals, and driving performance data, which is an essential stage in artificial intelli- gence algorithms. CNNs and pre-trained models are commonly used to extract useful features from images thanks to their great capacity of recognition proven in different domains. For the physiological methods, the wavelet packet transform is a frequently used approach for extracting characteristics from EEG and ECG data. Machine learn- ing algorithms, deep learning algorithms, and especially pre-trained models are com- monly used for detecting driver fatigue or distraction, but it is important to choose the feature extraction algorithm carefully in order to extract the meaningful features for classification. Since the performance of each algorithm differs depending on the fol- lowed approach, further research must be conducted to determine the best algorithms to use depending on the approach used. The following are the key findings drawn from this review: • Several approaches that are typically far from ideal can be used to identify different types of driver inattention, supporting multimodal fusion. • Eye features (e.g., eye closure) and recovery yawning are the most common indica- tors of driver fatigue. The true state of driver fatigue can also be reliably represented by EEG channels. • The indicators of distraction are mostly based on the identification of different sec- ondary task that divert attention away from the driving process, texting is that most dangerous one because involve the three type of distraction. • Cameras appear to be the most common sensor used to detect driver inattention, along with the electroencephalogram, which monitors driver brain activity, and the electrocardiogram, which provides information about the internal workings of the iJIM ‒ Vol. 16, No. 16, 2022 181 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car human body. Wearable sensors (such as smart watches) provide a novel non-invasive technique to benefit from driver-based physiological indicators. • The step of extracting information from images and sensors signals must be carefully performed. For real-time deployment, the time computation must be considered. • AI algorithms are useful for identifying various secondary tasks and determin- ing whether or not the driver is fatigued. Deep learning algorithms are typically employed in this field and have shown to be effective, especially pre-trained models. Some physical indicators can be a sign of distraction or fatigue only if their duration is adequate and a serious work must be done to estimate them in order to avoid false alarms, in this case, ML techniques based on time series are interesting to exploit to predict driver inattention. On the other hand, the physical approach based on facial signs and head orientation can have many false alarms when the driver has a disease in his eyes (blink several times) or a problem in the position of the head, so hybridization with the physiological approach can decrease its false alarms. So should not confuse disease states and fatigue states and maybe on some subjects these methods are not practicable. 6 Limits of review scope This review does not consider driving under the influence, which occurs when a person controls a vehicle after drinking an amount of alcohol or drugs (including pre- scription medicines) that renders them inattentive. In this study, only two forms of driver inattention are discussed: fatigue and distraction. The impact of emotions on driving quality has not been studied and it would be interesting to study and detect the emotions that have the most risk on driving and that affect the attention of the driver. On the other hand, the number of articles covering the three approaches used in this review is uneven, with the majority of publications focusing on the physical approach rather than the physiological and driving performance approaches, which reflects the strong interest of the scientific community in this approach but this may influence some of the results of this review. 7 Conclusion This paper reports on a systematic literature review that summarizes the existing re-search regarding driver inattention detection systems between 2014 and 2021. From an initial set of 1038 papers retrieved from five main publication sources, we selected 324 articles based on the title relevant to our study. The abstracts of the selected arti- cles were reviewed, which led to the extraction of an additional 100 research articles. These articles are then reviewed in depth, and 52 of them are selected for our primary study. We have reviewed various approaches available to monitor the driver inattention including physiological indicators, physical indicators, driving performance indicators and hybrid approach method. We also discussed the advantages and limitations of each approach. It has been found that because each detection method has its own set of 182 http://www.i-jim.org Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car drawbacks, the accuracy of a method based on a single approach is insufficient. A method based on a variety of parameters can produce a more reliable analysis result., so hybrid measurements in some cases can provide more reliable and robust solutions that will both accurately detect driver inattention and reduce the number of false alarms. Deep learning and machine learning techniques are widely used to extract fatigue symptoms and to identify the distraction while driving, especially CNN and SVM achieves the greatest performance. 8 References [1] “Global status report on road safety 2018: summary.” World Health Organization, Geneva, 2018. [2] J. L. Harbluk, Y. I. Noy, P. L. Trbovich, and M. Eizenman, “An on-road assessment of cognitive distraction: Impacts on drivers’ visual behavior and braking performance,” Accid. Anal. Prev, vol. 39, pp. 372–379, 2007. https://doi.org/10.1016/j.aap.2006.08.013 [3] “WHO|Global Status Report on Road Safety 2013,” WHO, 2015, [Online]. Available: https://www.who.int/violence_injury_prevention/road_safety_status/2013/en/ [4] |“WHO|World Report on Road Traffic injury Prevention,” WHO, 2014. [5] W. H. Organization, “Road traffic deaths index 2009 country rankings.” 2010. [6] National Highway Traffic Safety Administration (NHTSA), US Department of Transporta- tion. 2019. Distracted Driving. [7] J. M. Owens et al., Prevalence of drowsy driving crashes: Estimates from a large-scale naturalistic driving study. Washington, DC: Research Brief, AAA Foundation for Traffic Safety, 2018. [8] S. Jafarpour and V. Rahimi-Movaghar, “Determinants of risky driving behavior: A narrative review,” Med J Islam Repub Iran, vol. 28, no. 142, pp. 1–8, 2014. [9] M. Jannat, D. S. Hurwitz, C. Monsere, and K. H. Funk II, “The role of driver’s situational awareness on right-hook bicycle-motor vehicle crashes,” Saf. Sci, vol. 110, pp. 92–101, 2018. https://doi.org/10.1016/j.ssci.2018.07.025 [10] K. J. Parnell, N. A. Stanton, and K. Plant, “Where are we on driver distraction? Methods, approaches and recommendations,” Theor. Issues Ergon. Sci, vol. 19, pp. 578–605, 2018. https://doi.org/10.1080/1463922X.2017.1414333 [11] T. Louw, R. Madigan, O. Carsten, and N. Merat, “Were they in the loop during automated driving? Links between visual attentionand crash potential,” Inj. Prev, vol. 23, pp. 281–286, 2017. https://doi.org/10.1136/injuryprev-2016-042155 [12] T. A. Dingus et al., “Driver crash risk factors and prevalence evaluation using naturalis- tic driving data,” Proc. Natl. Acad. Sci. USA, vol. 113, pp. 2636–2641, 2016. https://doi. org/10.1073/pnas.1513271113 [13] S. Singh, “Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey” National Highway Traffic Safety Administration, 2018. [14] A. J. Khattak, N. Ahmad, B. Wali, and E. Dumbaugh, “A taxonomy of driving errors and violations: Evidence from the naturalistic driving study,” Accid. Anal. Prev, vol. 151, p. 105873, 2021. https://doi.org/10.1016/j.aap.2020.105873 [15] H. Nguyen, R. Q. Ivers, S. Jan, A. L. Martiniuk, Q. Li, and C. Pham, “The economic burden of road traffic injuries: Evidence from a provincial general hospital in Vietnam,” Inj. Prev, vol. 19, pp. 79–84, 2013. https://doi.org/10.1136/injuryprev-2011-040293 [16] G. A. Kumar, T. R. Dilip, L. Dandona, and R. Dandona, “Burden of out-of-pocket expendi- ture for road traffic injuries in urban India,” BMC Health Serv. Res, vol. 12, p. 285, 2012. https://doi.org/10.1186/1472-6963-12-285 iJIM ‒ Vol. 16, No. 16, 2022 183 https://doi.org/10.1016/j.aap.2006.08.013 https://www.who.int/violence_injury_prevention/road_safety_status/2013/en/ https://doi.org/10.1016/j.ssci.2018.07.025 https://doi.org/10.1080/1463922X.2017.1414333 https://doi.org/10.1136/injuryprev-2016-042155 https://doi.org/10.1073/pnas.1513271113 https://doi.org/10.1073/pnas.1513271113 https://doi.org/10.1016/j.aap.2020.105873 https://doi.org/10.1136/injuryprev-2011-040293 https://doi.org/10.1186/1472-6963-12-285 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car [17] W. H. Organization, “Road traffic injuries N358.” WHO, Geneva, 2013. [Online]. Available: http://www.who.int/mediacentre/factsheets/fs358/en/ [18] M. A. Regan and Lee, “Driver distraction: Theory, effects, and mitigation,” J, 2008. https://doi.org/10.1201/9781420007497 [19] D. Dawson, K. F. Reid, alcohol, S. Siuly, Y. Li, and Y. Zhang, “EEG Signal Analysis and Classification,” Nature, vol. 388, p. 235, 1997. https://doi.org/10.1038/40775 [20] A. M. Williamson, A. M. Feyer, R. P. Mattick, R. Friswell, and S. Finlay-Brown, “ Developing measures of fatigue using an alcohol comparison to validate the effects of fatigue on performance,” Accid. Anal. Prev, vol. 33, pp. 313–326, 2001. https://doi.org/10.1016/ S0001-4575(00)00045-2 [21] F. Dinges, “An overview of sleepiness and accidents,” J. Sleep Res, vol. 4, pp. 4–14, 1995. https://doi.org/10.1111/j.1365-2869.1995.tb00220.x [22] S. G. K. R. Oyini Mbouna and M.-G. Chun, “Visual analysis of eye state and head pose for driver alertness monitoring,” IEEE Transactions on Intelligent Transportation Systems, vol. 14, no. 3, pp. 1462–1469, Sep. 2013. https://doi.org/10.1109/TITS.2013.2262098 [23] B. G. Simons-Morton, F. Guo, S. G. Klauer, J. P. Ehsani, and A. K. Pradhan, “Keep your eyes on the road: Young driver crash risk increases according to duration of distraction,” J. Adolesc. Health, vol. 54, pp. 61–67, 2014. https://doi.org/10.1016/j.jadohealth.2013.11.021 [24] T. A. Dingus, V. L. Neale, S. G. Klauer, A. D. Petersen, and R. J. Carroll, “The development of a naturalistic data collection system to perform critical incident analysis: An inves- tigation of safety and fatigue issues in long-haul trucking,” Accid. Anal. Prev, vol. 38, pp. 1127–1136, 2006. https://doi.org/10.1016/j.aap.2006.05.001 [25] T. Kume et al., “Development of absentminded state detection and resolution methods using vehicle equipments,” Trans. Soc. Automot. Eng. Jpn, vol. 45, pp. 567–572, 2014. [26] Y. Saito, M. Itoh, and T. Inagaki, “Driver assistance system with a dual control scheme: Effectiveness of identifying driver drowsiness and preventing lane departure accidents,” IEEE Trans. Human-Mach. Syst, vol. 46, pp. 660–671, 2016. https://doi.org/10.1109/ THMS.2016.2549032 [27] S. Arefnezhad, S. Samiee, A. Eichberger, and A. Nahvi, “Driver drowsiness detection based on steering wheel data applying adaptive neuro-fuzzy feature selection,” Sensors, vol. 19, p. 943, 2019. https://doi.org/10.3390/s19040943 [28] T. Omi, “Detecting drowsiness with the driver status monitor’s visual sensing,” Denso Tech. Rev, vol. 21, pp. 93–102, 2016. [29] C. J. Naurois, C. Bourdin, A. Stratulat, E. Diaz, and J. L. Vercher, “Detection and prediction of driver drowsiness using artificial neural network models,” Accid. Anal. Prev, vol. 126, pp. 95–104, 2019. https://doi.org/10.1016/j.aap.2017.11.038 [30] E. Abe, K. Fujiwara, T. Hiraoka, T. Yamakawa, and M. Kano, “Development of drowsiness detection method by integrating heart rate variability analysis and multivariate statistical process control,” SICE J. Control Meas. Syst. Integr, vol. 9, pp. 10–17, 2016. https://doi. org/10.9746/jcmsi.9.10 [31] H. Iwamoto, K. Hori, K. Fujiwara, and M. Kano, “Real-driving-implementable drowsy driving detection method using heart rate variability based on long short-term mem- ory and autoencoder,” IFAC-PapersOnLine, vol. 54, pp. 526–531, 2021. https://doi.org/ 10.1016/j.ifacol.2021.10.310 [32] S. Arif, M. Arif, S. Munawar, Y. Ayaz, M. J. Khan, and N. Naseer, “EEG spectral compar- ison between occipital and prefrontal cortices for early detection of driver drowsiness,” in Proceedings of the of 2021 International Conference on Artificial Intelligence and Mecha- tronics Systems (AIMS), Jakarta, Indonesia, Apr. 2021, pp. 1–6. https://doi.org/10.1109/ AIMS52415.2021.9466007 184 http://www.i-jim.org http://www.who.int/mediacentre/factsheets/fs358/en/ https://doi.org/10.1201/9781420007497 https://doi.org/10.1038/40775 https://doi.org/10.1016/S0001-4575(00)00045-2 https://doi.org/10.1016/S0001-4575(00)00045-2 https://doi.org/10.1111/j.1365-2869.1995.tb00220.x https://doi.org/10.1109/TITS.2013.2262098 https://doi.org/10.1016/j.jadohealth.2013.11.021 https://doi.org/10.1016/j.aap.2006.05.001 https://doi.org/10.1109/THMS.2016.2549032 https://doi.org/10.1109/THMS.2016.2549032 https://doi.org/10.3390/s19040943 https://doi.org/10.1016/j.aap.2017.11.038 https://doi.org/10.9746/jcmsi.9.10 https://doi.org/10.9746/jcmsi.9.10 https://doi.org/10.1016/j.ifacol.2021.10.310 https://doi.org/10.1016/j.ifacol.2021.10.310 https://doi.org/10.1109/AIMS52415.2021.9466007 https://doi.org/10.1109/AIMS52415.2021.9466007 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car [33] A. Soultana, F. Benabbou, and N. Sael, “Context-awareness in the smart car: Study and analysis,” in Proceedings of the 4th International Conference on Smart City Applications, Oct. 2019, pp. 1–8. https://doi.org/10.1145/3368756.3369019 [34] B. Kitchenham and S. Charters, “Guidelines for performing systematic literature reviews in software engineering,” Keele University and Durham University Joint Report, Technical Report EBSE 2007-001, 2007. [35] K. Petersen, S. Vakkalanka, and L. Kuzniarz, “Guidelines for conducting systematic map- ping studies in software engineering: An update,” Information and Software Technology, vol. 64, pp. 1–18, 2015. https://doi.org/10.1016/j.infsof.2015.03.007 [36] N. Alioua, A. Amine, and M. Rziza, “Driver’s fatigue detection based on yawning extraction,” International Journal of Vehicular Technology, 2014. https://doi.org/10.1155/2014/678786 [37] A. Rahman, M. Sirshar, and A. Khan, “Real time drowsiness detection using eye blink monitoring,” in 2015 National Software Engineering Conference (NSEC), Dec. 2015, pp. 1–7. https://doi.org/10.1109/NSEC.2015.7396336 [38] J. Vicente, P. Laguna, A. Bartra, and R. Bailón, “Drowsiness detection using heart rate variability,” Med. Biol. Eng. Comput, vol. 54, pp. 927–937, 2016. https://doi.org/10.1007/ s11517-015-1448-7 [39] X. Tang, P. Zhou, and P. Wang, “Real-time image-based driver fatigue detection and mon- itoring system for monitoring driver vigilance,” in 2016 35th Chinese Control Conference (CCC), Jul. 2016, pp. 4188–4193. https://doi.org/10.1109/ChiCC.2016.7554007 [40] F. Zhang, J. Su, L. Geng, and Z. Xiao, “Driver fatigue detection based on eye state recog- nition,” in 2017, February, International Conference on Machine Vision and Information Technology (CMVIT), 2017, pp. 105–110. https://doi.org/10.1109/CMVIT.2017.25 [41] L. Wang, H. Wang, and X. Jiang, “A new method to detect driver fatigue based on EMG and ECG collected by port-able non-contact sensors,” Promet-Traffic & Transportation, vol. 29, no. 5, pp. 479–488, 2017. https://doi.org/10.7307/ptt.v29i5.2244 [42] J. Jakubowski and J. Chmielińska, “Detection of driver fatigue symptoms using transfer learning,” Bulletin of the Polish Academy of Sciences. Technical Sciences, vol. 66, no. 6, 2018. [43] J. Xu, J. Min, and J. Hu, “Real-time eye tracking for the assessment of driver fatigue,” Healthcare Technology Letters, vol. 5, no. 2, pp. 54–58, 2018. https://doi.org/10.1049/ htl.2017.0020 [44] Z. Xiao, Z. Hu, L. Geng, F. Zhang, J. Wu, and Y. Li, “Fatigue driving recognition network: Fatigue driving recognition via convolutional neural network and long short-term memory units,” IET Intelligent Transport Systems, vol. 13, no. 9, pp. 1410–1416, 2019. https://doi. org/10.1049/iet-its.2018.5392 [45] W. M. Shalash, “Driver Fatigue Detection with Single EEG Channel Using Transfer Learn- ing,” in 2019 IEEE International Conference on Imaging Systems and Techniques (IST), Dec. 2019, pp. 1–6. https://doi.org/10.1109/IST48021.2019.9010483 [46] W. Liu, J. Qian, Z. Yao, X. Jiao, and J. Pan, “Convolutional two-stream network using multi-facial feature fusion for driver fatigue detection,” Future Internet, vol. 11, no. 5, p. 115, 2019. https://doi.org/10.3390/fi11050115 [47] L. Mašanović, M. Vranješ, R. Džakula, and Ž. Lukač, “Driver monitoring using the in-vehicle camera,” in 2019 Zooming Innovation in Consumer Technologies Conference (ZINC), May 2019, pp. 33–38. https://doi.org/10.1109/ZINC.2019.8769377 [48] M. Chai, “Drowsiness monitoring based on steering wheel status,” Transportation Research Part D: Transport and Environment, vol. 66, pp. 95–103, 2019. https://doi.org/10.1016/j. trd.2018.07.007 iJIM ‒ Vol. 16, No. 16, 2022 185 https://doi.org/10.1145/3368756.3369019 https://doi.org/10.1016/j.infsof.2015.03.007 https://doi.org/10.1155/2014/678786 https://doi.org/10.1109/NSEC.2015.7396336 https://doi.org/10.1007/s11517-015-1448-7 https://doi.org/10.1007/s11517-015-1448-7 https://doi.org/10.1109/ChiCC.2016.7554007 https://doi.org/10.1109/CMVIT.2017.25 https://doi.org/10.7307/ptt.v29i5.2244 https://doi.org/10.1049/htl.2017.0020 https://doi.org/10.1049/htl.2017.0020 https://doi.org/10.1049/iet-its.2018.5392 https://doi.org/10.1049/iet-its.2018.5392 https://doi.org/10.1109/IST48021.2019.9010483 https://doi.org/10.3390/fi11050115 https://doi.org/10.1109/ZINC.2019.8769377 https://doi.org/10.1016/j.trd.2018.07.007 https://doi.org/10.1016/j.trd.2018.07.007 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car [49] Z. Liu, Y. Peng, and W. Hu, “Driver fatigue detection based on deeply-learned facial expres- sion representation,” Journal of Visual Communication and Image Representation, vol. 71, p. 102723, 2020. https://doi.org/10.1016/j.jvcir.2019.102723 [50] N. Alioua, A. Amine, A. Rogozan, A. Bensrhair, and M. Rziza, “Driver head pose estima- tion using efficient descriptor fusion,” EURASIP Journal on Image and Video Processing, vol. 2016, no. 1, p. 2, 2016. https://doi.org/10.1186/s13640-016-0103-z [51] W. Kim, H. K. Choi, B. T. Jang, and J. Lim, “Driver distraction detection using single convo- lutional neural network,” in 2017 International Conference on Information and Communi- cation Technology Convergence (ICTC), Oct. 2017, pp. 1203–1205. https://doi.org/10.1109/ ICTC.2017.8190898 [52] S. V. Deshmukh and O. Dehzangi, “ECG-based driver distraction identification using wave- let packet transform and discriminative kernel-based features,” in 2017 IEEE International Conference on Smart Computing (SMART-COMP), May 2017, pp. 1–7. https://doi.org/ 10.1109/SMARTCOMP.2017.7947003 [53] B. Baheti, S. Gajre, and S. Talbar, “Detection of distracted driver using convolutional neural network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition Workshops, 2018, pp. 1032–1038. https://doi.org/10.1109/CVPRW.2018.00150 [54] Y. Xing et al., “Identification and analysis of driver postures for in-vehicle driving activities and secondary tasks recognition,” IEEE Transactions on Computational Social Systems, vol. 5, no. 1, pp. 95–108, 2017. https://doi.org/10.1109/TCSS.2017.2766884 [55] O. Dehzangi and M. Taherisadr, “Driver distraction detection using MEL cepstrum rep- resentation of galvanic skin responses and convolutional neural networks,” in 2018 24th International Conference on Pattern Recognition (ICPR), Aug. 2018, pp. 1481–1486. https:// doi.org/10.1109/ICPR.2018.8545082 [56] H. M. Eraqi, Y. Abouelnaga, M. H. Saad, and M. N. Moustafa, “Driver distraction identifi- cation with an ensemble of convolutional neural networks,” Journal of Advanced Transpor- tation, 2019. https://doi.org/10.1155/2019/4125865 [57] J. M. Celaya-Padilla et al., “Texting & Driving” detection using deep convolutional neu- ral networks,” Applied Sciences, vol. 9, no. 15, p. 2962, 2019. https://doi.org/10.3390/ app9152962 [58] Y. Xing, C. Lv, H. Wang, D. Cao, E. Velenis, and F. Y. Wang, “Driver activity recognition for intelligent vehicles: A deep learning approach,” IEEE Transactions on Vehicular Tech- nology, vol. 68, no. 6, pp. 5379–5390, 2019. https://doi.org/10.1109/TVT.2019.2908425 [59] Y. Hu, M. Lu, and X. Lu, “Driving behaviour recognition from still images by using multi- stream fusion CNN,” Machine Vision and Applications, vol. 30, no. 5, pp. 851–865, 2019. https://doi.org/10.1007/s00138-018-0994-z [60] Q. Xiong, J. Lin, W. Yue, S. Liu, Y. Liu, and C. Ding, “A deep learning approach to driver distraction detection of using mobile phone,” in 2019 IEEE Vehicle Power and Propulsion Conference (VPPC), Oct. 2019, pp. 1–5. https://doi.org/10.1109/VPPC46532.2019.8952474 [61] L. Li, B. Zhong, C. Hutmacher, Y. Liang, W. J. Horrey, and X. Xu “Detection of driver man- ual distraction via image-based hand and ear recognition,” Accident Analysis & Prevention, vol. 137, p. 105432, 2020. https://doi.org/10.1016/j.aap.2020.105432 [62] C. Jin, Z. Zhu, Y. Bai, G. Jiang, and A. He, “A deep-learning-based scheme for detect- ing driver cell-phone use,” IEEE Access, vol. 8, pp. 18580–18589, 2020. https://doi.org/ 10.1109/ACCESS.2020.2968464 [63] M. S. Majdi, S. Ram, J. T. Gill, and J. J. Rodríguez, “Drive-net: Convolutional network for driver distraction detection,” in 2018 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI), Apr. 2018, pp. 1–4. https://doi.org/10.1109/SSIAI.2018.8470309 [64] L. Chen et al., “Driver fatigue detection via differential evolution extreme learning machine technique,” Electronics, vol. 9, no. 11, 2020. https://doi.org/10.3390/electronics9111850 186 http://www.i-jim.org https://doi.org/10.1016/j.jvcir.2019.102723 https://doi.org/10.1186/s13640-016-0103-z https://doi.org/10.1109/ICTC.2017.8190898 https://doi.org/10.1109/ICTC.2017.8190898 https://doi.org/10.1109/SMARTCOMP.2017.7947003 https://doi.org/10.1109/SMARTCOMP.2017.7947003 https://doi.org/10.1109/CVPRW.2018.00150 https://doi.org/10.1109/TCSS.2017.2766884 https://doi.org/10.1109/ICPR.2018.8545082 https://doi.org/10.1109/ICPR.2018.8545082 https://doi.org/10.1155/2019/4125865 https://doi.org/10.3390/app9152962 https://doi.org/10.3390/app9152962 https://doi.org/10.1109/TVT.2019.2908425 https://doi.org/10.1007/s00138-018-0994-z https://doi.org/10.1109/VPPC46532.2019.8952474 https://doi.org/10.1016/j.aap.2020.105432 https://doi.org/10.1109/ACCESS.2020.2968464 https://doi.org/10.1109/ACCESS.2020.2968464 https://doi.org/10.1109/SSIAI.2018.8470309 https://doi.org/10.3390/electronics9111850 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car [65] B. K. Savaş and Y. Becerikli, “Real time driver fatigue detection system based on multi- task ConNN,” IEEE Access, vol. 8, pp. 12491–12498, 2020. https://doi.org/10.1109/ ACCESS.2020.2963960 [66] S. Masood, A. Rai, M. N. D. Aakash Aggarwal, and M. Ahmad, “Detecting distraction of drivers using Convolutional Neural Network,” Pattern Recognition Letters, vol. 139, pp. 79–85, 0167–8655, 2020. https://doi.org/10.1016/j.patrec.2017.12.023 [67] D. Shi and H. Tang, “Research on safe driving evaluation method based on machine vision and long short-term memory network,” Journal of Electrical and Computer Engineering, vol. 2021, pp. 1–13, Apr. 2021. https://doi.org/10.1155/2021/9955079 [68] H. A. Rahim, A. Dalimi, and H. Jaafar, “Detecting drowsy driver using pulse sensor,” J. Teknol, vol. 73, p. 3, 2015. https://doi.org/10.11113/jt.v73.4238 [69] G. D. Furman, A. Baharav, C. Cahan, and S. Akselrod, “Early detection of falling asleep at the wheel: A heart rate variability approach,” in Proceedings of the 2008 Computers in Car- diology, Singapore, Sep. 2008, pp. 1109–1112. https://doi.org/10.1109/CIC.2008.4749240 [70] L. R. Hartley, P. K. Arnold, G. Smythe, and J. Hansen, “Indicators of fatigue in truck drivers,” Appl. Ergon, vol. 25, pp. 143–156, 1994. https://doi.org/10.1016/0003-6870(94)90012-4 [71] G. F. Wilson and R. D. O’Donnell, “Measurement of operator workload with the neu ro- psychological workload test battery,” in Advances in Psychology, vol. 52, P. A. Hancock and N. Meshkati, Eds. North-Holland: Haarlem, 1988, pp. 63–100. https://doi.org/10.1016/ S0166-4115(08)62383-3 [72] G. Mulder and W. R. E. H. Meulen, “Mental load and the measurement of heart rate vari- ability,” Ergonomics, vol. 16, pp. 69–83, 1973. https://doi.org/10.1080/00140137308924483 [73] K. Fujiwara, et al. Heart rate variability-based driver drowsiness detection and its validation with EEG. IEEE Transactions on Biomedical Engineering, vol. 66, no 6, p. 1769–1778, 2018. https://doi.org/10.1109/TBME.2018.2879346 [74] Q. Zhuang, Z. Kehua, J. Wang, and Q. Chen, “Driver fatigue detection method based on eye states with pupil and iris segmentation,” IEEE Access, vol. 8, pp. 173440–173449, 2020. https://doi.org/10.1109/ACCESS.2020.3025818 [75] Y. Ed-Doughmi, N. Idrissi, and Y. Hbali, “Real-time system for driver fatigue detec- tion based on a recurrent neuronal network,” J. Imaging, vol. 6, no. 3, p. 8, Mar. 2020. https://doi.org/10.3390/jimaging6030008 [76] L. Li, B. Zhong, C. Hutmacher, Y. Liang, and W. J. Horrey, “Xu Xu, Detection of driver manual distraction via image-based hand and ear recognition,” Accident Analysis & Preven- tion, vol. 137,2020, 105432, pp. 0001–4575. https://doi.org/10.1016/j.aap.2020.105432 [77] B. Fatima, A. R. Shahid, S. Ziauddin, A. A. Safi, and H. Ramzan, “Driver fatigue detection using viola jones and principal component analysis,” Applied Artificial Intelligence, vol. 34, no. 6, pp. 456–483, 2020. https://doi.org/10.1080/08839514.2020.1723875 [78] C. Huang, X. Wang, J. Cao, S. Wang, and Y. Zhang, “HCF: A hybrid CNN framework for behavior detection of distracted drivers,” IEEE Access, vol. 8, pp. 109335–109349, 2020. https://doi.org/10.1109/ACCESS.2020.3001159 [79] M. Oberoi, H. Panchal, and Y. Jian, “Driver distraction detection using transfer learning,” International Journal of Engineering Research & Technology (IJERT), vol. 09, no. 05, 2020. [80] P. Mao, K. Zhang, and D. Liang, “Driver distraction behavior detection method based on deep learning,” MS&E, vol. 782, no. 2, p. 022012, 2020. https://doi.org/ 10.1088/1757-899X/782/2/022012 [81] F. Faraji, F. Lotfi, J. Khorramdel, A. Najafi, and A. Ghaffari, “Drowsiness detection based on driver temporal behavior using a new developed dataset,” arXiv:2104.00125 iJIM ‒ Vol. 16, No. 16, 2022 187 https://doi.org/10.1109/ACCESS.2020.2963960 https://doi.org/10.1109/ACCESS.2020.2963960 https://doi.org/10.1016/j.patrec.2017.12.023 https://doi.org/10.1155/2021/9955079 https://doi.org/10.11113/jt.v73.4238 https://doi.org/10.1109/CIC.2008.4749240 https://doi.org/10.1016/0003-6870(94)90012-4 https://doi.org/10.1016/S0166-4115(08)62383-3 https://doi.org/10.1016/S0166-4115(08)62383-3 https://doi.org/10.1080/00140137308924483 https://doi.org/10.1109/TBME.2018.2879346 https://doi.org/10.1109/ACCESS.2020.3025818 https://doi.org/10.3390/jimaging6030008 https://doi.org/10.1016/j.aap.2020.105432 https://doi.org/10.1080/08839514.2020.1723875 https://doi.org/10.1109/ACCESS.2020.3001159 https://doi.org/10.1088/1757-899X/782/2/022012 https://doi.org/10.1088/1757-899X/782/2/022012 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car [82] V. P. B and S. Chinara, “Automatic classification methods for detecting drowsiness using wavelet packet transform extracted time-domain features from single-channel EEG signal,” Journal of Neuroscience Methods, vol. 347, p. 108927, Jan. 2021. https://doi.org/10.1016/j. jneumeth.2020.108927 [83] S. Mehta, S. Dadhich, S. Gumber, and A. Jadhav Bhatt, “Real-time driver drowsiness detec- tion system using eye aspect ratio and eye closure ratio,” 2019, February, https://doi.org/ 10.2139/ssrn.3356401 [84] N. S. Karuppusamy and B. Y. Kang, “Multimodal system to detect driver fatigue using EEG, gyroscope, and image processing,” IEEE Access, vol. 8, pp. 129645–129667, 2020. https:// doi.org/10.1109/ACCESS.2020.3009226 [85] J. Ma, Y. L. Murphey, and H. Zhao, “Real time drowsiness detection based on lateral dis- tance using wavelet transform and neural network,” in 2015 IEEE Symposium Series on Computational Intelligence, 2015, pp. 411–418. https://doi.org/10.1109/SSCI.2015.68 [86] J. Gwak, A. Hirao, and M. Shino, “An investigation of early detection of driver drows- iness using ensemble machine learning based on hybrid sensing,” Applied Sciences, vol. 10, no. 8, p. 2890, Apr. 2020. https://doi.org/10.3390/app10082890 [87] M. Atiquzzaman, Y. Qi, and R. Fries, “Real-time detection of drivers’ texting and eating behavior based on vehicle dynamics,” Transportation Research Part F: Traffic Psychology and Behaviour, vol. 58, pp. 594–604, 2018. https://doi.org/10.1016/j.trf.2018.06.027 [88] Z. Li, L. Chen, J. Peng, and Y. Wu, “Automatic detection of driver fatigue using driv ing operation information for transportation safety,” Sensors (Basel, Switzerland), 2017. https:// doi.org/10.3390/s17061212 [89] S. P. Rajamohana, E. G. Radhika, S. Priya, and S. Sangeetha, “Driver drowsiness detection system using hybrid approach of convolutional neural network and bidirectional long short term memory (CNN_BILSTM),” Materials Today: Proceedings, vol. 45, pp. 2897–2901, 2021. https://doi.org/10.1016/j.matpr.2020.11.898 [90] Y. Ma, G. Gu, B. Yin, S. Qi, K. Chen, and C. Chan, “Support vector machines for the iden- tification of real-time driving distraction using in-vehicle information systems,” Journal of Transportation Safety & Security, pp. 1–24, 2020. https://doi.org/10.1080/19439962.2020.1 774019 [91] Z. Li, S. Bao, I. V. Kolmanovsky, and X. Yin, “Visual-manual distraction detection using driving performance indicators with naturalistic driving data,” IEEE Transactions on Intelligent Transportation Systems, vol. 19, no. 8, pp. 2528–2535, Aug. 2018. https://doi.org/ 10.1109/TITS.2017.2754467 [92] M. Taherisadr and O. Dehzangi, “EEG-based driver distraction detection via game- theoretic-based channel selection,” in Advances in Body Area Networks I, Cham: Springer, 2019, pp. 93–105. https://doi.org/10.1007/978-3-030-02819-0_8 [93] K. T. Chui, W. Alhalabi, and R. W. Liu, Head motion coefficient-based algorithm for dis- tracted driving detection. Data Technologies and Applications, 2019. https://doi.org/10.1108/ DTA-09-2018-0086 [94] O. Jemai, I. Teyeb, T. Bouchrika, and C. Ben amar, “A novel approach for drowsy driver detection using eyes recognition system based on wavelet network,” Int. J. Recent Contrib. Eng. Sci. IT, vol. 1, no 1, p. 46, juill. 2013, https://doi.org/10.3991/ijes.v1i1.2929 [95] B. Akrout and W. Mahdi, “Hypovigilance detection based on eyelids behavior study,” Int. J. Recent Contrib. Eng. Sci. IT, vol. 1, no 1, p. 39, juill. 2013, https://doi.org/10.3991/ijes. v1i1.2927 [96] C. Huda, H. Tolle, and F. Utaminingrum, “Mobile-based driver sleepiness detection using facial landmarks and analysis of EAR values,” Int. J. Interact. Mob. Technol., vol. 14, no. 14, p. 16, août 2020, https://doi.org/10.3991/ijim.v14i14.14105 188 http://www.i-jim.org https://doi.org/10.1016/j.jneumeth.2020.108927 https://doi.org/10.1016/j.jneumeth.2020.108927 https://doi.org/10.2139/ssrn.3356401 https://doi.org/10.2139/ssrn.3356401 https://doi.org/10.1109/ACCESS.2020.3009226 https://doi.org/10.1109/ACCESS.2020.3009226 https://doi.org/10.1109/SSCI.2015.68 https://doi.org/10.3390/app10082890 https://doi.org/10.1016/j.trf.2018.06.027 https://doi.org/10.3390/s17061212 https://doi.org/10.3390/s17061212 https://doi.org/10.1016/j.matpr.2020.11.898 https://doi.org/10.1080/19439962.2020.1774019 https://doi.org/10.1080/19439962.2020.1774019 https://doi.org/10.1109/TITS.2017.2754467 https://doi.org/10.1109/TITS.2017.2754467 https://doi.org/10.1007/978-3-030-02819-0_8 https://doi.org/10.1108/DTA-09-2018-0086 https://doi.org/10.1108/DTA-09-2018-0086 https://doi.org/10.3991/ijes.v1i1.2929 https://doi.org/10.3991/ijes.v1i1.2927 https://doi.org/10.3991/ijes.v1i1.2927 https://doi.org/10.3991/ijim.v14i14.14105 Paper—A Systematic Literature Review of Driver Inattention Monitoring Systems for Smart Car 9 Authors Abdelfettah Soultana is a graduate of the Master’s degree in Software Quality from Hassan II University, Casablanca, Morocco, in 2015. The working title of his thesis is “Towards a contextual system for smart car management”. He is currently working on his PhD at the Laboratory of Information Processing and Modeling (LTIM) at the Ben M’sik Faculty of Science. His research focuses on machine learning, deep learning and Internet of things for driver context monitoring. (email: soultana.abdelfettah@gmail. com). Faouzia Benabbou is a professor of Computer Science and member of Compute Science and Information Processing laboratory. She is Head of the team “Cloud Com- puting, Network and Systems Engineering (CCNSE)”. She received his Ph.D. in Com- puter Science from the Faculty of Sciences, University Mohamed V, Morocco, 1997. His research areas include cloud Computing, data mining, machine learning, and Natu- ral Language Processing. (email: faouzia.benabbou@univh2c.ma). Nawal Sael is a professor of Computer Science and member of Computer Science and Information Processing laboratory at faculty of science Ben M’sik (Casablanca, Morocco). She received her Ph.D. in Computer Science from the Faculty of Sciences, University Hassan II Casablanca, Morocco, 2013 and her engineer degree in software engineering from ENSIAS, Morocco, in 2002. Here research interests include data min- ing, educational data mining, machine learning, deep learning and Internet of things. (email: saelnawal@hotmail.com). Sara Ouahabi is a assistant professor of Computer Science and member of Com- puter Science and Information Processing laboratory at faculty of science Ben M’sik (Casablanca, Morocco). She received her Ph.D. in Computer Science from the Fac- ulty of Sciences, University Hassan II Casablanca, Morocco. Here research interests include Computer Communications (Networks), Educational Technology, Information Science. (email: sara.ouahabi@gmail.com). Article submitted 2022-06-07. Resubmitted 2022-07-08. Final acceptance 2022-07-08. Final version published as submitted by the authors. iJIM ‒ Vol. 16, No. 16, 2022 189 mailto:soultana.abdelfettah@gmail.com mailto:soultana.abdelfettah@gmail.com mailto:faouzia.benabbou@univh2c.ma mailto:saelnawal@hotmail.com mailto:sara.ouahabi@gmail.com