Acta Polytechnica CTU Proceedings doi:10.14311/APP.2017.12.0032 Acta Polytechnica CTU Proceedings 12:32–37, 2017 © Czech Technical University in Prague, 2017 available online at http://ojs.cvut.cz/ojs/index.php/app MICROSLEEPS AND THEIR DETECTION FROM BIOLOGICAL SIGNALS Martin Holub∗, Martina Šrutová, Lenka Lhotská Czech Technical University in Prague, Faculty of Electrical Engineering, Department of Cybernetics, Karlovo nám. 13, Praha 2, 121 35, Czech Republic ∗ corresponding author: MartinHolub@Mail.com Abstract. Microsleeps (MS) are a frequently discussed topic due to their fatal consequences. Their detection is necessary for the purpose of sleep laboratories, where they provide an option for the quantifying rate of sleep deprivation level and objective evaluation of subjective sleepiness. Many studies are dealing with this topic for automotive usage to design a fatigue countermeasure device. We made a research of recent attitude to the development of the automated MS detection methods. We created an overview of several MS detection approaches based on the measurement of biological signals. We also summarized the changes in EEG, EOG and ECG signals, which have been published over the last few years. The reproducible changes in the entire EEG spectrum, primarily with the increased activity of delta and theta, were noticed during a transition to fatigue. There were observed changes of blinking rate and reduction of eye movements during the fatigue tasks. MS correspond with variations in the autonomic regulation of the cardiovascular function, which can be quantified by HRV parameters. The decrease in HR, VLF, and LF/HF before falling asleep was revealed. EEG signal, especially its slow wave activity, considered to be the most predictive and reliable for the level of alertness. In spite of the detection from EEG signal is the most common method, EOG based approaches can also be very efficient and more driver-friendly. Besides, the signal processing in the time domain can improve the detection accuracy of the short events like MS. Keywords: Microsleep, automatic detection, detection methods, EOG, EEG, EKG. 1. Introduction Microsleeps (MS) are mostly detected for the purpose of automotive usage and sleep laboratories. The eval- uation of MS stages in the sleep laboratories provides an option for the quantifying rate of sleep depriva- tion level and an objective assessment of subjective sleepiness. They use the definition of microsleep (MS), which arose as a consequence of the evaluation method of the PSG (polysomnography) or MSLT (multiple sleep latency test) data, where each 30-second epoch is evaluated separately. To score this epoch as a sleep, more than 15 seconds should show some sleep activ- ity [1]. Episodes with sleep symptoms, but with a short duration between 3 and 15 seconds are defined as the microsleeps. The epoch would be classified as a sleep in case, they were longer than 15 seconds [2]. The theta rhythm with slow eye movements, K com- plexes, or sleep spindles is included. Microsleeps can be accompanied by the same behavioural changes as during the states of drowsiness. In spite of the short duration of microsleep, it can be inconvenient and dangerous for many people and for their surroundings to perform monotonous activity while being tired and drowsy. Many studies are dealing with this topic for automotive usage to design a fatigue countermeasure device. For those kinds of studies, it isn’t important to distinguish the events of drowsiness, microsleeps or fatigue correctly by the terminology, because all the states associated with the loss of attention are dangerous in the traffic. In the studies observed, the mentioned lapses of responsiveness are often mani- fested by absent or prolonged responses and short suspension of performance. These symptoms are ac- companied by droopy eyes, slow eyelid closure, head nodding and many changes in biological signals. We aimed to make a research of recent attitudes for the development of the automated MS detection meth- ods. We created an overview of several MS detection approaches based on biological signals measurement. They are mostly based on the processing EEG (elec- troencephalography) signals or EEG accompanied by EOG (electrooculography) signals. We also summa- rized the changes in the basic biological signals like EEG, EOG and ECG (electrocardiography), which have been published over the last few years. 2. Signal changes 2.1. Changes in EEG signals The changes in the entire EEG frequency spectrum were noticed during transition to fatigue [3–8]. The increased activity of delta and theta rhythms was pri- marily observed. It could reflect decreased cortical arousal during a monotonous task. The removing noise from the EEG is crucial for observation of right changes. The filtration method is especially necessary 32 http://dx.doi.org/10.14311/APP.2017.12.0032 http://ojs.cvut.cz/ojs/index.php/app vol. 12/2017 Microsleeps and Their Detection from Biological Signals for the low frequency signal, which is influenced by artefacts from activities such as breathing and move- ment. Slow activity is prone to be also contaminated by eye movement artefact. Various areas of the brain behaved differently during fatigue, during its onset delta and theta activity were present mostly in the frontal, central, and parietal areas of the brain with some anterior alpha and posterior beta [3, 5]. The occipitoparietal alpha spread to anterior areas and become more centrofrontal and temporal alpha dur- ing drowsiness [3]. The reproducibility of the EEG magnitude changes in the delta, theta, alpha and beta bands during drive fatigue was tested [9]. There was revealed high reproducibility for the delta and theta bands [9]. During the transitional phase to fatigue, there were no significant differences in the delta and alpha magnitudes across the entire brain [9]. The inter-hemispheric EEG coherence analysis was shown in the study of Jap et al. [10], which revealed that the interhemispheric coherence level was significantly higher at the frontal and occipital sites compared to the central, parietal, and temporal sites throughout the driving sessions. 2.2. Changes in EOG signals Due to the rich connectivity between eyes and brain, different EOG analyses can be used to identify the states similar to fatigue. The changes of the blinking rate were observed during the fatigue tasks in the published papers [5]. The work of Lal [5] showed, that the fast eye movements and conventional blinks during alert were replaced by no eye movements and small fast rhythmic blinks during fatigue, and that the slow blinks were present during deeper drowsiness for all the subjects. Wierwille et al. [11] declared that slow eyelid closure is a reliable estimate of the level of drowsiness. In the study of Papadelis et al. [12], a significant increase of eye blinks’ number and duration before driving errors was observed. 2.3. Changes in ECG signals The stages of microsleeps, drowsiness or fatigue cor- respond with the changes in the autonomic regula- tion of the cardiovascular function. The HRV (Heart Rate Variability) parameters are used to quantify these changes. The gradual and sustained decrease in the VLF range minutes before falling asleep was observed [13]. The ratio LF/HF, which corresponds with the sympathovagal balance, was very low com- pared to baseline wake values for about 5 minutes before the event [13]. It is a sign of the relaxation tak- ing over. There is a significant surge in this balance after the falling asleep due to the increased stress [13]. Heart rate decreased significantly during falling asleep event [13] and fatigue (by 7 bpm) [5]. Same changes in the heart rate were observed during prolonged and monotonous driving [14]. Also, the RR variability declined over the falling asleep events in the study of Furman [13]. 3. Detection approaches In the following section, there are presented and com- pared (Table 1) detection methods, which have been issued during the recent years and which were dealing with the states similar to microsleeps. 3.1. Detection by Vuckovic et al. (published 2002) The classification of the drowsy states from full spectrum EEG recordings was published in the pa- per [15]. Parameterization was based on a moving cross-spectral density csd. The interhemispheric csd was calculated from homologous contralateral elec- trodes [15]. The intrahemispheric csd was calculated from electrodes on the left hemisphere [15]. The csd was computed for the signal lasting one epoch and the signal from the same electrode delayed by 50% of the epoch [15]. The main reason to choose interhemi- spheric csd was that the correlation of homologue electrodes decreases in drowsiness [16]. As well as the Intrahemispheric csd detects the transition from alertness to drowsiness and vice versa because of the changes in the synchronization of the intrahemispheric EEG signal [16]. A convolutional neural network, auto encoder, and linear dynamic system described accu- rately in [15] were used for classification. 3.2. Detection by Golz et al. (published 2007) Different MS detection approaches and their combina- tions were presented in the paper [17]. The adaptive signal processing was based on the signals of the brain electric activity, variation in the pupil size, and eyelid movements. Totally fifteen signals had been acquired, seven signals of EEG, vertical and horizontal EOG channel, and six signals from eye tracking system (the pupil size and two coordinates of eye gaze on the plane of projection). The features were firstly ex- tracted in the frequency domain by one linear method of modified periodogram and then in the state space by using the nonlinear method of the delay vector variance (DVV) [18]. The results show, that peri- odogram was more powerful for the feature extraction than the DVV method and that the complementing power spectral density by DVV features showed only small improvements. The achieved features were used as input vectors in automatically learning classifica- tion algorithm, where the decision process was based on the support vector machines and three learning vector quantization neural networks. All algorithms were optimized to gain maximal classification accu- racy. The best results were achieved by the fusion of features for all available signals. Proposed methods were created with respect to the future online driver monitoring. [17] 33 M. Holub, M. Šrutová, L. Lhotská Acta Polytechnica CTU Proceedings Figure 1. The table shows the comparison of the published methods. 3.3. Detection by Furman et al. (published 2008) This study [13] checked a feasibility of the new detec- tion approach based on the processing of the ECG signal, where the trends of heart rate variability (HRV) were observed. The HRV parameters were used to look into the changes in the autonomic regulation of the cardiovascular function. Physiological changes were quantified before, during and after an event of microsleep. It was found that there was a gradual and sustained decrease in the VLF range minutes be- fore falling asleep events. The sympathovagal balance, which was represented by LF/HF, was very low com- pared to baseline wake values for about 5 minutes before the events. It was a sign of relaxation taking over. After falling asleep there was a surge in this bal- ance signifying an increased stress aimed to overcome the drowsiness. The mean HR and RR variability decreased during falling asleep events, thus confirm the autonomic changes. During a microsleep, there was a decrease in sympathetic drive with increased vagal activity. [13] 3.4. Detection by Tsai et al. (published 2009) Real-time system [19] for detecting a drowsy driver was based on EEG signal processing. Six channels of EEG were acquired by a novel active dry electrode non-invasive system. The wavelet transform using Daubechies 2 wavelet was applied on the each channel. Integrated EEG and zero crossing features were com- puted from the scales corresponding to the frequency of beta, alpha and theta rhythms. All 36 parameters were used as an input vector to the artificial neural network with 18 hidden nodes and two output nodes. System rejected the detection accuracy of drowsy sub- ject 90.9% with too many warnings signals while the subject was totally alert. [19] 3.5. Detection by Garcés et al. (published 2010) The automatic detector of drowsiness episodes [20] was based on the processing of one EEG channel. The Butterworth band-pass filter with cut-off frequencies of 0.5 and 60 Hz was used for signal pre-processing 34 vol. 12/2017 Microsleeps and Their Detection from Biological Signals and then segmented into the blocks of duration of 1, 2, 5 and 10 seconds. The length of 5 seconds was considered the most appropriate to take the decision in the real time. Twelve features were computed from spectral analysis and wavelet decomposition of the EEG signal: the central frequency, the first quartile frequency, the maximum frequency, the total energy of spectrum, the power of theta and alpha bands, the number of zero crossing and the integrated EEG for scales 3, 4 and 5 of Daubechies 2 wavelet trans- form. In this case, the equivalent filters of mentioned scales had approximately same frequency as beta, al- pha and theta rhythms. Twenty-five neural networks with three layers (first - 12 neurons, last - one neu- ron with tan-sigmoid function, hidden-between 5 and 30 neurons) were tested for classification. The best architecture was neural network (12-20-1), for which 81.7% of drowsiness was detected. [21] 3.6. Detection by Peiris et al. (published 2011) The detection of lapses in responsiveness [22] was based on the signal processing of the electroencephalo- gram. The features using spectral analysis (mean spectral power, normalized spectral power and spec- tral power ratios) and three nonlinear methods (frac- tal dimension, approximate entropy and Lempel-Ziv complexity) were calculated and compare to deter- mine their efficacy in EEG-based detection of lapses. Principal component analysis was used to reduce the dimensionality of the feature matrix. The detector model created using mean spectral power features achieved the best failure state estimation performance. There were reached sensitivity 73.5% and selectivity 25.5%. Among the nonlinear approaches the detec- tor based on the Lempel-Ziv complexity showed the highest performance. The performance of the detec- tor was no greater by complementing spectral power features by Lempel-Ziv complexity. It was similar to the finding of [17]. The use of meta-learner weights showed a marginally higher performance than the use of constrained least-squares metalearner weights. [22] 3.7. Detection by Zhu et al. (published 2014) The drowsiness detection [23] was based on two elec- trooculography signals, vertical and horizontal. These signals were filtered, saturated and normalized dur- ing pre-processing. The automatic detection technic of slow eye movements [21, 24, 25] based on wavelet transform was used for the feature detection. The features were slow eye movements number, duration of closing phase of blink, peak and mean velocities in closing and opening phases of blink. The wavelet transformation was also used for the extraction of two other features - ratio of low and high frequency on both horizontal and vertical EOGs. The linear dynamic system was used for post processing. The convolution neural network, which was described in detail in the paper [23], was used for classification. 3.8. Detection by Holub et al. (published 2015) MS classifier [26] was based on the independent analy- sis of each couple of EOG signals in the time domain. The basic idea of the algorithm was to observe differ- ences between the minimum and maximum values in the appropriately chosen sliding window, which was gradually applied to the entire length of both EOG signals. By merging information from the signals of the left and the right eye was calculated the final feature, which was able to appropriately distinguish MS events from the rest of the awake signal [26]. This parameter highlighted MS states, which were mani- fested in the signal as the episodes without significant eye movements and fast changes. These episodes were terminated by the specific high-frequency wave, which corresponded to blink and arousal from the MS. The final parameter was thresholded to distinguish MS from the rest of the signal. The differences between fix and adaptable thresholding were presented. The fix threshold was appropriately set as the same value for all signals, while the calculation of adaptable thresh- old was based on the signal similarity of MS episodes with sleep stages NREM1 and NREM2, which were used to set this limit value. There were reached preci- sion 67% and recall 82% in the method of adaptable thresholding. 4. Conclusions EEG signal has been considered by the researchers as the most predictive and reliable for the level of alertness. Previous studies confirmed that detection of changes in slow wave activity might form the ba- sis of a fatigue countermeasure device. These ap- proaches use mostly wavelet transform or methods of periodogram, which can be suitable for the detection of drowsy states, but they can provide the insuffi- cient time resolution for detection of very short events like MS. To achieve a good frequency resolution, it is necessary to analyse the longest possible part of the signal. But the time resolution deteriorates with increasing length of the sliding window, which is ap- plied to the analysed signal. This fact is undesirable due to the MS detection of an event that may take only 3 seconds. The window has to be always chosen long enough with respect to the stationarity of the signal and frequency resolution. The same problem of insufficient time resolution can be in the detection from ECG, where the heart rate variability is counted. In comparison to detection in the frequency domain, the signal processing in the time domain can provide higher accuracy and faster response of the detector. In spite of the detection from EEG signal, it is the most common method, EOG based approaches can also be very efficient and more driver-friendly. The electrodes for the measuring of EOG signals can be 35 M. Holub, M. Šrutová, L. Lhotská Acta Polytechnica CTU Proceedings easily implemented for example to headbands, hats or glasses, which provide unlike measuring multi-channel EEG greater comfort and easy application for the drivers. Acknowledgements This work has been supported by the project grant applica- tion No. SGS16/231/OHK3/3T/13 of the Czech Technical University in Prague. References [1] C. Iber. The AASM manual for the scoring of sleep and associated events: rules, terminology and technical specifications. American Academy of Sleep Medicine, 2007. [2] A. J. Blaivas, R. Patel, D. Hom, et al. Quantifying microsleep to help assess subjective sleepiness. Sleep medicine 8(2):156–159, 2007. doi:10.1016/j.sleep.2006.06.011. [3] J. Santamaria, K. H. Chiappa. The eeg of drowsiness in normal adults. Journal of clinical Neurophysiology 4(4):327–382, 1987. doi:10.1097/00004691-198710000-00002. [4] P. Artaud, S. Planque, C. Lavergne, et al. An on-board system for detecting lapses of alertness in car driving. 14th ESV conference, session 2, intelligent vehicle highway system and human factors, Munich, Germany 1973. [5] S. K. Lal, A. Craig. Driver fatigue: electroencephalography and psychological assessment. Psychophysiology 39(3):313–321, 2002. doi:10.1017/S0048577201393095. [6] J. Santamaria, K. H. Chiappa. The EEG of drowsiness. Demos Publications, 1987. [7] S. K. Lal, A. Craig. Electroencephalography activity associated with driver fatigue: Implications for a fatigue countermeasure device. Journal of Psychophysiology 15(3):183, 2001. [8] L. Torsvall, et al. Sleepiness on the job: continuously measured eeg changes in train drivers. Electroencephalography and clinical Neurophysiology 66(6):502–511, 1987. doi:10.1016/0013-4694(87)90096-4. [9] S. K. Lal, A. Craig. Reproducibility of the spectral components of the electroencephalogram during driver fatigue. International Journal of Psychophysiology 55(2):137–143, 2005. doi:10.1016/j.ijpsycho.2004.07.001. [10] B. T. Jap, S. Lal, P. Fischer. Inter-hemispheric electroencephalography coherence analysis: assessing brain activity during monotonous driving. International Journal of Psychophysiology 76(3):169–173, 2010. [11] W. W. Wierwille, L. A. Ellsworth. Evaluation of driver drowsiness by trained raters. Accident Analysis & Prevention 26(5):571–581, 1994. doi:10.1016/0001-4575(94)90019-1. [12] C. Papadelis, Z. Chen, C. Kourtidou-Papadeli, et al. Monitoring sleepiness with on-board electrophysiological recordings for preventing sleep-deprived traffic accidents. Clinical Neurophysiology 118(9):1906–1922, 2007. doi:10.1016/j.clinph.2007.04.031. [13] G. D. Furman, A. Baharav, C. Cahan, S. Akselrod. Early detection of falling asleep at the wheel: A heart rate variability approach. In 2008 Computers in Cardiology, pp. 1109–1112. IEEE, 2008. [14] J. F. O’Hanlon, G. R. Kelley. Comparison of performance and physiological changes between drivers who perform well and poorly during prolonged vehicular operation. In Vigilance, pp. 87–109. Springer, 1977. [15] A. Vuckovic, V. Radivojevic, A. C. Chen, D. Popovic. Automatic recognition of alertness and drowsiness from eeg by an artificial neural network. Medical Engineering & Physics 24(5):349–360, 2002. doi:10.1016/S1350-4533(02)00030-9. [16] Y. Wada, Y. Nanbu, Y. Koshino, et al. Inter-and intrahemispheric eeg coherence during light drowsiness. Clinical EEG and Neuroscience 27(2):84–88, 1996. doi:10.1177/155005949602700207. [17] M. Golz, D. Sommer, M. Chen, et al. Feature fusion for the detection of microsleep events. The Journal of VLSI Signal Processing Systems for Signal, Image, and Video Technology 49(2):329–342, 2007. doi:10.1007/s11265-007-0083-4. [18] T. Gautama, D. P. Mandic, M. M. Van Hulle. The delay vector variance method for detecting determinism and nonlinearity in time series. Physica D: Nonlinear Phenomena 190(3):167–176, 2004. [19] P.-Y. Tsai, W. Hu, T. B. Kuo, L.-Y. Shyu. A portable device for real time drowsiness detection using novel active dry electrode system. In 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 3775–3778. IEEE, 2009. [20] A. G. Correa, E. L. Leber. An automatic detector of drowsiness based on spectral analysis and wavelet decomposition of eeg records. In 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, pp. 1405–1408. IEEE, 2010. [21] E. Magosso, M. Ursino, A. Zaniboni, et al. Visual and computer-based detection of slow eye movements in overnight and 24-h eog recordings. Clinical neurophysiology 118(5):1122–1133, 2007. doi:10.1016/j.clinph.2007.01.014. [22] M. T. Peiris, P. R. Davidson, P. J. Bones, R. D. Jones. Detection of lapses in responsiveness from the eeg. Journal of neural engineering 8(1):016003, 2011. [23] X. Zhu, W.-L. Zheng, B.-L. Lu, et al. Eog-based drowsiness detection using convolutional neural networks. In IJCNN, pp. 128–134. 2014. [24] E. Magosso, F. Provini, P. Montagna, M. Ursino. A wavelet based method for automatic detection of slow eye movements: A pilot study. Medical engineering & physics 28(9):860–875, 2006. doi:10.1016/j.medengphy.2006.01.002. [25] E. Magosso, M. Ursino, A. Zaniboni, E. Gardella. A wavelet-based energetic approach for the analysis of biomedical signals: application to the electroencephalogram and electro-oculogram. Applied Mathematics and Computation 207(1):42–62, 2009. doi:10.1016/j.amc.2007.10.069. 36 http://dx.doi.org/10.1016/j.sleep.2006.06.011 http://dx.doi.org/10.1097/00004691-198710000-00002 http://dx.doi.org/10.1017/S0048577201393095 http://dx.doi.org/10.1016/0013-4694(87)90096-4 http://dx.doi.org/10.1016/j.ijpsycho.2004.07.001 http://dx.doi.org/10.1016/0001-4575(94)90019-1 http://dx.doi.org/10.1016/j.clinph.2007.04.031 http://dx.doi.org/10.1016/S1350-4533(02)00030-9 http://dx.doi.org/10.1177/155005949602700207 http://dx.doi.org/10.1007/s11265-007-0083-4 http://dx.doi.org/10.1016/j.clinph.2007.01.014 http://dx.doi.org/10.1016/j.medengphy.2006.01.002 http://dx.doi.org/10.1016/j.amc.2007.10.069 vol. 12/2017 Microsleeps and Their Detection from Biological Signals [26] M. Holub, M. Šrutová, L. Lhotská. Adaptable microsleep detection based on eog signals: A feasibility study. In Computational Intelligence for Multimedia Understanding (IWCIM), 2015 International Workshop on, pp. 1–4. IEEE, 2015. 37 Acta Polytechnica CTU Proceedings 12:32–37, 2017 1 Introduction 2 Signal changes 2.1 Changes in EEG signals 2.2 Changes in EOG signals 2.3 Changes in ECG signals 3 Detection approaches 3.1 Detection by Vuckovic et al. (published 2002) 3.2 Detection by Golz et al. (published 2007) 3.3 Detection by Furman et al. (published 2008) 3.4 Detection by Tsai et al. (published 2009) 3.5 Detection by Garcés et al. (published 2010) 3.6 Detection by Peiris et al. (published 2011) 3.7 Detection by Zhu et al. (published 2014) 3.8 Detection by Holub et al. (published 2015) 4 Conclusions Acknowledgements References