J Global Clinical Engineering Vol.5 Issue 1: 2022 8 Received March 26, 2021, accepted April 21, 2022, date of publication May 16, 2022 Comparison of Automatic Sleep Stage Classification Methods for Clinical Use By Alexei Labrada, Elsa Santos Febles, José Manuel Antelo Cuban Neuroscience Center (CNEURO), Cuba ABSTRACT Sleep stage scoring is necessary for diagnosing several sleep disorders. However, it is an intensive and repetitive task and a vital automation candidate. This work seeks to evaluate different kinds of Machine Learning based classification algorithms available in the scientific literature to determine which one fits better the clinical practice requirements. The comparison is made with a predefined experimental design, using electroencephalography, electrooculography, and electromyography signals from the polysomnographic records of the Sleep-EDFx dataset. The comparison considers the accuracy and speed of algorithms based on Linear Discriminate Analysis, Support Vector Machines, Random Forests, and Artificial Neural Networks. The latter group includes the Deep Neural Networks DeapFeatureNet, based on Convolutional Neural Networks, and DeepSleepNet, additionally based on Recurrent Neural Networks. It is determined that several of the tested algorithms boast high accuracy levels (85%). From them, DeepSleepNet is chosen as the fittest due to its considerable advantage in execution time. Nevertheless, the final result should always be reviewed by the experts. Keywords – polysomnography, sleep stage scoring, machine learning, deep learning, signal processing. Copyright © 2021. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY): Creative Commons - Attribu- tion 4.0 International - CC BY 4.0. The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. INTRODUCTION Sleep stage scoring is necessary for diagnosing several sleep disorders, including insomnia, sleep apnea, narcolepsy, and hypersomnia. According to the American Academy of Sleep Medicine (AASM), this operation entails the divi- sion of a polysomnographic record (PSG) in consecutive 30-second windows, called epochs. Each epoch has to be classified as wakefulness (W), REM sleep (R), or one of three non-REM sleep stages: N1, N2, or N3. * Addition- ally, AASM defines the rules that have to be followed to perform the scoring based on the visual examination of each epoch of the PSG record. A PSG record shows the behavior throughout the time of various electrophysiological signals. The three most http://www.globalce.org http://globalce.org http://globalce.org 9 J Global Clinical Engineering Vol.5 Issue 1: 2022 Labrada, Febles, Antelo : Comparison of Automatic Sleep Stage Classification Methods for Clinical Use important signals are (1) electrical activity in the cerebral cortex, measured using electroencephalography (EEG); (2) in the face muscles, using electromyography (EMG); and (3) the eye movements, using electrooculography (EOG). It may also include the cardiac activity or electrocardiogram (ECG), the respiratory activity, and the body movements. The scoring rules rely on identifying various patterns in the signals, including the Alpha, Beta, Theta, and Delta Activity, K complexes, Spindles, REM, and SEM.** Table 1 summarizes some of these patterns. TABLE 1. Common Patterns in Polysomnographic Signals Pattern Stage Signal Frequency Morphology Alpha Activity W, N1 EEG 8 - 13 Hz Beta Activity W, N1, R EEG 14 - 30 Hz Theta Activity NREM, R EEG 4 - 8 Hz Delta Activity N3, R EEG 0.5 - 4 Hz Spindle N2, N3 EEG 12 - 14 Hz K Complex N2, N3 EEG 0.5 - 1.5 Hz Biphasic high amplitude peak Slow waves N3 EEG 0.5 - 2 Hz High amplitude waves EEG = electroencephalography. The PSG records may last for 8 hours, so the number of epochs is close to a thousand. Therefore, the scoring process is intensive, repetitive, and prone to errors. The scientific literature describes many algorithms that allow the automation of the process by using various Machine Learning techniques. However, the low inter-scorer agreement level,1,2 among other limitations, has limited the accuracy of the algorithms and, hence, the reach of the automation process. For instance, Fraiwan et al. 3 use the Continuous Wavelet Transform of the EEG signals as features and a Linear Dis- criminant Analysis (LDA) based classifier. As a result, they reach an 84% accuracy level with the MIT-BIH4,5 dataset records. Susmakova & Krakovska6 also use an LDA-based classifier, but their algorithm extracts a wider variety of features from different signals. Furthermore, they prove the importance of the information contained within the EOG and EMG signals to discriminate some of the stages. Koley & Dey7 evaluate the performance of a Support Vector Machine (SVM) based classifier with different combinations of features. Their algorithm has an 89% accuracy on their own dataset, close to the inter-scorer agreement level. Aboalayon et al.8 also use an SVM clas- sifier, reaching a 92.5% accuracy on records from the Sleep-EDF5,9 dataset. Set et al. 10 compare the performance of different classi- fiers, including Decision Trees (DT), Random Forests (RF), SVM, and Artificial Neural Networks (ANN). Moreover, they employ various feature extraction techniques, counting the Discrete Wavelet Transform (DWT). As a result, they determine that the RF obtains the best results, reaching a 97% accuracy with their own records. Finally, Aboalayon et al.11 compare the DT, SVM, ANN, K-Nearest Neighbors, Naive Bayes (NB), and LDA classifiers. In their work, the DT classifier obtained the best results with a 93% accuracy on records from the Sleep-EDF dataset. Finally, the Deep Learning techniques also have gained a foothold in sleep stage scoring. For example, Zhang et al.12 propose using a Recurrent Neural Network (RNN) as a classifier but using conventional feature extraction methods. Their algorithm reaches 80.25% accuracy on the SHHS5 dataset records. Alternatively, Yildirim et al.13 present a Convolution Neural Network-based algorithm that uses convolutional layers for feature extraction, with a 91% accuracy on Sleep-EDF records. Additionally, Supratak et al.14 use a Convolutional Neural Networks (CNN) combined with an RNN, reaching an 82% accuracy on the same records. The goal of this work is to select a sleep stage scoring algorithm to facilitate the work of the experts. Furthermore, the algorithm should be included in a software system J Global Clinical Engineering Vol.5 Issue 1: 2022 10 Labrada, Febles, Antelo : Comparison of Automatic Sleep Stage Classification Methods for Clinical Use for the clinical analysis of polysomnographic records. Therefore, the selection should be based on the accuracy of the predictions and consider execution time and the general availability of the input data. With that in mind, the performance of several algorithms from the scientific literature will be compared using the same records and in similar conditions. MATERIALS The work uses PSG records from the Sleep Cassette dataset belonging to Sleep-EDFx.5,9 The dataset has 153 subjects between 25 and 101 years old and was scored by several experts following the Rechtschaffen and Kales (R & K)15 rules. The records include two EEG and one EOG signal, samples at 100 Hz, and one EMG signal at 1 Hz. Both EOG and EMG signals are considered in this work, but only the Fpz-Cz channel is used from the EEG signals. That way, all the implemented algorithms depend only on the minimum parameters of any PSG record.1 The dataset is split into two parts of approximately the same number of records. The first half contains the subjects with identifications 00 through 38 and is reserved for training the scoring algorithms. The second one, with subjects 40 through 82, is used to evaluate and compare the performance of said algorithms. METHODS The analyzed algorithms’ execution time can be split into three main phases: Data preprocessing, feature ex- traction, and classification. The preprocessing and feature extraction phases are implemented in the Python and C# programming languages. For the classification, the work additionally employs the Weka software system16,17 from the University of Waikato, New Zealand. Preprocessing The goal of the preprocessing phase is to prepare the data for the feature extraction phase. To achieve it, all signals are uniformly sampled at 100 Hz, and no digital filtering is applied beyond what is already included in the dataset: 0.5 to 100 Hz range for EEG and EOG and 0.7 to 16 Hz, for EMG. The records are segmented in 30-second windows that match the epochs that will be classified later. Also, the third and fourth non-REM sleep stages from R & K are combined into one Slow Wave Sleep or N3 stage1,7 to fit better the AASM stages. Additionally, the unknown or invalid sleep stages are excluded from consideration. The wake stages before the first and posterior to the last sleep stages are also excluded from the training dataset records. The latter operation reduces the disparities in the amounts of epochs classified with each sleep stage. Besides, more importantly, for the RNN classifiers, it does not affect the continuity of a record’s epochs. Feature Extraction The feature extraction phase obtains descriptive values that reflect the information inside the relevant signals for the classification process. The values or features used in this work are obtained by analyzing the signals in each epoch in the time domain, frequency domain, time-frequency domain, and other nonlinear means. Descriptive Statistics These features are obtained by computing descriptive statistics from the signal’s samples. The Mean, Variance, Kurtosis, Skewness, and 75th Percentile have been em- ployed in this work. Entropy Entropy is a measure of the irregularity of a signal in the time domain.18 Equation 1 shows the formula proposed by Shannon for this measure: where p(x_i ) is the probability of a signal sample hav- ing the value x_i. Other estimation methods, including the Approximate Entropy, are displayed in equation 2. The values of ϕ can be obtained using an algo- rithm that represents the signal in the phase domain Xi={xi, x(i+1), ..., x(i+(m-1)) } and calculates the distance between those patterns using the L1 norm. Then, (1) (2) 11 J Global Clinical Engineering Vol.5 Issue 1: 2022 Labrada, Febles, Antelo : Comparison of Automatic Sleep Stage Classification Methods for Clinical Use where is the amount of Xj patterns that satisfy ∥Xi-Xj ∥1≤r. In this work, the pattern length (m) is 2 and r is the standard deviation of the signal in the epoch, multiplied by 0.1, as estimated in.18 Largest Lyapunov Exponent The Largest Lyapunov Exponent (LLE) indicates how unpredictable a signal is. It has been demonstrated that it can help discriminate the N1 and N2 stages.7 The algorithm proposed by19 allows estimating LLE by calculating the distances between the most similar trajectories, which are also distant in the time domain. Equation 4 describes this distance, where τ is the threshold in time domain and Xi={xi, x(i+J), ..., x(i+(m-1)J)} is a trajectory in phase domain. Once the distances have been calculated, the LLE can be obtained using linear regression with equation 5. In our work we use the values 10 and 7 for m and J, re- spectively, while τ is the mean period of the signal (MNF-1). Fractal Dimension The fractal dimension estimates the fractional dimen- sions of the geometric shape of a signal in the time do- main.18 This measure is especially useful for recognizing the N3 stage.7 The Higuchi algorithm calculates the fractal dimen- sion as the slope of the mean squares fit of the values of log(L(k)) against log(1/k) for k between 1 and kmax. The values of L(k) are calculated using the equation 6: where Lm(k) is the mean length of the sequence calculated with equation 7: In this work we use the value 40 for kmax, that was estimated in.18 Discrete Fourier Transform The Fast Fourier Transform (FFT) algorithm efficiently estimates the frequency spectrum. The spectrum can be used to obtain the mean frequency of the signal, the spectral entropy, and the relative spectral density of the relevant frequency bands (Table 1). The mean frequency can be calculated using equation 8: where M in the amount of frequency bins, fi are the fre- quency values and P is the normalized spectral frequency (∑Pi=1).20 Similarly, the spectral entropy of a frequency band can be obtained from equation 9: where fl and fh are the minimum and maximum fre- quencies, respectively and Nf is the amount of frequency bins in the range [fl,fh].18 High Order Spectra The High Order Spectra analysis can extract features related to third-order statistics of a signal.21 Before cal- culating the features, the Bispectrum has to be estimated using equation 10, (3) (4) (5) (6) (7) (8) (9) J Global Clinical Engineering Vol.5 Issue 1: 2022 12 Labrada, Febles, Antelo : Comparison of Automatic Sleep Stage Classification Methods for Clinical Use where Xi is the Short-Time Fourier Transform (STFT) of the signal on the i-th window and W is the number of windows. The STFT in a vicinity of xi is the FFT of the product of the signal and a window function centered on xi. 22 In our work, we use 2 seconds long Haan windows, with 1 second (50%) of overlap between consecutive windows. The Bispectrum is symmetric in both axes, so its domain of interest is defined in the expression 11. Once the Bispectrum is calculated, it is possible to calculate its mean amplitude, the Normalized Bispectral Entropy (equation 12), its logarithmic sum (equation 13) and its mean frequency (equation 14): Wavelet Transform The Wavelet Transforms translate a signal into the time-frequency domain. The transformation approxi- mates the signal inside a time window by a Wavelet base (ψ) using different time scales.22 The scale factors are inversely proportional to the frequency of the Wavelet base, as stated in equation 15, where Ts is the sampling period and fψ is the mean frequency of the Wavelet base (3). The DWT decomposes the signal in two coefficient vectors with N/2 values, satisfying where Hψ and Gψ are dual filters with sub-sampling, related to the Wavelet base.22 The a1 vector contains an approximation of the original signal in the frequency range [0,1/4 fs ], while d1 is a detail vector in the frequency range [1/4 fs,1/2 fs ], where fs is the sampling frequency.10 The DWT can be computed again from vector a1, in order to obtain the vectors a2 and d2 with frequency ranges [0,1/8 fs ] and [1/8 fs,1/4 fs ], respectively. Thus, successively, the signal can be decomposed in L levels, after which the vectors d1,d2,...,dL,aL belong to different frequency bands. The entropy of each relevant frequency band (Table 1) along the epoch in question can be calculated from the transform. We use the Daubechies function (db1) as the Wavelet base for the EOG signals and the reverse biorthogonal function (rbio3.3) for the EEG signals. Given the 100 Hz sampling frequency of the signals, once they are decomposed into 5 levels, the frequencies of the coefficient vectors approximately match the frequency bands in Table 1. Classification The classification phase is responsible for assigning a sleep stage to each epoch contingent on the features extracted from it. In our work, we use classifiers based on Linear Discriminate Analysis,3 SVMs,23 RF,23,24 ANN, and NB.23 Several kinds of Neural Networks have been analyzed, including Multilayer Perceptrons (MLP),10,25 CNN, and RNN. Specifically, we have tested the networks DeepFea- tureNet (DFN) and DeepSleepNet (DSN),14 implemented on Python using Tensorflow. The former is a CNN, while the latter is a hybrid network combining a CNN and an RNN. Both algorithms use CNN for feature extraction, so they do not require the methods described in section Feature Extraction. The implementation proposed for a single signal has been expanded to process the EOG, EMG, and EEG signals.14 This was achieved by taking advantage of the (10) (11) (12) (13) (14) (15) (16) 13 J Global Clinical Engineering Vol.5 Issue 1: 2022 Labrada, Febles, Antelo : Comparison of Automatic Sleep Stage Classification Methods for Clinical Use capacity of CNN layers to process several input channels and by increasing the size of the filters proportionally to the number of channels. The DFN network has been trained with 75 epochs, while DSN has required 25 more in fine-tuning. The source code is available at https:// github.com/ALabrada/deepsleepnet. For the remaining classifiers, it has been used the implementations available in Weka, using their respective default parameters. Evaluation The performance of each algorithm has been analyzed, considering the accuracy (Acc) and Cohen’s kappa coef- ficient. Additionally, the classification performance of the individual stages is considered using the Precision (PR) and Recall (RE) metrics. RESULTS The classification algorithms have been trained with the first half of the PSG records of the Sleep Cassette da- taset. The set has 76 records that belong to 39 different subjects with identifiers 00 through 38. Table 2 shows the distribution of the stages assigned by the experts to the 74354 epochs that have been used from those records. The 10-fold cross-validation technique has been used to estimate the hyper-parameters of the models and the validation error. Table 3 shows the estimated errors. The trained classifiers have been tested using the second half of the Sleep Cassette dataset, and the results have been compared. The set has 77 records that belong to 39 subjects with identifiers 40 through 82. A total of 68.8% of the 208349 epochs belong to the wake stage. TABLE 2. Sleep Stage Distribution of the Analyzed Epochs Stage Training Testing (partial) Testing (full) Count Percent Count Percent Count Percent W 14884 20.0 33410 33.9 143265 68.8 N1 7536 10.1 14013 14.2 14013 6.7 N2 30143 40.5 33906 34.4 33906 16.3 N3 7954 10.7 5104 5.2 5104 2.4 R 13837 18.6 12062 12.2 12062 5.8 Total 74354 100.0 98495 100.0 208349 100.0 TABLE 3. Validation Error using the Training Records Type Acc Kappa PR RE W N1 N2 N3 R W N1 N2 N3 R LDA 77.29 0.6882 0.902 0.438 0.775 0.817 0.785 0.804 0.398 0.869 0.843 0.695 NB 64.79 0.5324 0.748 0.320 0.738 0.507 0.659 0.689 0.249 0.668 0.925 0.619 RF 83.09 0.7674 0.868 0.623 0.830 0.896 0.822 0.904 0.365 0.906 0.863 0.825 SVM 79.49 0.7155 0.858 0.501 0.788 0.872 0.787 0.867 0.286 0.894 0.848 0.745 MLP 80.60 0.7334 0.883 0.515 0.808 0.867 0.790 0.874 0.342 0.885 0.838 0.794 DFN 74.27 0.6630 0.969 0.287 0.883 0.652 0.822 0.789 0.692 0.721 0.912 0.658 DSN 78.10 0.7055 0.906 0.326 0.854 0.756 0.911 0.901 0.427 0.812 0.726 0.817 AVG 76.80 0.6865 0.876 0.430 0.811 0.767 0.797 0.833 0.394 0.822 0.851 0.736 https://github.com/ALabrada/deepsleepnet https://github.com/ALabrada/deepsleepnet J Global Clinical Engineering Vol.5 Issue 1: 2022 14 Labrada, Febles, Antelo : Comparison of Automatic Sleep Stage Classification Methods for Clinical Use Following the procedure that has been described in section Preprocessing, the disparity between stages can be decreased by reducing this quantity to the 33.9%. Table 4 shows a performance comparison between the algorithms using only the selected epochs, while Table 5 shows the same comparison, but with all the epochs. Finally, Table 6 compares the execution time of the algorithms while classifying the whole test dataset. The execution time of the algorithms that use classifiers imple- mented in Weka is further split into the feature extraction and classification phases. The data has been collected in a personal computer with an Intel Core i5-4570 processor (CPU), 16 GB of DDR3-1600 memory (RAM), and executed in Microsoft .NET Framework. DISCUSSION The results show that the test error is less than the validation error when using the full records, but it is greater when using the selected subset of the epochs. This apparent discrepancy can be explained due to the previously mentioned high proportion of epochs classified with wake stages. Every one of the analyzed algorithms obtains relatively high precision and recall results clas- sifying this stage. In contrast, all algorithms attain poor precision and recall results that classify the N1 stage in absolute and relative terms. This behavior is consistent with other studies from the scientific literature,24 especially those TABLE 4. Performance Comparison of the Classifiers using the Partial Test Dataset Type Acc Kappa PR RE W N1 N2 N3 R W N1 N2 N3 R LDA 69.43 0.5776 0.911 0.385 0.664 0.465 0.723 0.759 0.279 0.840 0.752 0.563 NB 55.09 0.4109 0.841 0.329 0.594 0.241 0.555 0.604 0.231 0.582 0.954 0.515 RF 73.98 0.6335 0.858 0.504 0.692 0.637 0.737 0.853 0.183 0.887 0.756 0.652 SVM 72.93 0.6213 0.866 0.433 0.697 0.591 0.718 0.842 0.215 0.863 0.774 0.619 MLP 71.22 0.6009 0.817 0.399 0.724 0.558 0.682 0.856 0.244 0.784 0.770 0.634 DFN 67.78 0.5670 0.968 0.303 0.743 0.517 0.898 0.697 0.640 0.734 0.776 0.454 DSN 73.88 0.6308 0.864 0.347 0.772 0.812 0.959 0.894 0.419 0.744 0.553 0.675 AVG 69.19 0.5774 0.875 0.386 0.698 0.546 0.753 0.786 0.316 0.776 0.762 0.587 TABLE 5. Performance Comparison of the Classifiers using the Full Test Dataset Type Acc Kappa PR RE W N1 N2 N3 R W N1 N2 N3 R LDA 83.45 0.6804 0.981 0.340 0.644 0.429 0.655 0.913 0.279 0.840 0.752 0.563 NB 69.02 0.4682 0.966 0.222 0.504 0.197 0.380 0.766 0.231 0.582 0.954 0.515 RF 86.43 0.7263 0.966 0.466 0.666 0.622 0.711 0.947 0.183 0.887 0.756 0.652 SVM 85.73 0.7147 0.969 0.382 0.679 0.563 0.675 0.942 0.215 0.863 0.774 0.619 MLP 85.10 0.699 0.955 0.347 0.709 0.532 0.666 0.947 0.244 0.784 0.770 0.634 DFN 80.14 0.6366 0.991 0.270 0.687 0.415 0.879 0.862 0.579 0.774 0.772 0.432 DSN 85.30 0.6973 0.953 0.318 0.773 0.812 0.967 0.970 0.537 0.688 0.442 0.474 AVG 82.17 0.6604 0.969 0.335 0.666 0.510 0.705 0.908 0.324 0.774 0.746 0.556 15 J Global Clinical Engineering Vol.5 Issue 1: 2022 Labrada, Febles, Antelo : Comparison of Automatic Sleep Stage Classification Methods for Clinical Use using the Sleep-EDFx dataset.13,14,26–28 The DFN and DSN algorithms reach around 20% higher recall measures for this stage, but its influence is mitigated by lower values in other stages. The low classification accuracy of the N1 stage can affect the result of the sleep quality analysis,29 which makes the algorithms unsuitable for standalone usage and, thus, require the intervention of the experts. From the first five algorithms, the ones using more conventional strategies, the RF-based classifier obtains the best results. This confirms the conclusions that were reached by previous studies.10,30 Furthermore, SVM, MLP, and LDA also obtain satisfactory results according to both performance metrics. From the two last algorithms based on Deep Learning, DSN reaches superior results in all metrics other than DFN. However, during validation, our implementation of DSN is 4% lower in accuracy and 6% lower in Kappa score than the one reported by Supratak et al.14 with the same dataset, but using different hyper-parameters and half of the PSG records. Regarding the traditional algorithms, the accuracy of DSN classifying the test dataset is equivalent to the accuracy of RF within 1%. Considering that several of the algorithms reach similar accuracy levels, their execution times are used as tie-breakers. The results in Table 6 prove that, from the analyzed algorithms, the ones based on Deep Learning require a significantly lower amount of time to identify the sleep stages of a PSG record. CONCLUSIONS As part of our work, we have compared the performance of a wide range of sleep stage scoring algorithms avail- able in the scientific literature to find the one that better matches clinical use requirements. With that in mind, ac- curacy and speed are used as the selection criteria for the comparison. The results prove that the RF, SVM, MLP, and DSN algorithms reach the greater accuracy levels while classifying, exceeding 85% in this metric and 0.69 in Cohen’s kappa. Moreover, from them, DSN is significantly faster, requiring less than 30 seconds to score a record’s epochs on average. The combination of both criteria determines that DSN is the most appropriate sleep stage scoring al- gorithms for the context of the clinical practice, from the set of candidates taken into consideration. Nevertheless, the algorithms are much less accurate in classifying the N1 stage, so the experts should review the sleep stage scoring performed by DSN. REFERENCES 1. Malhotra RK, Avidan AY. Atlas of sleep medicine. In: 2nd ed. Elsevier; 2014. pp. 77–99. 2. Daker-Hopfe H, Anderer P, Zeitlhofer J, Boeck M, Dorn H, Gruber G, et al. Interrater reliability for sleep scoring according to the Rechtschaffen & Kales and the new AASM standard. J Sleep Res. 2009;78–84. https://doi. org/10.1111/j.1365-2869.2008.00700.x 3. Fraiwan L, Lweesy K, Khasawneh N, Fraiwan M, Wenz H, Dickhaus H. Classification of sleep stages using multi- wavelet time frequency entropy and LDA. Methods of information in medicine. 2010 Jan;49:230–7. https:// doi.org/10.3414/ME09-01-0054 4. Ichimaru Y, Moody GB. Development of the poly- somnographic database on CD-ROM. Psychiatry and Clinical Neurosciences. 1999;53:175–7. https://doi. org/10.1046/j.1440-1819.1999.00527.x 5. Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, et al. PhysioBank, PhysioToolkit, and PhysioNet : Components of a new research re- source for complex physiologic signals. Circulation. 2000 Jun 13;101(23). https://doi.org/10.1161/01. CIR.101.23.e215 TABLE 6. Comparison of the Execution Time of the Algorithms Type Time Extraction Classification Total Average per record LDA 54758.69 s 6.08 s 54764.77 s 711.23 s NB 54758.69 s 5.07 s 54763.76 s 711.22 s RF 54758.69 s 10.97 s 54769.66 s 711.29 s SVM 54758.69 s 1.41 s 54760.10 s 711.17 s MLP 54758.69 s 2.64 s 54761.33 s 711.19 s DFN - - 766.55 s 9.66 s DSN - - 1695.68 s 22.02 s https://doi.org/10.1111/j.1365-2869.2008.00700.x https://doi.org/10.1111/j.1365-2869.2008.00700.x https://doi.org/10.3414/ME09-01-0054 https://doi.org/10.3414/ME09-01-0054 https://doi.org/10.1046/j.1440-1819.1999.00527.x https://doi.org/10.1046/j.1440-1819.1999.00527.x https://doi.org/10.1161/01.CIR.101.23.e215 https://doi.org/10.1161/01.CIR.101.23.e215 J Global Clinical Engineering Vol.5 Issue 1: 2022 16 Labrada, Febles, Antelo : Comparison of Automatic Sleep Stage Classification Methods for Clinical Use 6. Susmakova K, Krakovska A. Discrimination ability of individual measures used in sleep stages classification. Artificial Intelligence in Medicine. 2008 Nov;44(3):261– 77. https://doi.org/10.1016/j.artmed.2008.07.005 7. Koley B, Dey D. An ensemble system for automatic sleep stage classification using single channel EEG signal. Com- puters in Biology and Medicine. 2012;42(12):1186–95. https://doi.org/10.1016/j.compbiomed.2012.09.012 8. Aboalayon K, Ocbagabir H, Faezipour M. Efficient sleep stage classification based on EEG signals. In: Applica- tions and technology conference. 2014. https://doi. org/10.1109/LISAT.2014.6845193 9. Kemp B, Zwinderman AH, Tuk B, Kamphuisen HAC, Oberye JJL. Analysis of a sleep-dependent neuronal feedback loop: The slow-wave microcontinuity of the EEG. IEEE Transactions on Biomedical Engineering. 2000;47(9):1185–94. https://doi.org/10.1109/10.867928 10. Sen B, Peker M, Cavusoglu A, Celebi FV. A compara- tive study on classification of sleep stage based on EEG signals using feature selection and classification algorithms. J Med Syst. 2014; https://doi.org/10.1007/ s10916-014-0018-0 11. Aboalayon K, Faezipour M, Almuhammadi W, Moslehpour S. Sleep stage classification using EEG signal analysis: A comprehensive survey and new investigation. Entropy. 2016 Aug;18. https://doi.org/10.3390/e18090272 12. Zhang Y, Yang Z, Lan K, Liu X, Zhang Z, Li P, et al. Sleep stage classification using bidirectional LSTM in wear- able multi-sensor systems [Internet]. 2019. Available from: https://arxiv.org/abs/1909.11141 13. Yildirim O, Baloglu UB, Acharya UR. A deep learning model for automated sleep stages classification us- ing PSG signals. Int J Environ Res Public Helth. 2019; https://doi.org/10.3390/ijerph16040599 14. Supratak A, Dong H, Wu C, Guo Y. DeepSleepNet: A model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Transactions on Neural Sys- tems and Rehabilitation Engineering. 2017; https:// doi.org/10.1109/TNSRE.2017.2721116 15. Rechstaffen A, Kales A. A manual of standardized terminology, techniques and scoring system for sleep stages of human subjects. In: A manual of standardized terminology. Los Angeles, California: University of California; 1986. 16. Frank E, Hall MA, Holmes G, Kirkby R, Pfahringer B, Witten IH. Weka: A machine learning workbench for data mining. In: Data mining and knowledge discovery handbook: A complete guide for practitioners and researchers [Internet]. Berlin: Springer; 2005. pp. 1305–14. Available from: http://researchcommons. waikato.ac.nz/handle/10289/1497 17. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH. The WEKA data mining software: An update. SIGKDD Explorations. 2009;11(1):10–8. https://doi. org/10.1145/1656274.1656278 18. Sabeti M, Katebi S, Boostani R. Entropy and complex- ity measures for EEG signal classification of schizo- phrenic and control participants. Artificial Intelligence in Medicine. 2009 Nov;47(3):263–74. https://doi. org/10.1016/j.artmed.2009.03.003 19. Rosenstein MT, Collins JJ, Luca CJD. A practical method for calculating largest Lyapunov exponents from small data sets. Physica D: Nonlinear Phe- nomena. 1993 Jun;65(1-2):117–34. https://doi. org/10.1016/0167-2789(93)90009-P 20. Phinyomark A, Thongpanja S, Hu H, Phukpattaranont P, Limsakul C. The usefullness of mean and median fre- quencies in electromiography analysis, computational intelligence in electromiography analysis - a perspec- tive on current applications and future challenges. In IntechOpen; 2012. https://doi.org/10.5772/50639 21. Acharya R, Chua EC-P, Chua KC, Min LC, Tamura T. Analysis and automatic identification of sleep stages using higher order spectra. International Journal of Neural Systems. 2010 Nov;20(6):509–21. https://doi. org/10.1142/S0129065710002589 22. Stark H-G. Wavelets and signal processing: An appli- cation-based introduction. Springer; 2005. https:// doi.org/10.1007/3-540-27481-2 23. Tzimourta KD, Tsilimbaris A, Tzioukalia K, Tzallas AT, Tsipouras MG, Astrakas LG, et al. EEG-based automatic sleep stage classification. Biomedical Journal of Scien- tific & Technical Research. 2018;7(4):6032–7. https:// doi.org/10.26717/BJSTR.2018.07.001535 https://doi.org/10.1016/j.artmed.2008.07.005 https://doi.org/10.1016/j.compbiomed.2012.09.012 https://doi.org/10.1109/LISAT.2014.6845193 https://doi.org/10.1109/LISAT.2014.6845193 https://doi.org/10.1109/10.867928 https://doi.org/10.1007/s10916-014-0018-0 https://doi.org/10.1007/s10916-014-0018-0 https://doi.org/10.3390/e18090272 https://arxiv.org/abs/1909.11141 https://doi.org/10.3390/ijerph16040599 https://doi.org/10.1109/TNSRE.2017.2721116 https://doi.org/10.1109/TNSRE.2017.2721116 http://researchcommons.waikato.ac.nz/handle/10289/1497 http://researchcommons.waikato.ac.nz/handle/10289/1497 https://doi.org/10.1145/1656274.1656278 https://doi.org/10.1145/1656274.1656278 https://doi.org/10.1016/j.artmed.2009.03.003 https://doi.org/10.1016/j.artmed.2009.03.003 https://doi.org/10.1016/0167-2789(93)90009-P https://doi.org/10.1016/0167-2789(93)90009-P https://doi.org/10.5772/50639 https://doi.org/10.1142/S0129065710002589 https://doi.org/10.1142/S0129065710002589 https://doi.org/10.1007/3-540-27481-2 https://doi.org/10.1007/3-540-27481-2 https://doi.org/10.26717/BJSTR.2018.07.001535 https://doi.org/10.26717/BJSTR.2018.07.001535 17 J Global Clinical Engineering Vol.5 Issue 1: 2022 Labrada, Febles, Antelo : Comparison of Automatic Sleep Stage Classification Methods for Clinical Use 24. Fraiwan L, Lweesy K, Khasawneh N, Wenz H, Dickhaus H. Automated sleep stage identification system based on time–frequency analysis of a single EEG channel and random forest classifier. Computer Methods and Programs in Biomedicine. 2012;108:10–9. https:// doi.org/10.1016/j.cmpb.2011.11.005 25. Ronzhina M, Janousek O, Kolarova J, Novakova M, Hon- zik P, Provaznik I. Sleep scoring using artificial neural networks. Sleep Medicine Reviews. 2012;16(3):251–63. https://doi.org/10.1016/j.smrv.2011.06.003 26. Dong H, Supratak A, Pan W, Wu C, Matthews PM, Guo Y. Mixed neural network approach for temporal sleep stage classification. IEEE Transactions on Neural Sys- tems and Rehabilitation Engineering. 2018; https:// doi.org/10.1109/TNSRE.2017.2733220 27. Koushik A, Amores J, Maes P. Real-time sleep staging using deep learning on a smartphone for a wearable EEG. CoRR [Internet]. 2018;abs/1811.10111. Available from: http://arxiv.org/abs/1811.10111 28. Mousavi S, Afghah F, Acharya UR. SleepEEGNet: Auto- mated sleep stage scoring with sequence to sequence deep learning approach. Pławiak P, editor. PLOS ONE [Internet]. 2019 May;14(5). Available from: http:// dx.doi.org/10.1371/journal.pone.0216456 29. Mendonça F, Mostafa SS, Morgado-Dias F, Ravelo-García AG, Penzel T. A review of approaches for sleep quality analysis. IEEE Access. 2019;7:24527–46. https://doi. org/10.1109/ACCESS.2019.2900345 30. Boostani R, Karimzadeh F, Torabi-Nami M. A comparative review on sleep stage classification methods in patients and healthy individuals. HAL. 2017 Mar;140:77–91. https://doi.org/10.1016/j.cmpb.2016.12.004 https://doi.org/10.1016/j.cmpb.2011.11.005 https://doi.org/10.1016/j.cmpb.2011.11.005 https://doi.org/10.1016/j.smrv.2011.06.003 https://doi.org/10.1109/TNSRE.2017.2733220 https://doi.org/10.1109/TNSRE.2017.2733220 http://arxiv.org/abs/1811.10111 http://dx.doi.org/10.1371/journal.pone.0216456 http://dx.doi.org/10.1371/journal.pone.0216456 https://doi.org/10.1109/ACCESS.2019.2900345 https://doi.org/10.1109/ACCESS.2019.2900345 https://doi.org/10.1016/j.cmpb.2016.12.004