Acta Polytechnica CTU Proceedings https://doi.org/10.14311/APP.2023.41.0020 Acta Polytechnica CTU Proceedings 41:20–27, 2023 © 2023 The Author(s). Licensed under a CC-BY 4.0 licence Published by the Czech Technical University in Prague POSSIBILITIES OF USING COMMERCIAL ELECTRONICS TO MEASURE DRIVER PHYSIOLOGICAL FUNCTIONS IN A VEHICLE SIMULATOR David Lehet Czech Technical University in Prague, Faculty of Transportation Sciences, Department of Vehicle Technology, Konviktská 20, 110 00 Prague, Czech Republic correspondence: David.Lehet@cvut.cz Abstract. The aim of this paper is to explore the potential of using inexpensive, commercial electronics such as a smartwatch or chest belt to measure human physiological functions while driving in vehicle simulator, both to improve safety and to use these devices in other experiments to facilitate their conduct and replace more complex laboratory equipment. In the first part of the paper, the signs of stress and cognitive load through physiological functions are researched, then the possibilities of measuring these signs are explored. This is followed by an experiment in which two commonly available devices are compared in terms of accuracy and usefulness for measurement during the experiment. Keywords: ECG, heart rate, vehicle safety, design of experiment, vehicle simulator. 1. Introduction For the design of the vehicle, and in particular the interaction of its systems with the driver, it is essential that the systems are properly tested. One of the best tools for this testing is a vehicle simulator, which allows both standard and hazardous situations to be simulated repeatedly in a safe environment and under controlled conditions. Another important part of the design process is the ability to record objective data from the participants in the experiment. One part of this objective data can be the driver’s physiological functions, the evaluation of which can be used to objectively assess their stress and overall workload. Traditionally, laboratory tools such as ECG or EEG are used for these measurements. Although these in- struments are very accurate, they carry several dis- advantages. In particular, they are more complex to evaluate and also introduce discomfort and thus noise into the measurement. The aim of this paper is therefore to explore the possibility of using commercial electronics in the form of a smartwatch or chest strap to measure the heart rate of a participant in an experiment. 1.1. Physiological functions and their measurement This part of the paper describes choosen physiological functions and how they could be measured. 1.2. EEG EEG, or electroencephalography, captures the electri- cal activity of the human brain [1]. To measure EEG, electrodes that sense electrical activity must be placed on the head (surface of the skull) of the person being measured. The number of electrodes varies according to the specific EEG application [2]. Yan et al. [3] conducted an experiment in which they compared the accuracy using 16 and 63 electrodes. The results showed that the reduction in accuracy when using only 16 electrodes is small, which can be used in designing further experiments. Much research has been done on the principles of driver distraction monitoring using EEG. Of these, for example, Lin et al. [4] found experimentally that driver distraction correlates with increased frontal theta and beta wave activity (types of brain activity monitored by EEG). Zhang et al. conducted research in which they showed that EEG is useful for observing how a driver notices various stimuli (for example, when interacting with assistance systems). They conducted the experi- ment in both a simulator and a real vehicle and note that the EEG was more prone to inaccuracies in the in-vehicle experiment [5]. Kumar et al. in their research used EEG to capture both visual and cognitive distraction. As a result of the measurements, the EEG was highly accurate at measuring both types of distraction [6]. In recent years, there has been an increasing num- ber of studies on this topic focusing on new ways of processing data obtained by EEG, especially using neural networks, for example [7–10]. 1.3. ECG Electrocardiography is another method that measures the electrical signals generated by the human body, in this case the signals produced by the heart. The basic output is activity (electrical signal) over time. This signal can then be used to calculate heart rate, heart rate variability (the evolution of the time differences between successive heartbeats), and possibly even respiratory rate [8]. 20 https://doi.org/10.14311/APP.2023.41.0020 https://creativecommons.org/licenses/by/4.0/ https://www.cvut.cz/en vol. 41/2023 Driver Physiological Functions in a Vehicle Simulator The basic elements of the ECG curve are the P- wave – a smaller deflection indicating depolarization of the atria, the P-R interval – the time between the first deflection of the P-wave and the first deflection of the QRS complex – this represents depolarization of the ventricles, the time difference between the peaks of the R-waves represents the heart rate. The T-wave represents atrial repolarization [11]. Electrodes are used to capture the ECG signal, as in EEG, and in this case they are placed on the limbs and chest of the subject [12]. Medical ECGs use 12 electrodes, while other devices (such as smartwatches) make contact with the skin differently. In the case of smartwatches, this is a single-lead ECG (e.g. con- ducted through a circuit connection between the back of the watch, the body of the person wearing the watch and a finger placed on another part of the watch) [13]. ECG is more suitable for measuring driver stress, as shown for example by Keshan et al. [14] when they used ECG data to classify three levels of stress with a success rate of 88.24%, detecting high levels of stress with a success rate of 100% (naive Bayes classifier with the difference in mean heart rate as a feature) or 98% (decision tree and eight different features). Pourmohammadi and Maleki [15] in their research compared EMG (electrical muscle activity) and ECG in stress detection. Again, using ECG, they were able to detect different levels of stress with high accuracy (greater than 95%). It can be seen that when using ECG, the heart rate is used as one of the signs for measurement. Heart rate variability is also used quite often. Therefore, if there were a way to measure these variables without ECG (for example, the aforementioned smart watches), this would lead to a simplification of the experimental process. 1.4. GSR The principle of the GSR measurement is to apply a weak and constant (undetectable to humans) voltage to the skin and to observe how the skin conductivity changes depending on the stimuli (changes due to perspiration). Sikder et al. [16] used GSR measurements to deter- mine the cognitive load of students taking an online exam. There was an attempt to asses stress based on the GSR time domain during thirteen different tasks with three different levels of stress. In a research focusing on comparing cognitive load measurement options, Tervonen et al. [17] describe the capabilities of different wearable sensors and the processing of data from them using different lengths of data. Among other things, they conclude that cognitive load can be detected from GSR but more slowly than from ECG. Borisov et al. [18] used GRS as one component to monitor cognitive load using human physiological manifestations. A combination of the use of GSR and EEG was proposed by Manikandan et al. [19]. They proposed a system that monitors driver distraction based on these two variables. The measured signal is processed and divided into time and frequency domain. Accord- ing to the authors, the proposed system is capable of accurately measuring fatigue and auditory and visual distraction. The research shows that from the physiological sig- nals it is possible to monitor distraction (EEG) as well as stress and cognitive load (ECG, GSR). The next section will focus on identifying the possibili- ties of commercial sensors (smart watches and similar devices) whose application would facilitate the mea- surement process. Advances in wearable electronics (miniaturization of the necessary electronics to enable wearable com- puting) could allow for easier measurement within an experiment, mainly due to ease of use [20]. Today, these electronics are capable of recording user data in real time. The most important part of this sector today is mainly smartwatches [21]. Another advantage of smartwatches would be within the measurement of cognitive load, where they would not add additional problems to the participant of the experiment, such as ECG electrodes [21]. 2. Material and methods This part of the papes describes the experiment that was carried out to identify capabilities of commerical sensors. 2.1. Used sensors Several devices were used for the measurements. It was a vehicle simulator used for simulating driving in a Superb III equipped with automatic transmission. The scene used was a combination of an intramural and extramural road, with a maximum speed of 90 km.h−1. Next was an ECG sensor – VLV-Scope device (Faculty of biomedical engineering, CTU in Prague). A two-lead ECG sensor was used as a reference for the measurements. The electrodes were placed in the subclavian wells. Next was Wahoo TickrX chest belt. This was the first sensor used for comparison, it works on the principle of electrodes with contact in the lower chest area, so it is a simple ECG. The second sensor compared was a sports watch with a pulse oximeter function. The sensor is in the form of an elastic sleeve on the thumb. 2.2. Pilot measurement The aim of the pilot measurements was to see if it was possible to measure the selected devices at all. For this purpose, a run with four probands was carried out and the data from these runs were subsequently processed. Two methods were proposed for evaluation, these were compared in the pilot measurement and one of them was selected. Time synchronization was achieved by simultaneous starting and stopping the 21 David Lehet Acta Polytechnica CTU Proceedings Figure 1. Output of the ECG probe. measuring devices. Outputs of the sensors: EG probe The output of the ECG probe is a text file (see Figure 1) that can be processed in software such as MS Office Excel or Matlab. The output is from a program that allows you to measure several variables at once (in addition to the ECG, skin resistance or muscle activity, for example). Since the sensors required for this scan were not active during the experiment, other items are unusable except for the time trace and the ECG recording. Chest strap Unlike the ECG probe, the output of the chest strip measurement is just a file containing a time trace and an instantaneous heart rate reading (see Table 1). Hr Time 0 2/19/2020 10:02:25 AM 115 2/19/2020 10:02:26 AM 115 2/19/2020 10:02:28 AM Table 1. Chest strap output. Pulse oximeter The output of a pulse oximeter is similar to that of a chest belt (Table 2). Index HR SpO2 Motion Vibration 0 80 98 0 NEPRAVDA 1 80 98 0 NEPRAVDA 2 80 98 0 NEPRAVDA Table 2. Chest strap output. Again, this is a text file that contains the instanta- neous heart rate data, labeled with a serial number. As this is an oximeter, the data also includes a record of blood oxygen saturation. The first problem that had to be solved was the processing of the ECG data. From the ECG recording, the instantaneous heart rate can be calculated using Equation 1. HR = 60 R − R (1) Where the R−R term represents the time difference of two consecutive R peaks. Since the frame rate of the ECG probe is 1 kHz, one recording represents 1ms. To calculate the time difference between the peaks, it was necessary to localize the peaks in the signal. For this, the findpeaks function included in Matlab was used. A plot of the raw data is in Figure 3 (x-axis is time, y-axis is ECG signal strength). After application of findpeaks some parts of the ECG curve were wrongly labelled as peaks. This shortcoming was corrected by applying the minpeakprominence and minpeakdistance function parameters. The first of these arguments modifies the minimum difference that must exist between points for a peak to be identified, and the second argument modifies the minimum distance at which peaks are identified. The value of the first argument was ex- perimentally set to 15000. The value of the second argument was easier to find because it had to roughly correspond to the time between heartbeats. It was set to 400. The result is the identification of peaks on the ECG curve, without false positives and without missing them. The details of the curve are shown in the Figure 2 through Figure 4. After finding a procedure to correctly identify the peaks, it was possible to calculate the instantaneous heart rate from the ECG using the aforementioned equation. The result is in the Figure 4. For compari- son, data had to be resampled to the lowest common sampling frequency. Comparison of all three devices is in Figure 5. Based on this data, a method could be selected to compare the different devices. Two methods were considered. The first method consisted of summing 22 vol. 41/2023 Driver Physiological Functions in a Vehicle Simulator Figure 2. Raw ECG. Figure 3. Identification of ECG peaks, MPP – 15 000, MPD – 400, detail 1. Figure 4. Instantaneous heart rate calculated from ECG. 23 David Lehet Acta Polytechnica CTU Proceedings Figure 5. Results of the pilot measurement. Absolute distance Chest strap Pulse oximeter ID1 22111.0 11763.0 ID2 2177.1 1450.6 ID3 3484.6 1345.7 ID4 2542.2 3947.1 Table 3. Absolute distances of the assessed devices from the ECG. the absolute distances of the measured heart rate of the device under consideration from the reference (i.e., the ECG), according to Equation 2. HRdistance = numberofrecords∑ i=1 ∣∣HRreference(i) − HRdevice(i) ∣∣ (2) In Matlab this function was implemented as: ID(:,4)=abs(ID(:,1)-ID(:,2)); Distance = sum(ID (:,4)); Column 4 contained the calculated data, columns 1 and 2 contained the ECG record and the device under assessment. The result was stored in the variable Distance. The results of this method for the 4 pilot measure- ments are shown in the Table 3. This method takes into account the entire measure- ment process, the closer the sensor is to the standard, the lower the values become. The disadvantage is the poor comparability between different measurements, mainly because the value will be higher the longer the measurement itself. The results are also not very intuitive to look at. The advantage is then taking into account the magnitude of the error with which the sensor is measuring. The second proposed method involves calculating what percentage of the measured values are from a certain distance from the standard. In Matlab, the calculation was implemented as follows: for i=1:length(ID) if ID(i,2)>((ID(i,1)*0.9)) && ID(i,2)<((ID(i,1)*1.1)) ID(i,8)=1; else ID(i,8)=0; end end KPI_10_Pas=(sum(ID (:,8))/length(ID))*100 The values 0.9 and 1.1 represent the distance from the standard, in this case 10%, at which the sensor must be located. The method was tested for two different distances, 10% and 15%. The value of 10% was chosen based on the “American National Standard of Cardiac monitors, heart rate meters, and alarms” and the value of 15% was chosen as an additional value [22]. If the checked value of the instantaneous heart rate was within the permissible distance, it was marked as 1, otherwise as 0. After checking the entire record, the data vector was summed and the proportion of records that were sufficiently accurate was calculated. The proportion is expressed as a percentage. The results for all four measurements are shown in the Table 4. 24 vol. 41/2023 Driver Physiological Functions in a Vehicle Simulator Chest strap Pulse oximeter 10% 15% Difference 10% 15% Difference ID1 39.06 52.03 12.97 24.77 46.53 21.76 ID2 82.02 90.35 8.33 78.51 94.74 16.23 ID3 60.70 71.14 10.44 69.65 87.56 17.91 ID4 63.61 84.59 20.98 41.64 63.93 22.29 Table 4. Comparison of two parameters for evaluating sensor accuracy. First measurement Second measurement ID Strap [%] Oxi [%] Strap [%] Oxi [%] 1 57.77 73.65 67.3 63.87 2 66.5 81.55 0.39 59.38 3 60.8 57.6 70.64 62.84 4 54.51 63.53 53.31 60.66 5 42.72 16.02 89.04 0.68 6 75 75.55 18.25 65.87 7 69.77 79.07 87.18 52.56 8 75.97 82.47 16.39 63.93 9 50.54 42.93 0.76 50 10 69.68 73.4 8.92 78.57 Mean values 62.326 64.577 41.218 55.836 Standard deviation 10.40 19.94 34.03 19.80 Table 5. Measurement results. It is clear from the table that the assessed de- vices are accurate enough to be assessed for dis- tances from the reference signal (0.9−EKG_reference, 1.1−EKG_reference) (the intervals are open). Comparing the two methods, it is clear that al- though the second method does not reflect the mag- nitude of the error (it does not distinguish whether the sensor is off by 1 or 50 beats), it is significantly easier to interpret. Moreover, the two methods are consistent in their results. Therefore, the method based on the distance from the reference signal was chosen for further processing. 3. Results For the final comparison, a total of 20 runs were measured using 10 probands, with each proband com- pleting 2 runs. Both genders and all somatotypes were represented in the measurements. Measurements were processed using the procedures described above. The results are presented in the Table 5. The interpretation of the results is complicated by the fact that the chest belt, although more accurate (the highest accuracy of all measurements was achieved by the belt – 89.04%), suffered from signal dropouts, which are marked in red in the table. The most accurate result of the pulse oximeter was an accuracy of 82.47%. Interestingly, when comparing measurements in which the chest belt did not expe- rience signal dropouts, the average values for both sensors are similar. For illustration, the best results of both sensors are presented in Figures 6 and 7 and their strengths and weaknesses are described on the measurements. Figure 6 shows the aforementioned best measure- ment using a chest strap. However, it can also be seen here that the belt is slower than the reference ECG in responding to changes in heart rate, and some sharp fluctuations are not detected at all. In this measure- ment, the pulse oximeter readings were very poor, but this was not due to failure. Figure 7 shows the best pulse oximeter measure- ment. It can be seen that although the pulse dynamics are less well tracked by the oximeter than by the chest belt, the tracking of the overall trend is quite good. 4. Conclusion and future work In this article, two commonly available sensors – a chest belt and a pulse oximeter in the form of a watch – have been selected for a comparison of usage of common commercial sensors for monitoring driver during a test in a vehicle simulator. Using pilot mea- surements, a suitable methodology was designed and selected to evaluate their accuracy against a reference signal in the form of a two-lead ECG. From the measurements it is clear that while both devices are capable of accurate enough measurement 25 David Lehet Acta Polytechnica CTU Proceedings Figure 6. Measurement, ID5_2. Figure 7. Measurement ID 8_1. to be used during experiments, the chest strap is capable of the more accurate measurement. This is to be expected as the technology used by its sensors is more accurate than the photoplethysmography on which the pulse oximeter is based. Slower reactions of the oximeter show that it is more suited for long term measurement, while better dynamics of the chest strap make it useful for measuring driver during stressful events. However, the ergonomics of the measurement com- plicate the choice of a suitable device. For the partici- pant in the experiment, it is obviously better if he or she just puts on a smartwatch instead of a chest strap. However, both options are an improvement against ECG electrodes. Further research in this direction should focus on collecting a larger sample of data using the described ones, on which the results of the measurement could be statistically verified. Another weakness of the experiment was imperfect way of synchronizing the measuring devices. Although it was sufficient for a proof of concept such as this, subsequent experiment should use more precise method. Another part of the experiment that should be perfected in subsequent iterations is the problem of sampling. Each device had a different sampling rate and even though it was again deemed sufficient for this experiment, devices with higher and more similar sampling rate should be chosen next time. It must also be kept in mind that the commercial electron field is rapidly evolving, testing should be repeated periodically using newer equipment but the same (but updated) methodology. 26 vol. 41/2023 Driver Physiological Functions in a Vehicle Simulator References [1] E. K. S. Louis, L. C. Frey. Electroencephalography – An introductory text and atlas of normal and abnormal findings in adults, children, and infants. American Epilepsy Society, Chicago, IL, USA, 2016. ISBN 13-978-0-9979756-0-4. [2] I. Hrazdira, V. Mornstein. Lékařská biofyzika a přístrojová technika. Neptun, 2001. [3] L. Yan, Y. Chen, J. Zhang, et al. Distraction detection of driver based on EEG signals in a simulated driving with alternative secondary task. In 2021 IEEE 2nd International Conference on Human-Machine Systems (ICHMS), pp. 1–3. IEEE, 2021. https://doi.org/10.1109/ichms53169.2021.9582456 [4] C.-T. Lin, H.-Z. Lin, T.-W. Chiu, et al. Distraction- related EEG dynamics in virtual reality driving simulation. In 2008 IEEE International Symposium on Circuits and Systems, pp. 1088–1091. IEEE, 2008. https://doi.org/10.1109/iscas.2008.4541611 [5] H. Zhang, R. Chavarriaga, Z. Khaliliardali, et al. EEG-based decoding of error-related brain activity in a real-world driving task. Journal of Neural Engineering 12(6):066028, 2015. https://doi.org/10.1088/1741-2560/12/6/066028 [6] S. P. Kumar, J. Selvaraj, R. Krishnakumar, A. Sahayadhas. Detecting distraction in drivers using electroencephalogram (EEG) signals. In 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC), pp. 635–639. IEEE, 2020. https: //doi.org/10.1109/iccmc48092.2020.iccmc-000118 [7] G. Li, W. Yan, S. Li, et al. A temporal–spatial deep learning approach for driver distraction detection based on EEG signals. IEEE Transactions on Automation Science and Engineering 19(4):2665–2677, 2022. https://doi.org/10.1109/tase.2021.3088897 [8] H. V. Koay, J. H. Chuah, C.-O. Chow, Y.-L. Chang. Detecting and recognizing driver distraction through various data modality using machine learning: A review, recent advances, simplified framework and open challenges (2014–2021). Engineering Applications of Artificial Intelligence 115:105309, 2022. https://doi.org/10.1016/j.engappai.2022.105309 [9] A. Joshi, A. Kamble, A. Parate, et al. Drowsiness detection using EEG signals and machine learning algorithms. ITM Web of Conferences 44:03030, 2022. https://doi.org/10.1051/itmconf/20224403030 [10] X. Li, L. Yang, X. Yan. An exploratory study of drivers’ EEG response during emergent collision avoidance. Journal of Safety Research 82:241–250, 2022. https://doi.org/10.1016/j.jsr.2022.05.015 [11] E. A. Ashley, J. Niebauer. Conquering the ECG. In Cardiology Explained, chap. 3. Remedica, London, 2004. https://www.ncbi.nlm.nih.gov/books/NBK2214/ [12] S. Trojan. Lékařská fyziologie. Grada Publishing, 2003. [13] N. Isakadze, S. S. Martin. How useful is the smartwatch ECG? Trends in cardiovascular medicine 30(7):442–448, 2020. https://doi.org/10.1016/j.tcm.2019.10.010 [14] N. Keshan, P. V. Parimi, I. Bichindaritz. Machine learning for stress detection from ECG signals in automobile drivers. In 2015 IEEE International Conference on Big Data (Big Data), pp. 2661–2669. IEEE, 2015. https://doi.org/10.1109/bigdata.2015.7364066 [15] S. Pourmohammadi, A. Maleki. Stress detection using ECG and EMG signals: A comprehensive study. Computer Methods and Programs in Biomedicine 193:105482, 2020. https://doi.org/10.1016/j.cmpb.2020.105482 [16] M. S. B. B. Sikder, S. M. Salimullah, I. Ahmed, et al. Galvanic skin response based cognitive load measurement for online exams. In 2021 5th International Conference on Electrical Information and Communication Technology (EICT), pp. 1–4. IEEE, 2021. https://doi.org/10.1109/eict54103.2021.9733564 [17] J. Tervonen, K. Pettersson, J. Mäntyjärvii. Ultra-short window length and feature importance analysis for cognitive load detection from wearable sensors. Electronics 10(5):613, 2021. https://doi.org/10.3390/electronics10050613 [18] V. Borisov, E. Kasneci, G. Kasneci. Robust cognitive load detection from wrist-band sensors. Computers in Human Behavior Reports 4:100116, 2021. https://doi.org/10.1016/j.chbr.2021.100116 [19] P. Manikandan, M. R. S. Reddy, S. Mehatab, P. M. Sai. Automobile drivers distraction avoiding system using galvanic skin responses. In 2021 6th International Conference on Communication and Electronics Systems (ICCES), pp. 1818–1821. IEEE, 2021. https://doi.org/10.1109/icces51350.2021.9489076 [20] S. Voges, K.-F. Becker, D. Schütze, et al. Highly miniaturized integrated sensor nodes for industry 4.0. International Symposium on Microelectronics 2019(1):000415–000422, 2019. https://doi.org/10.4071/2380-4505-2019.1.000415 [21] S. Mach, P. Storozynski, J. Halama, J. F. Krems. Assessing mental workload with wearable devices – reliability and applicability of heart rate and motion measurements. Applied Ergonomics 105:103855, 2022. https://doi.org/10.1016/j.apergo.2022.103855 [22] F. Sartor, J. Gelissen, R. van Dinther, et al. Wrist- worn optical and chest strap heart rate comparison in a heterogeneous sample of healthy individuals and in coronary artery disease patients. BMC Sports Science, Medicine and Rehabilitation 10(1):10, 2018. https://doi.org/10.1186/s13102-018-0098-0 27 https://doi.org/10.1109/ichms53169.2021.9582456 https://doi.org/10.1109/iscas.2008.4541611 https://doi.org/10.1088/1741-2560/12/6/066028 https://doi.org/10.1109/iccmc48092.2020.iccmc-000118 https://doi.org/10.1109/iccmc48092.2020.iccmc-000118 https://doi.org/10.1109/tase.2021.3088897 https://doi.org/10.1016/j.engappai.2022.105309 https://doi.org/10.1051/itmconf/20224403030 https://doi.org/10.1016/j.jsr.2022.05.015 https://www.ncbi.nlm.nih.gov/books/NBK2214/ https://doi.org/10.1016/j.tcm.2019.10.010 https://doi.org/10.1109/bigdata.2015.7364066 https://doi.org/10.1016/j.cmpb.2020.105482 https://doi.org/10.1109/eict54103.2021.9733564 https://doi.org/10.3390/electronics10050613 https://doi.org/10.1016/j.chbr.2021.100116 https://doi.org/10.1109/icces51350.2021.9489076 https://doi.org/10.4071/2380-4505-2019.1.000415 https://doi.org/10.1016/j.apergo.2022.103855 https://doi.org/10.1186/s13102-018-0098-0 Acta Polytechnica CTU Proceedings 41:20–27, 2023 1 Introduction 1.1 Physiological functions and their measurement 1.2 EEG 1.3 ECG 1.4 GSR 2 Material and methods 2.1 Used sensors 2.2 Pilot measurement 3 Results 4 Conclusion and future work References