Acta Polytechnica CTU Proceedings doi:10.14311/APP.2017.12.0068 Acta Polytechnica CTU Proceedings 12:68–73, 2017 © Czech Technical University in Prague, 2017 available online at http://ojs.cvut.cz/ojs/index.php/app CROSS-STUDY RESEARCH ON UTILITY AND VALIDITY OF DRIVING SIMULATOR FOR DRIVER BEHAVIOR ANALYSIS Michal Matowicki∗, Ondřej Přibyl Czech Technical University, Faculty of Transportation Sciences, Na Florenci 25, Prague 1, Czech Republic ∗ corresponding author: matowicki.m@gmail.com Abstract. Driving is one of the most ordinary and universal everyday tasks and, at the same time, one of the most complex and dangerous. It requires a full range of sensory, perceptual, cognitive, and motor functions, all of which can be affected by a wide range of stressors and experience levels. Therefore, exploring of human behaviour while controlling a vehicle is a crucial task in improving traffic safety. Experimental studies can always be conducted with on-road tests, however, using a simulator is safer and more cost-effective. The main goal of this paper is to demonstrate if and under what conditions could a driving simulator provide sufficient results required for a proper study of driver behavior. It discusses its limits and advantages. Overall, the research reviewed in this paper indicates that simulator driving behaviour approximates (relative validity of speed and lateral position of vehicle on road), but does not exactly replicate (absolute validity), on-road driving behaviour. Keywords: driving simulator, relative validity, absolute validity, cross-study analysis. 1. Introduction With rapid growth of technological advancement, not only executive elements or traffic engineering like in- telligent transport systems, is shifting toward more sophisticated and ecologically and economically valid solutions. Another example is research on the trans- port subjects which is reaching for advanced tools like driving simulators. However, although supported in many government and scientific organizations [1] the usefulness and validity of results achieved in a driving simulator remain a very hot topic in transport engineering - especially when driving behaviour and the psychology of the drivers are concerned. Although this topic seems to be of significant importance (either supports of diminishing effects and results of driving simulator experiment), it seems to be often forgotten during the execution of the experiment, which may, at times, lead to meaningless conclusions. Hence, it is very important to produce a literature review on the topic in search of significant data to utilize simu- lator results to make conclusions on real-world driver behaviour. This document includes a table (Table 1) which summarizes the most important papers on the topic, as well as further analysis and description of chosen papers, all of which are divided into several categories depending on the kind of validity confirmed and pa- rameters measured in the experiment. The table al- lows for quick familiarization with the most important research reviewed and introduces an in depth analysis of specific studies. Chapter two consists of subsec- tions that are dedicated to particular kinds of validity which can and should be of concern when setting up a driving simulator experiment. This paper produced a cross-study review and analysis of confirmed and proven validity measures for driving simulator exper- iment measures, and is concluded by summarizing recommendations and predictions regarding the fu- ture of driving simulators as tools for various studies on driving behaviour. 2. Reviewed studies Despite the importance that is placed on the simu- lator as a tool for investigating driver performance and behaviour, an extensive literature review reveals that only a very limited number of studies specifically evaluating the behavioural validity of simulators was conducted [3]. Majority of the available literature on simulator validation has focused on measures of speed, lateral position, and braking responses (Ta- ble 1). The following summary of literature research reviews literature on the given topic with a specific emphasis on the description of performance measures, the tasks and conditions, and first of all the evidence provided for behavioural validity of each simulator and the behavior of the subject "driving" it. 2.1. Speed adjustment Speed is one of the most studied measures of driver behaviour and, thus, one of most commonly analised with respect to simulator validity [2, 5–7, 9–11, 13, 15, 17]. Overall, these studies rather consistently provide results confirming relative but not absolute validity of speed data in driving simulators. Most explanatory ones were for example Klee et al. [10] who examined the validity of a fixed-base simulator with respect to forward speed. 30 drivers drove the same road section on both driving simulator and in an instrumented vehicle. The speed values were measured at 16 points, while measures in 10 of them were similar in both ob- served environments. While authors do not conclude relative validity of the simulator and focus on lack of 68 http://dx.doi.org/10.14311/APP.2017.12.0068 http://ojs.cvut.cz/ojs/index.php/app vol. 12/2017 Cross-Study Research of Driving Simulator Authors achieved validity Bella, 2008 [2] speed and complexity of menoeuvre Mullen, Charlton, Devlin & Bedard,2011 [3] relative validity of lateral position across different road types Watts & Quimby, 1979 [4] relativity of perceived hazard risk Blaauw, 1982 [5] absolute speed validity & relative lateral displacement Riemersma et al., 1990 [6] relative validity of speed reduc- tion measures Tornros, 1998 [7] relative validity of speed and lat- eral positioning Wade & Hammond, 1998 [8] relative validity of lane position Reed & Green, 1999 [9] absolute validity on speed control relative validity on effects of phone dialling Klee et al., 1999 [10] relative validity of vehicle velocity Bittner et al., 2002 [11] relative validity of speed in road curves Blana & Golias, 2002 [12] No relative validity confirmed Godley et al., 2002 [13] relative validity of speed counter- measures Lee, Lee & Cameron, 2003 [14] relative validity of visual attention-age relation Bella, 2005 [15] relative validity of speed McAvoy 2007 [16] No validity for speed adjustment in work zones Shinar & Ronnen 2007 [17] relative validity of speed adjust- ing Table 1. Studies elaborating driving simulator validity. absolute validity, it can be observed that speeds in all 10 locations with similar results were 5-10 km/h slower for simulated driving. Moreover, described speed dif- ference provided the same results in each case which implies relative validity of simulator. Another author providing evidence for the relative validity of the sim- ulator maintained speed and its correlation with the on-road speed is Bella [2, 15]. Bella provides two studies in which the simulators were designed with identical road sections to those where real driving was performed by experiment participants. In real world experiments, speed measurement points were specified along the road instead of the instrumented vehicle driving record. The first experiment was con- ducted on road section with construction zone with speed limitation applied, whereas the second study described a highway in normal condition. A bilat- eral Z-test for non-matched samples was performed in both data sets to estimate whether the recorded driving speeds were significantly different from those in the simulator. Consistently with other researches Bella conducted that simulator speeds were different than in the real world, although in [2] mean speed in simulator was higher than in real world while in [15] it was on the contrary. Nevertheless, the differences in simulator speeds and in the field speeds for each point were not significantly different, demonstrating at minimum, interactive relative validity of static sim- ulator tool for assessing speed. Unconventional way of assessing speed as simulator validity indicator was presented by Shinar & Ronen [17] where two speed related measures were observed: speed estimation (i.e. driver perception of the current speed of vehicle) and production of speed (i.e. adjustment of the vehicle’s speed to reach predetermined speed). Sixteen partici- pants took part in an experiment where both on-road driving and simulator (with shielded speedometer) drives were performed. During each drive, partici- pants were repeatedly asked to estimate the current speed of vehicle and produce seven different speeds 69 Michal Matowicki, Ondřej Přibyl Acta Polytechnica CTU Proceedings ranging from 40 to 100 km/h. Authors found that estimated speeds differed significantly in the simulator only for speeds of 40 and 100 km/h where the first speed was always estimated higher in simulator and latter one was always estimated lower. On the other hand, second investigated variable showed consistently higher produced speeds in simulator with a difference of approximately 25 km/h. Conclusions of multiple analysed studies pointed out towards absolute validity of most of the estimated speeds and vast majority of the studies suggests relative validity for speed pro- duced in the simulator. This seems to form firm foundations for performing scientific research with the use of driving simulators. 2.2. Lateral position of vehicle Second often validated measure in driving simulators is the lateral position of the vehicle on the simulated road. One of the most interesting studies in the matter of lateral positioning validity was performed by Wade and Hammond [8]. The authors measured positioning deviation as distance from the centreline of the road. In a tested sample of 26 participants, the mean devia- tion of the distance from centreline was statistically significantly larger in virtual driving when compared to the on-road conditions. Authors determined that in the real-world it is clear that drivers travel more ac- curately near the centreline when compared to virtual driving experience. This led to the conclusion that absolute validity was not demonstrated in performed research. Nevertheless, Mullen, Charlton, Devlin and Bedard [3] noticed that similar to results in speed researches mean deviations on different types of roads showed a familiar pattern when compared to the real environment (i.e. the road with the highest mean deviation in the real-world was also the road with the highest mean deviation in the virtual environment) which indicates relative validity. However, it is a no- ticeable fact that data were collected on straight road section with a relatively low speed limit of 35 mph (ap- prox. 56 km/h). Although Blana & Golias [12] tried to answer the question of speed influencing validity of lateral displacement measures, it remains unknown. In their research scientists conducted experiment in a simulator where distance between front wheel of the vehicle on the passenger side and white line de- termining the edge of the road were measured. A sample of 100 on-road drivers in vehicle instrumented with video cameras pointing toward road surface drove on elaborated highway section, while corresponding road was also driven in simulator by 100 participants. The results clearly pointed towards higher average distance measured in the real-world conditions when compared to simulator on both straight sections and curves. This indicates that drivers in the on-road conditions position their vehicles closer to the cen- treline of the road lane. This conclusion was further confirmed by smaller standard deviations (SD) for the measures recorded in real-world conditions. Although the direction of displacement from the road edge was consistent for curved and straight road sections for all speeds measured, the magnitude of the difference in displacement and SD varied depending on speed, leading to inconclusive results in terms of absolute or relative validity of the simulator. Again, these results seem to provide strong argument supporting driving simulators as recommended research tools. 2.3. Road safety countermeasures The simulator validity for evaluating the relation of a driver with road design and traffic control devices was another issue often examined. Riemersma et al. [6] performed two experiments aiming to understand, explain and describe these effects. First, the Daimler- Benz driving simulator in which the authors created a scenario where the driver entered a Dutch village area. The goal of the simulator was to register changes in desired speed adjustments. The drivers were divided into 2 groups and drove the car either on a normal road or on the same road with special speed reduction ob- jects. Three infrastructure changes were implemented in the second scenario, those being a median strip, coloured asphalt and a portal gate. Analysis of the ve- hicle speeds in both scenarios revealed decreased speed values in the scenario with special objects installed. Moreover, faster drivers reduced their speeds more than average speed drivers. The second experiment involved both real-world driving and in-simulator ride, utilizing their university driving simulator to perform speed reducing infrastructure testing. The experiment was conducted on 24 participants, of which each drove through the village entrance 12 times. Infrastructure changes varied during the rides. Driving speed com- parison between real-world and simulator revealed higher average speed of in-simulator driving than on road conditions at distance of approx. 400 meters from village entrance. Nevertheless, an interesting finding was that, after the 400m distance point, simulator drivers reduced their speed to a greater extend, thus entering the village at a lower speed. The magnitude of speed reduction was evaluated as 3 times bigger for simulator than real-world (25,7% and 8,6% respec- tively) this can indicate similarly to chapter 2.5.2, relative but not absolute validity of the simulator re- sults. Due to these results, the authors concluded that a simulator was an effective tool for evaluating speed reduction methods. Following research of McAvoy and Datta [16], found in their study that AMOS II simulator has contrary results for evaluating night traffic speed control de- vices in construction zones. A model construction zone on a highway with warning lights was designed and implemented into simulator conditions. 35 drivers took a ride in simulator at the road described above, while in naturalistic study vehicles average speed was measured on road with the use of radar guns. The summary of the results led the researchers to conclude that the simulator failed to provide proof for relative 70 vol. 12/2017 Cross-Study Research of Driving Simulator or absolute validity standards. A suggesteed explana- tion for that fact was the subject’s perception of risk associated with driving through real-world work zones at night in a simulator-provided experience. The final paper reviewed in relation to drivers’ behaviour stud- ied using a simulator was the work of Godley et al. [13]. In this study again, similarly to Riemersma work [6], relative validity of simulator results was found for effect of rumble strips as a speed-reduction method. 2.4. Complex driving behaviours Another examined validity in short history of driving simulator experiments is in relation to the complex be- haviours of drivers such as tasks of divided attention. Reed & Green [9] investigated validity of the driving simulator constructed at the University of Michigan Transportation Research Institute. The goal of the study was to assess decrements in driving performance during a manual car phone task. Participants of the experiment were divided into 2 groups: young (20- 30 years) and old (above 60 years old), and received instructions to dial a manual phone call during ve- hicle driving tasks both in the on-road setting and in a simulator. Conclusions of the work exhibited that speed parameters (SD of speed and throttle posi- tion) were similar in both studied conditions, whereas lane positioning (i.e. mean lateral speed and SD of steering wheel position) varied significantly more in the simulator. Overall, for seven of the ten variables measured in the experiments, a significant correlation was found between two environments (authors provide correlation factor r in range of 0.43 to 0.76). For all experiment participants regardless of the age group, the phone task resulted in a decrease in driving abili- ties in terms of speed control and lateral position of vehicle. Worth mentioning is the fact that, just like in the previous studies, the magnitude of impact of particular driving conditions was higher in a simu- lator than on the road. Additionally, older drivers seemed to suffer more from the phone dialing task than young drivers in terms of driving abilities. Few other studies in this area were performed, but ones worth mentioning are Lee, Lee and Cameron [14] (distinguishing between drivers with different abili- ties), Lee & Lee [18] (identifying drivers at risk of future violations), Reimer et al. [19] and Salvucci et al. [20] who investigated an iPhone, button style flip-phone and iPod distraction on driver performance. Results of these works however, are based on very few participants therefore, although similar to the ones described above, they will not be summoned in detail here. 2.5. Physiological measures The final validity of importance investigated in this research are psychological responses. While the pre- vious sections described a direct way of measuring driving behaviour, physiological responses are com- monly used to determine driver awareness and thus produce some useful insight into driver behaviour and reactions across both simulator and real-world con- texts. An important condition that is required from driving simulators all over the world is that simula- tions should induce and provoke similar physiological responses as real-world experience. First research conducted on that topic was Watts and Quimby [4], who used skin conductance measures for subjects in a Transport and Road Research laboratory simula- tor. Due to the technological limitations of the time, driving simulation was performed in form of film pre- sented to the drivers that was placed to create the impression of being the driver of a vehicle while film from a drive on a road was displayed. The presented film was 30 minutes long, and consisted of 45 haz- ardous situations for which drivers had to evaluate the perceived danger on a 10-point scale. Conduc- tance of skin was measured for all participant during the film. The measurements were then repeated for a real-world driving condition and the results were compared. In the presented results, the risk assess- ments were comparable for both environments, and moreover, the perceived risk ratings were positively associated with skin conductance levels (r = 0.78). A more recent study performed by Slick et al. [21] utilized measures of both skin conductance and heart rate in both simulator and real-world conditions. The experiment performed in DriveSafety DS- 600c motion- based simulator was made of four specific driving tasks performed on both stop sign and intersection traffic lights. The study suggests that there are no signifi- cant differences between skin conductance and heart rate measures between the simulator and realworld driving conditions. The only exception for female drivers was that their mean change in heart rate was significantly smaller in the simulated condition than while driving on-road. The most recent study is one analyzed by Mullen, Charlton, Devlin and Bedard [3]. Bedard in his research measured heart rate and respiration for participants exposed to threatening situations while driving STISIM 400 simulator. Two intersection scenarios were investigated in depth with the green light turning into orange immediately after driver approached the intersection, and vehicle ap- pearing from the right shoulder of the road in front of the driver. Results indicated high measure of pres- ence of the simulator. In each observed situation, the driver’s heart rate increased significantly, and 65% of the participants showed increased respiration rates in risky situations. Overall, these studies suggest that physiological measures in general demonstrate the validity of simulators. Overall, these studies suggest that physiological measures in general demonstrate the validity of simu- lators. 3. Summary and Conclusions Many studies have investigated the validity of simu- lators for examining driving behaviour, leading to a 71 Michal Matowicki, Ondřej Přibyl Acta Polytechnica CTU Proceedings conclusion that there is sufficient evidence to suggest that simulators provide a valid tool for assessing vari- ous driving performance measures. Some important findings on the topic of simulator implementation can be found in studies like [22–24], where overwhelm- ing majority of the studies suggest that there are no obstacles in using driving simulators. These stud- ies suggest that simulators provide approximated be- haviour and reactionsof the drivers (relative validity) , despite the fact that it does not replicate on-road behaviour (absolute validity) exactly. The studies under review confirm a moderate to high correlation of the simulator-achieved results for a whole range of cognitive measures of drivers as well as physiological reactions. This implies that a driving simulator is a sufficient tool for majority of research purposes, and thus may be accepted as suitable for the collection of desired data. Nevertheless, it is of high importance to remember that the fact that a driving simulator was shown to be valid in a previous setting does not guarantee it will be suitable for the next one. There- fore, it is highly recommended to perform additional simulator validation of the results for each specific set- ting and the scenario designed for a given simulation experiment. This ensures the researchers are capable of utilising this data to create solutions applicable to real-world situations. On road measures will surely be necessary and prevail over the driving simulator method in research situations in which absolute va- lidity of measured variables is required. Although there are certain drawbacks and limitations to current simulator deployment in transport research, there is a confident promise of overcoming this issue through technological advancement and, thus, broadening of simulator validity by creating better imitation and perceptual projection of on-road condition for simula- tor drivers. This will undoubtedly increase the role of simulators as a tool for driving performance mea- surements as well as for studying driving behaviour. Such studies are certainly important not only with re- spect to the traditional transportation research tasks (transportation planning, study of traffic flow charac- teristics, optimization of traffic control and others), but also for new and emerging tasks such as within the field of Smart Cities (resp. Smart Mobility), study of drivers using cooperative vehicles by Přibyl and Svítek [25]. The authors of this paper place high importance on understanding which real-life conditions a driving simulator can replicate and aimed to provide guidance in this respect through this document. Acknowledgements This article was supported by funds from grant no. SGS16/186/OHK2/2T/16. References [1] E. Commission. WHITE PAPER Roadmap to a Single European Transport Area – Towards a competitive and resource efficient transport system. Tech. rep., European Union Commission, 2011. Date-Added: 2011-03-08. [2] F. Bella. Driving simulator for speed research on two-lane rural road. ELSEVIER Accident Analysis & Prevention 40:1078–1087, 2008. [3] N. Mullen, A. Charlton, J.and Devlin, M. Bedard. Simulation Vlidity: Behaviors Observed on the Simulator and on the Road. Handbook of Driving Simulation for Engineering, Medicine, and Psychology. CRC Press, 2011. [4] G. J. Watts, A. R. Quimby. Design and validation of driving simulator for use in perceptual studies. Tech. rep., Transport and Road Research Laboratory, 1979. [5] G. J. Blaauw. Driving experience and task demands in simulator and instrumented car: A validation study. Human Factors 24:473–486, 1982. [6] A. R. R. Riemersma, J. B. J.and van der Horst, W. Hoekstra, G. M. M. Alink, N. Otten. The validity of a driving simulator in evaluating speed-reducing measures. Traffic Engineering and Control 31:416–420, 1990. [7] J. Tornros. Driving behaviour in real world in a real and simulated road tunnel – A validation study. Accidents Analysis & Prevention 30:497–503, 1998. [8] M. G. Wade, C. Hammonds. Simulator validation: Evaluating driver performance in simulation and the real world. Tech. rep., Minnesota Department of Transportation, 1998. [9] M. P. Reed, P. Green. Comparison of driving performance on-road and in low-cost simulator using a concurrent telephone dialling task. Ergonomics 42:1015–1037, 1999. [10] H. Klee, C. Bauer, E. Radwan, H. Al-Deek. Preliminary validation of driving simulator based on forward speed. Transportation Research Record 1689:33–39, 1999. [11] A. C. Bittner, O. Simsek, W. H. Levison, J. L. Campbell. On-road versus simulator data in driver model development: Driver performance model experience. Transportation Research Record 44:303–313, 2002. [12] E. Blana, J. Golias. Differences between vehicle lateral displacement on the road and in a fixed-base simulator. Human Factors 44:303–313, 2002. [13] S. T. Godley, T. J. Triggs, B. N. Fildes. Driving simulator validation for speed research. Accident Analysis & Prevention 34:589–600, 2002. [14] H. C. Lee, A. H. Lee, D. Cameron. Validation of driving simulator by measuring the visual attention skill of older adult drivers. American Journal of Occupational Therapy 57:324–328, 2003. [15] F. Bella. Validation of driving simulator for work zone design. Transportation Research Record 1937:136–144, 2005. [16] D. S. McAvoy, K. L. Schattler, T. K. Datta. Driving simulator validation for nighttime construction work zone devices. Transposrtation Research Record 2015:55–63, 2007. [17] D. Shinar, A. Ronen. Validation of speed perception and production in a single screen simulator. Advances in Transportation Studies Special Issue:51–56, 2007. 72 vol. 12/2017 Cross-Study Research of Driving Simulator [18] H. C. Lee, A. H. Lee. Identifying older drivers at risk of traffic violations by using a driving simulator: A 3-years longitudinal study. American Journal of Occupational Therapy 57:97–100, 2005. [19] B. Reimer, B. Mehler, B. Donmez, et al. A driving simulator study examining phone dialling with iphone vs. button style flip-phone. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 56(1):2191–2195, 2012. [20] D. D. Salvucci, D. Markley, M. Zuber, D. P. Brumby. ipod distraction: effects of portable music-player use on driver performance. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems pp. 243–250, 2007. [21] R. Slick, D. F. Evans, E. Kim, J. Steele. Using simulators to train novice teen drivers: Assessing psychological fidelity as a precursor to transfer of training. Paper presented at Driving Simulation Conference – Tsukuba 2006. [22] P. Bouchner, S. Novotny, R. Pieknik, O. Sykora. Assessments of grade crossing warning and signalization devices - driving simulator study. pp. 179–183. 2011. Cited By 0. [23] P. Bouchner, S. Novotny. Development of advanced driving simulator: Steering wheel and brake pedal feedback. pp. 170–174. 2011. Cited By 0. [24] A. Mashko, P. Bouchner, D. Rozhdestvenskiy, S. Novotny. Virtual traffic signs – assessment of an alternative adas user interface with use of driving simulator. Advances in Transportation Studies 1:37–50, 2016. Cited By 0. [25] O. Pribyl, M. Svitek. System-oriented approach to smart cities. Proceedings of the First IEEE International Smart Cities Conference ISBN 978-1-4673-6551-2 2015. 73 Acta Polytechnica CTU Proceedings 12:68–73, 2017 1 Introduction 2 Reviewed studies 2.1 Speed adjustment 2.2 Lateral position of vehicle 2.3 Road safety countermeasures 2.4 Complex driving behaviours 2.5 Physiological measures 3 Summary and Conclusions Acknowledgements References