DOI: 10.3303/CET24111038 Paper Received: 25 February 2024; Revised: 9 April 2024; Accepted: 5 August 2024 Please cite this article as: Amazu C.W., Demichela M., Fissore D., Leva M.C., 2024, Comparing Two Control Room Intervention Procedure Formats: Preliminary Insights from Eye Tracking Measures, Chemical Engineering Transactions, 111, 223-228 DOI:10.3303/CET24111038 CHEMICAL ENGINEERING TRANSACTIONS VOL. 111, 2024 A publication of The Italian Association of Chemical Engineering Online at www.cetjournal.it Guest Editors: Valerio Cozzani, Bruno Fabiano, Genserik Reniers Copyright © 2024, AIDIC Servizi S.r.l. ISBN 979-12-81206-11-3; ISSN 2283-9216 Comparing Two Control Room Intervention Procedure Formats: Preliminary Insights from Eye Tracking Measures Chidera W. Amazu a,*, Micaela Demichela a, Davide Fissore a, Maria Chiara Leva b aSAfeR - Department of Applied Sciences and Technology, Politecnico di Torino, Italy. bFaculty of Sciences and Health, Technological University Dublin, Ireland. chidera.amazu@polito.it Control rooms rely on operating and emergency procedures to facilitate human-machine interaction and uphold safety in critical systems. These procedures can significantly affect operators' cognitive states, including situational awareness and performance. Therefore, it is imperative to thoroughly examine the impact of diverse procedure designs on operators' cognitive states and behaviour during process intervention. This study scrutinised two intervention procedure representation formats—paper-based and digitised screen-based procedures—originally designed for a prior experimental study. Employing the eye-tracking technique from cognitive science studies during data collection provided comprehensive insights into these two supports. The paper outlines design disparities and potential effects on operators' cognitive states, using eye movement metrics such as fixation, saccade parameters, visits to areas of interest (AOI), and pupil diameter for comparison. Results indicated that participants with paper-based procedures exhibited fewer fixations and longer saccade parameters than those using screen-based procedures, highlighting the impact of design choices on support tools for control room operators. 1. Introduction Engaging in information search and reading is integral to daily activities. However, how information is presented, whether as text, charts, or images, and the organisational pattern can influence processes like information processing, attention, recall, and mental load (Just and Carpenter, 1980). For instance, during reading, individuals pause at points of higher processing loads, such as longer or infrequent words. In control rooms, reading-intensive tasks arise when operating or emergency procedures and essential tools are used to guide operators in managing processes and ensuring safety. These procedures contribute significantly to accidents due to complexity, incompleteness, or inaccuracy (Park and Jung, 2003). Identified problems, including procedure complexity, search difficulty, and unfamiliar terminologies, lead to increased processing loads for control room operators. This can impact task completion time and accuracy, depending on operators' experience or training (Just and Carpenter, 1980). Prolonged use of procedures can also induce mental fatigue, affecting cognitive performance, attention, and error rates (Hinss et al., 2022). Researchers such as Rayner and Fischer (1996) and Shi et al. (2023) have utilized eye-tracking metrics to study the impact of systems, text, and representation styles during reading and searching in both control and non-control room scenarios. While previous studies have explored reading comprehension and cognitive workload, none have specifically examined the influence of different procedure formats (presentation styles, search support design) on operators' attention, workload, and visual fatigue. Therefore, this study aims to answer the research question: How do paper-based vs. digitised screen-based procedures impact control room operators? 2. Intervention Procedures Procedure complexity is one of the reasons for non-compliant behaviour when using procedures. This study compares a paper-based procedure commonly used in process plants and a digitised screen procedure designed by Amazu et al. (2023). The new digitised procedure was designed to reduce complexity precisely during search activities, potentially reducing search times. In both designs, the context and writing style remained the same. A hierarchical, rule-based task representation format was used to write the procedures, 223 detailing each step to resolve plant alarms. In terms of representation, one is presented on a display interface, and the other on screen. The authors reckon that the use of eye-tracking metrics can further give insight into any potential impact of each format on the cognitive states of the operators. The procedures are detailed below. 2.1 Digitised procedure For the digitised screen-based format, the procedure for each alarm is organised for potentially easier navigation, as shown in Figure 1a. They are arranged by plant section (Tank, Methanol, Compressor, Heat Recovery, Reactor, and Absorber plant sections) and by alarm number from 01 - n. The procedures in Figures 1a and b are within the tank section, and for the alarm, PAL01 (low-pressure alarm). 2.2 Paper procedures: For the sake of organisation and to support searching in this group, the paper procedure in Figure 2a-d has the table of contents at the beginning of the book (Figure 2b) and a colour tag at the beginning of each plant section in Figure 2d. Also, the content and writing style are similar to the digitised procedure in Figure. 2c. For this specific procedure, PAL01, in Figure 2c, the last task steps 8 - 6, essentially continuations of the first, differed in colour from those presented on the screen in Figure 1. The context of the intervention procedure, including the text style and others, was new to all who participated in the study presented in the paper. The paper- or screen-based formats were new to all participants, and all received training on how this is used or how it can be understood task by task. Hence, the effect of possible variation due to familiarity is removed. Figure 1: (a). procedure for PAL 01 steps 1 - 6, (b) continues for PAL01 steps 8 - 14. The left sidebars show the flow of alarm arrangements for easy navigation. Figure 2: (a). front page paper procedure, (b). sample table of contents, c. procedure for PAL01 (pressure alarm low) steps 1 to 14, (d) Actual look of procedure during the test with the colour tag indicating the start of the new plant section. 3. Methods 3.1 Case study and scenarios The experiment utilised three interface displays that provided information in real-time about a simulated formaldehyde production facility and some support tools, such as the different procedure formats to be used. This is based on the setup by Amazu et al. (2023). Three scenarios were run progressively with a 5-minute break in between. The digitised screen-based procedure in Figure 1 was presented on a Support Display. This display contained the alarm list and key graphical trends on its side and beneath. The control room included two 224 supervisors who observed the operators and acted as a reference in case of need to communicate or make inquiries depending on the instructions from the procedure. This paper will focus on the last and most critical scenario, scenario 3. 3.2 Scenario This critical scenario required participants to prevent the plant from overheating by keeping the plant stable within 18 minutes. 3.3 Groups: GPP and GSP The groups to be compared, their similarities and differences are broken down as follows; • Group Paper Procedure (GPP): the participants had the paper procedure with alarms prioritised. • Group Screen Procedure (GSP): Like the participants in GPP, those in GSP had alarms prioritised but used a digitised screen-based procedure. 3.4 Participants The results of 8 randomly selected participants in the test are presented in this paper. Four, 4, in each group (male = 6, female = 2 and mean age, M = 24, standard deviation, SD = 0.92). 3.5 Procedure The participants considered in this paper participated in each scenario only once. The experiment included signing the consent forms, training, filling out the preliminary questionnaire, testing the scenarios and breaks, completing workload, situational awareness questionnaires, and debriefing. This critical scenario lasted 18 – 20 minutes, depending on the participant’s progress and potential simulator lag. Before the critical scenario, scenario 3, participants completed questionnaires to assess their workload and situational awareness while wearing or not wearing the eye tracker, depending on their preference. They were asked to take a break before commencing the scenario. When ready, they wore the eye tracker, which was calibrated and validated. The eye-tracker used in this study for data collection was Tobii Pro Glasses 3, with a sampling rate of 50Hz. Only the participants wore the eye- tracker, while the supervisors observed without it. 3.6 Eye Tracker The following eye-tracking metrics have been selected for this study. A brief description of what they represent and similar studies in which these metrics have been used is provided in Table 1. • Heat Maps: These are used to analyse the area of the stimuli where participants paid more or less attention. This is based on the quantity of fixation at each point on a spatial distribution, not necessarily the visitation order (Mahanama et al., 2022). Red indicates more fixation numbers, and green indicates even less. • Pupillometry: This refers to the measurement of changes in pupil size. Studies have shown that pupil size is influenced by sympathetic and parasympathetic activations, leading to pupil dilations and contractions, respectively. Factors of influence include arousal levels, low or high luminance, bright or dark stimuli, low or high workload, expected or unexpected stimuli, and more (Fan et al., 2023). The indications of this metric on some of these factors, as observed by some studies, are shown in Table 1. • AOI analysis: AOIs are a tool or the region selected from a displayed stimulus to extract metrics specifically for these regions (Pederson, 2023). From the area of interest shown in Figure 3, the following metrics were selected for the group-wise comparison: fixation duration, no visits, saccade amplitude, saccade distance and pupil diameter. A brief explanation of the implications of this study is given below. • Number of visits or revisits: this metric refers to the number of times the AOI is visited. It indicates repeated attraction for better or worse. • Fixation duration (ms): how long the eye stays in a position (Mahanama et al., 2022) et al. They are known to be 50 - 600ms long and reflect the effort required for cognitive processing, and longer fixations indicate deeper processing (Rayner and Fischer, 1996). • Saccades Amplitude (°) and Duration (ms): Saccades are eye movements between fixations. The saccade amplitude is how far a saccade travels during eye movement, which is dependent on the task; for example, in reading, it is known to be approximately 2° for 7 - 8 letters in standard font size (Rayner and Fischer, 1996). They tend to decrease with increasing task and cognitive load (Mahanama et al., 2022; Shi and Rothrock, 2022). Therefore, larger saccade amplitudes indicate less difficulty in a search task or reading and less cognitive load. • The saccade duration is, in essence, the duration of a saccade and is known to be 20 - 40 ms (Rayner and Fischer, 1996). Studies have shown that longer durations can indicate increased cognitive processing (Das et al., 2017). 225 Table 1: Eye tracking metrics and their indications as identified in the literature. Measure Indication (s) Reference (s) 1. Mean Saccade Amplitudes A larger mean saccade could mean less search difficulty. Decreases as mental workload increases. Shi et al., 2023; Shi and Rothrock, 2023. 2. Fixation Duration Longer fixation duration indicates more difficulty in extracting information and more effort, but it also indicates that they are more likely to avoid errors. Also, higher cognitive loads lead to increased fixation durations. Shi et al., 2023; Li et al., 2018 3. Saccade duration Longer duration indicates decreased processing capacity. Reflects selective attention. Shi and Rothrock, 2023 4. No of visits/ Revisits In this paper's context, longer visits can indicate more attention demand, reading or processing effort or less effective support. Pedersen, 2023 5. Pupil diameter Smaller pupil sizes indicated lower cognitive workload during a task. More dilations are experienced given attention-demanding contexts. Bhavsar et al., 2016 4. Analysis Methods Tobii Prob Lab software was used for data analysis. The start time for the analysis for each participant from the overall 18 minutes was at the point of the first look at the intervention procedure after initiating the first critical alarm. The assisted mapping was then run between 6 minutes and 27-28 seconds, the selected time of interest (TOI) for each participant. The authors assume that this time window was critical for intervention, and an analysis of this time frame is sufficient given the time demand to analyse the entire recording. Area of interest (AOI) analysis was performed to extract eye-tracking metrics. This involved defining the paper and screen procedures as the key AOIs for this paper, as shown in Figure 3, before or while running assisted mapping for each participant. Unlike the mapping process, it is only done once and can be adapted for all participants. The heatmap was further generated by accumulating the eye movements of the participants over the TOI. The results from Tobii Pro Lab were further analysed using Python (version 3.9.18) to get the descriptive stats (mean +/- standard deviation) and grouped ANOVA in RapidMiner (free version 10.1.003) for a group comparison. A p-value of <0.05 was considered statistically significant. The data used to calculate the mean and SD for the pupil diameter was based on the difference between the participant's eye movement data during the selected TOI and their collective median base values (i.e. their median eye movement value within the first 1 minute 30 seconds of the start of scenario 3). Hence subtracting the latter from the former. This was done for each group and is known as the subtractive baseline correction approach, as recommended by (Mathôt et al., 2018). The other metrics were used without baseline correction. Figure 3: The green area on the table represents the AOI for the procedure on paper. The yellow area on the screen represents the AOI for the digitised screen-based procedure. 5. Results 5.1 Heat Maps Figures 4a and b show the heat map for GPP and GSP, respectively. As can be seen from both images, GSP had higher accumulated gaze duration on their AOI than GPP. However, both had longer gaze durations on the alarm support box. 226 Figure 4a: G2 accumulated eye movement. The average fixation duration on paper procedure AOI (Fig. 3) is 197.10 +/- 68.67 (Mean and SD respectively). Figure 4b: G3 accumulated eye movement. The average fixation duration on screen-based procedure AOI (Fig. 3) is 292.63 +/- 157.08 (Mean and SD, respectively. 5.2 Descriptive Stats: Pupillometry and AOI measures GPP had a shorter average fixation duration (p = .31) and longer number of visits (p = .96), average saccade amplitude (p = .02) and saccade duration (p = .11) on their AOI compared to GSP. In contrast, GSP had a significantly shorter mean pupil diameter (p = .001), as shown in Table 2. Table 2: Eye movement data's Mean and standard deviation, SD, after baseline correction of pupil diameter. Eye movement measures Mean +/- SD GPP GSP 1. Average fixation duration (ms) 197.25 +/- 68.67 292.63 +/- 157.08 2. No of visits (count) 5.13 +/- 3.20 5 +/- 3.70 3. Average pupil diameter (mm) 0.46 +/- 0.11 -0.51 +/- 0.29 4. Average saccade amplitude (°) 5.85 +/- 0.89 3.95 +/- 0.80 5. Saccade duration (ms) 37.07 +/- 3.77 32.39 +/- 3.29 6. Discussion 6.1 Heat Maps The high gaze on the alarm box can be understood to be due to the alarm flood situation during this scenario. Hence, increased bottom-up attention due to the alarm sounds and colours forces operators to pay more attention to that area of interest. This effect of constant alarm annunciations can also impact operators' top- down attention on the procedures, paper or screen. However, the GSP group fixated a lot more on their AOI than GPP, especially on the AOI area relevant for sorting through the alarms. This can indicate better top-down attention in the GSP group than in the GPP. The heat map analysis aligns with the outcomes from the average fixation duration data in Table 2. 6.2 Group-wise comparison Although no similar studies compare such procedure format, other relatable studies can be used to interpret the results obtained. In Table 2, longer saccade amplitude and duration were observed in GPP compared to GSP, indicating that those with the paper procedure had less search difficulty but decreased processing capacity than those with the screen procedure. Their short fixation durations might explain this reduced processing capacity. GSP showed a longer average fixation duration and shorter saccade parameters than GPP. Fan et al. (2023) observed the same fixation vs saccade behaviour pattern for less visually complex interaction modes in their study. Longer fixations show less divided attention and more deliberate processing on the AOI, as also observed in Figures 4a and b, and have been observed amongst expert users (Liu et al., 2013). However, longer fixations can also indicate difficulty in information processing and are likely to cause increased visual fatigue over time (Fan et al., 2023). GPP had a significantly higher average pupil size than GSP, indicating an increased mental workload while using the paper procedure. The smaller pupil size in GSP can be due to the effect of luminance, given that more light causes pupil constriction (Maqsood and Schumacher, 2017). 7. Conclusion From these preliminary observations, the GSP is ideal in cases of immediate emergency intervention, i.e., when using emergency procedures on similar complex tasks. For prolonged monitoring, such as when using operating procedures, the GPP setup is ideal. Based on this preliminary observation, the searching tasks seem less 227 difficult for those using paper than those on the screen. The number of participants considered for comparison in this study limits the statistical significance of some of the outcomes. The paper also needs to consider the overall intervention time frame, as the intervention lasted much longer than the TOI used in this paper. Further analysis of the progression of the outcomes towards a later time of interest can be compared with the results in this paper to observe the effect of prolonged or continuous use. Acknowledgements This work has been done within the Collaborative Intelligence for Safety-Critical Systems project (CISC). The CISC project has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement no. 955901 References Amazu C. W., Briwa H., Demichela M., Fissore D., Baldissone G., Leva M. C., 2023, “Analysing” 557 Human- in-the-Loop” for Advances in Process Safety: A Design of Experiment in a Simulated Process 558 Control Room,” 33rd European Safety and Reliability Conference (ESREL 2023), no. November, 559 pp.2780–2787 Bhavsar P., Srinivasan B., Srinivasan R., 2015, Pupillometry Based Real-Time Monitoring of Operator’s Cognitive Workload To Prevent Human Error during Abnormal Situations. Industrial & Engineering Chemistry Research. 55. 10.1021/acs.iecr.5b03685. Das L., Srinivasan B., Srinivasan R., 2017, Cognitive Behavior Based Framework for Operator Learning: Knowledge and Capability Assessment through Eye Tracking. 10.1016/B978-0-444-63965-3.50498-0. Fan L., Wang J., Li Q., Song Z., Dong J., Bao F., Wang X., 2023, Eye movement characteristics and visual fatigue assessment of virtual reality games with different interaction modes. Frontiers in neuroscience, 17, 1173127. https://doi.org/10.3389/fnins.2023.1173127 Hinss M. F., Brock A. M., Roy R. N., 2022, Cognitive effects of prolonged continuous human-machine interaction: The case for mental state-based adaptive interfaces. Frontiers in neuroergonomics, 3, 935092. https://doi.org/10.3389/fnrgo.2022.935092 Just M.A., Carpenter P.A., 1980, A theory of reading: from eye fixations to comprehension. Psychological Review, 87 4, 329-54. Li W.C., Kearney P., Braithwaite G., Lin J.J.H., 2018, “How Much is Too Much on Monitoring Tasks? Visual Scan Patterns of Single Air Traffic Controller Performing Multiple Remote Tower Operations.” International Journal of Industrial Ergonomics 67: 135–144. doi:10.1016/j.ergon. 2018.05.005. Mahanama B., Jayawardana Y., Rengarajan S., Jayawardena G., Chukoskie L., Snider J., Jayarathna S., 2022, Eye Movement and Pupil Measures: A Review. Front. Comput. Sci. 3:733531. doi: 10.3389/fcomp.2021.733531 Maqsood F., Schumacher U., 2017, Effects of varying light conditions and refractive error on pupil size, Cogent Medicine, 4:1, DOI: 10.1080/2331205X.2017.1338824 Mathôt S., Fabius J., Van Heusden E. et al., 2018, Safe and sensible preprocessing and baseline correction of pupil-size data. Behav Res 50, 94–106. https://doi.org/10.3758/s13428-017-1007-2 Park J., Jung, W., 2003, The operators’ non-compliance behavior to conduct emergency operating procedures— comparing with the work experience and the complexity of procedural steps. Reliability Engineering & System Safety, 82(2), pp.115–131. doi:https://doi.org/10.1016/s0951-8320(03)00123-6. Pedersen M., Mowla L., 2023, 10 most used eye tracking metrics and terms. Retrieved from https://imotions.com/blog/learning/10-terms-metrics-eye-tracking/#8-revisits Rayner K., Fischer M. H., 1996, Mindless reading revisited: eye movements during reading and scanning are different. parkPerception & psychophysics, 58(5), 734–747. https://doi.org/10.3758/bf03213106 Shi C., Rothrock L., Noah B., 2023, Using Eye-Tracking to Predict Cognitive Workload in a Control Room Environment. Human-Automation Interaction. Automation, Collaboration, & E-Services, vol 10. Springer, Cham. https://doi.org/10.1007/978-3-031-10780-1_12 Shi C., Rothrock L., 2023, Using eye movements to evaluate the effectiveness of the situation awareness rating technique scale in measuring situation awareness for smart manufacturing. Ergonomics, 66(8), 1090–1098. https://doi.org/10.1080/00140139.2022.2132299 228 https://imotions.com/blog/learning/10-terms-metrics-eye-tracking/#8-revisits https://doi.org/10.3758/bf03213106 https://doi.org/10.1007/978-3-031-10780-1_12 https://doi.org/10.1080/00140139.2022.2132299 129amazu.pdf Comparing Two Control Room Intervention Procedure Formats: Preliminary Insights from Eye Tracking Measures