DOI: 10.3303/CET2291089 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Paper Received: 9 February 2022; Revised: 22 March 2022; Accepted: 7 May 2022 
Please cite this article as: Leitner R., Miller R., 2022, Human Factors Assessment of a New Control Room System in a Nuclear Power Plant, 
Chemical Engineering Transactions, 91, 529-534  DOI:10.3303/CET2291089 
  

 CHEMICAL ENGINEERING TRANSACTIONS  
 

VOL. 91, 2022 

A publication of 

 

The Italian Association 
of Chemical Engineering 
Online at www.cetjournal.it 

Guest Editors: Valerio Cozzani, Bruno Fabiano, Genserik Reniers 

Copyright © 2022, AIDIC Servizi S.r.l. 
ISBN 978-88-95608-89-1; ISSN 2283-9216 

Human Factors Assessment of a New Control Room System 

in a Nuclear Power Plant 

Rodney Leitnera, Rainer Millerb 

aHFC Human-Factors-Consult GmbH, Köpenicker Str. 325, 12555 Berlin, Germany 
bMTO Safety GmbH, Gethsemanestr. 4, 10437 Berlin, Germany 

rodney.leitner@human-factors-consult.de, miller@mto-safety.de 

At Olkiluoto nuclear power plant (Finland), Areva and Siemens built a third unit (OL3) as a pressurised water 

reactor of the EPR construction line. The EPR is a generation III+ reactor and equipped with a fully digital HMI 

system (Process Information and Control System, PICS). As part of the “Integrated Final Control Room System 

Validation”, a large-scale human factors investigation was conducted. The aim of this study was to review human 

factors aspects of the ‘control room system’ in order to ensure that the operating crew is able to safely operate 

the reactor in all possible operating conditions using the tools available in the control room.  

For this study, the aspects to be evaluated were first defined based on the human factors requirements. A total 

of six human factors aspects were identified that were to be investigated. These aspects are task performance 

(defined performance targets), human errors (number and significance of observed errors), situation awareness 

(correct understanding of the current and future situation at critical points), communication (correct and sufficient 

communication), coordination (awareness of colleagues' tasks) and mental workload (Self-assessment of 

available mental resources). Four operating scenarios were defined which varied in content, difficulty, and 

complexity and which cover a wide range of conceivable operating conditions (normal to emergency operations). 

These four scenarios were conducted in the full scope simulator of OL3 with different shift crews.  

This paper describes the methodology in detail, which generally has proven to be very successful. It highlights 

the advantages and disadvantages of the methodology developed and provides valuable guidance for future 

investigations in the context of human performance in control rooms. 

1. Introduction 

The Olkiluoto nuclear power plant is located on the southwest coast of Finland. The plant is operated by the 

Finnish energy company TVO. Unit 3 (OL3) has been ordered as a turnkey delivery from the supplier consortium 

CFS formed by AREVA GmbH, AREVA NP SAS, and Siemens AG. Following first criticality in December 2021 

the first connection of the OL3 plant to the grid is planned for early 2022. 

The OL3 EPR is equipped with a fully HMI system (Process Information and Control System, PICS). In normal 

operation, the power plant will be controlled from the operator's control room via operating terminals. The OL3 

standard operating concept is designed for shift crews of three operators: Reactor Operator (RO), Turbine 

Operator (TO), and Shift Supervisor (SSV). If any incidents or major malfunctions occur, the shift team can 

involve an additional safety engineer (SE). Each operator has eight screens at his disposal, the display of each 

screen can be selected from a large number of pre-defined displays. At a central location in the main control 

room, three large power walls (Plant Overview Panel) provide plant status information to all operators. A backup 

system with conventional control panels and partly analogue displays and control elements is provided as a 

safety measure (Safety Information and Control System, SICS) in case that the PICS would be unavailable. 

Figure 1 shows a schematic representation of the full scope simulator (FSS) of the main control room of OL3; 

the control panel on the left of the control room is the SICS.  

The OL3 plant supplier has planned a final evaluation of the capabilities of the 'control room system' prior to 

start of nuclear operation in order to demonstrate that all requirements related to the human aspects of the 

operation are satisfactorily fulfilled in the control room (Koskinen and Norros, 2010). This study has been 

performed in 2019. The following aspects were considered as control room system:  

529



• trained plant operators (crew) 

• the operating manual (OM) defining the operational tasks of the crew, and 

• the integrated main human-machine interfaces, most notably the Process Information and Control System 

(PICS) as well as the Safety Information and Control System (SICS). 

In order to ensure the independence of the validation, the planning and implementation were outsourced. The 

HFC Human-Factors-Consult GmbH formed a consortium with MTO Safety GmbH for the study. Engineers, 

psychologists, and human factors specialists are represented in the consortium. The composition of the 

consortium guaranteed a great experience with human factors studies (HFC) as well as practical experience 

from nuclear operations and safety (MTO Safety). This paper describes the planning and implementation of the 

study, i.e., what aspects were examined in the study and which methods were used to measure the relevant 

variables. It also reports on hands-on experiences in conducting the study. 

 

 

Figure 1: Schematic representation of the full-scale simulator of the main control room  

2. Investigated Human Factors aspects  

The preparation for the study started in 2015 with the definition of relevant aspects to be analysed. The choice 

of these aspects was based on human factors requirements which in turn have been derived by considering the 

relevant standards and regulations within the domain of controls rooms in general and more specifically nuclear 

power plant control rooms. The standards considered include among others the IEC-1771 (1995) and IEC-

60964 (1989). The Finnish regulation YVL 5.5 (STUK, 2002) has to be considered for the main control room 

and man-machine interfaces in the control room and their validation. Furthermore, the NUREG-0711, Rev. 3 

(O’Hara et al., 2012) provides additional guidance. Based on the literature, a total of six aspects were identified 

that were to be investigated in the study. These aspects are task performance, human errors, situation 

awareness, communication, coordination, and mental workload. These aspects were operationalized using one 

or more testing instruments and one or more variables each. Figure 2 shows a schematic overview of the 

measurement time points of these human factors aspects. 

Success criteria were defined in advance for all human factors aspects. In order to increase the reliability of the 

measurement of the variables, subjective assessments and the collection of qualitative data (e.g., collection of 

the subjective assessments of the crew members) were avoided as far as possible and an attempt was made 

to develop clear criteria for the measurement of all variables. In the following, the six aspects and their 

operationalization are described in more detail.  

2.1 Task performance 

Task performance was the primary evaluation aspect. It considered the correctness and completeness with 

which the shift crews fulfilled their tasks. To determine the task performance, global success criteria (which were 

valid for all scenarios), scenario-specific success criteria, and task performance key nodes were defined. 

530



Success criteria 

At the end of a scenario, it was evaluated whether the following three global success criteria were met: (1) no 

unforeseen escalation of the scenario, (2) no damage of major equipment during the scenario, and (3) no 

relevant delay during the scenario should be caused by the behaviour of the crew. In addition, three to five 

scenario-specific success criteria (such as “crew is not going to SICS” or “reactor is timely at 25% power”) were 

defined for each scenario. 

Task performance key nodes 

Furthermore, four to eight critical points or decisions in the process were identified ('key nodes') for each 

scenario to evaluate the task performance. The expected behaviour at these key nodes was predefined. The 

generic assessment question was: does the crew behave as expected at the task performance key nodes? Data 

collection was done during the simulator sessions by observers for key node performance criteria and after the 

session during debriefing for scenario-specific and global success criteria.  

2.2 Human Error 

Based on a phenotype-oriented approach of human errors, errors of omission and errors of commission (Rouse 

and Rouse, 1983) were identified based on direct observation of the operators´ behaviours and related to the 

predefined target path for the scenario. If an operator deliberately deviates from specifications, e.g. from the 

OM, then this is not evaluated as an error, but as a deviation (and thus a topic of task performance and not of 

human error). Besides the occurrence of errors, it was observed whether the errors were detected by the 

operators and whether they were corrected. If errors occurred that were not detected and not corrected, the 

severity of these errors was classified by the study team as high, medium, or low. The impact of each error was 

evaluated case by case, with a focus on the probability of consequences before the error was detected and 

corrected, and the consequences of the error on plant safety and integrity. 

No errors with high severity should have been left undetected and uncorrected by the shift team. The number 

of errors with medium and low severity, which were not detected and not corrected had to be evaluated by the 

study team if it was acceptable. For the evaluation of acceptability, the length of a scenario, the number of tasks, 

and the type of operating procedure (Normal, Abnormal, or Emergency Operation) had to be taken into account. 

2.3 Situation Awareness 

In safety research, the concept of situation awareness was developed to describe the adequate understanding 

of the present state and the foreseeable future as a prerequisite for any safe operation (Dominguez, 1994; 

Endsley, 1995). For the study, the concept of shared situation awareness was adopted, meaning that situation 

awareness was ensured as soon as at least one member of the crew was aware of the present state and was 

able to act accordingly. To assess situation awareness, a multi-method approach was adopted: 

Method 1 – Status reports and team briefings  

One method to assess situation awareness was based on analysing the content and thoroughness of team 

briefs and status reports which are a genuine element of the team interaction in the control room. Status reports 

were understood here as summarized short descriptions of the operational status of the plant or the status of 

the most important parameters and operating conditions. Status reports were triggered by external requests 

(e.g. management calls, appearance of a safety engineer, shift changes, etc.), the relevant plant parameters 

and deviations from normal operation at that time were expected to be reported. Team briefs were requested at 

operational changes, or before starting safety-critical actions.  

Method 2 – Questions about future own tasks and colleagues’ tasks 

During freezes, additional questions were asked about future tasks and their objectives (“When the current task 

is completed, which will be your next task and its objective”) to determine whether future states could be correctly 

anticipated. These questions had to be answered concerning the own tasks as well as concerning the tasks of 

the colleagues in the shift crew (open-ended questions). 

Method 3 – Relevant process parameters or aspects 

A well-established method for the assessment is the Situation Awareness Global Assessment Technique 

(SAGAT) (Endsley, 1995) which allows for a real-time assessment during simulations a simulated environment 

and in the freeze ask agents about their understanding of the situation. In the study, the simulation was stopped 

at certain points (freezes) and operators were asked what the process aspects or process parameters in the 

current situation are relevant for the decision on further procedures. In order to predefine the correct answers 

for the freezes, we asked three experts (2 simulator trainers and one process engineer) independently of each 

other. For each freeze, six process parameters/aspects were offered in a multiple-choice test, of which three 

531



were correct and three were incorrect ("distractors"). In order to find a good balance of difficulty, the distractors 

should neither be absolutely irrelevant in the future nor too similar to the correct parameters.  

2.4 Communication 

For the evaluation of the communication, a distinction was made between the content of the communication and 

its quality. As for the quality of communication, it was distinguished between the communication types 2-way 

communication, 3-way communication, briefing, etc. For each of these communication types, observable 

behaviours were defined in order to assess the quality, e.g. if the communication was given in a "face-to-face" 

manner if the receiver showed attention, a reaction, or expressed or showed understanding. At relevant points 

in the scenarios, the quality of communication was evaluated not only according to observable criteria but also 

using an overall assessment of the communication process. A subjective assessment by the observers was 

used here (rating of poor, average, or good communication). 

2.5 Coordination 

Satisfactory coordination was characterized by the fact that all operators knew what their colleagues were doing 

(task awareness). For example, the supervisor should know which tasks the turbine operator (TO) and reactor 

operator (RO) are performing, the TO should know which task the RO is performing, and vice versa. The 

operators were asked: “Which task is performed by your colleague RO/TO?”. Questionnaires were used to 

collect data for coordination during freezes in parallel with methods 2 and 3 for situational awareness. 

2.6 Workload 

For workload assessment, the Bedford Workload Scale (Roscoe and Ellis, 1990) was used. The scale is 

unidimensional and evaluates whether it was possible to complete the task if the workload was tolerable for the 

task and if the workload was satisfactory without reduction. The scale uses the concept of spare capacity to 

define the levels of workload. A short explanation of the scale before the beginning of the scenario allowed the 

operators to use it properly and to rate their workload within seconds. This allowed repeated measurements of 

subjective workload during the scenarios without too much intrusion into the primary task. The workload rating 

was announced via loudspeaker every 20 minutes and the self-assessment of workload was performed 

immediately without freezing the scenario, on a tablet computer at their workstation. 

 

 

Figure 2: Schematic representation of the type of measurements during a scenario 

3. Preparation of the study 

The study was conducted in the full scope simulator (FSS) of the OL3 main control room. The FSS can be 

observed from an observation room through a reflective one-way window. All information displayed on the 

control room screens is also available on the screens in the observation room.  

3.1 Participants 

The scenarios were carried out with a total of six trained and licensed crews. Each crew participated in two 

different trials. The crews were neither informed about the test plan and the total number of scenarios nor which 

scenario they would be working on. All test participants had to declare in writing that they would not pass on 

any information about the courses of the test or details of the scenarios. 

532



3.2 Scenarios 

Four scenarios were defined which varied in content, difficulty, and complexity and which cover a wide range of 

conceivable operating conditions, from normal operation to abnormal operation to emergency operation. For 

each of the four scenarios, a target path with an expected sequence of specified control tasks or switching 

actions was defined, using the operating manual (OM). Each scenario was planned to be carried out three times 

within the study, using a different shift crew each time. This resulted in a balanced test plan of 12 trials (see 

Fig. 3). One trial was carried out per day. The scenarios covered normal operation (scenario A), abnormal 

operation (scenario B), and emergency operation (scenarios C and D). Furthermore, the scenarios contained 

numerous unexpected disturbances and additional tasks, e.g.: (1) smoke in control room, fire alarm, problems 

with components, (2) problems with digital controls, failure of electronic OM, and (3) short-term requests from 

network operator, communication with plant management, shift takeover, and so on. The scenarios were 

planned with a targeted duration of three to six hours.  

 

Figure 3: Distribution of crews to the scenarios 

3.3 Observers  

Three Human Factors Experts from the HFC/MTO Safety consortium served as observers for the study. In 

addition, three simulator trainers with thorough knowledge of the OL3 procedures and operational manual were 

asked to observe and evaluate task performance and human error. Each shift crew member was observed by 

an HF Expert and a simulator trainer. An additional expert from the plant owner TVO was responsible to simulate 

the external communication of the operators via telephone (e.g., plant management, network operator, etc.). 

3.4 Execution 

Before crews started their first scenario, they were introduced to the study. The crews then practised handling 

the tablet computers for data acquisition (questionnaires and workload on a test basis). The tablet computers 

were then taken to the workplaces of the operators. All crew members were provided with all information on the 

starting conditions for the scenario, such as the actual status of the plant and unavailable components. For each 

scenario at three predetermined points, the simulation was stopped (freeze). During these freezes, the crews 

had to take their tablets and leave the simulator. In an adjacent room, the tablet computers were used to collect 

data on situation awareness and coordination. After the freezes, the operators went back to their workplaces 

and the simulation was resumed. After each trial, a systematic debriefing was carried out, in which the test crews 

were asked about specific behavioural patterns and give their feedback on any inconsistencies concerning the 

OM or other system parts. The debriefing then continued without the operators to initially evaluate task 

performance (global & specific success criteria) and human error together with all observers. 

3.5 Documentation 

A special paper-based ‘observation tool’ was developed to structure the data collection. The observers filled in 

whether all expectations on the six human factors constructs were met (e.g. for task performance, whether 

predefined important tasks have been successfully completed). With the help of the observation tool, the course 

of the scenario could be followed and it was used to identify the measuring points and to document the 

measurement results immediately in writing. It was used to structure the debriefing. Questionnaires to assess 

situation awareness, coordination and workload were prepared and collected on mobile tablet computers. Audio 

and video recordings were used as a backup for the data acquisition with the observation tool. For the video 

recordings of the crews' actions, the three cameras permanently installed in the control room were used. In 

addition, the SICS panel and the screens of the operators were recorded with three mobile video cameras. 

4. Discussion 

4.1 Validity 

Investigations with the behavioural observation of this kind depend on the simulation being as realistic as 

possible. Only then conclusions can be drawn from the behaviour in the test situation to the behaviour in real 

situations. With two exceptions, the behaviour of the crews showed no signs that a realistic simulation would 

533



not have been successful. In one scenario, the crew is led by a phone call to the decision to evacuate the main 

control room (MCR) and switch to the remote shutdown station (RSS). During these trials, some difficulties were 

inducing a common understanding of the dangerousness of the situation. The crews interpreted the situation 

more or less problematically and therefore wanted to leave the MCR in some cases very quickly or not at all. It 

became apparent in the study that the members of the shift teams were sometimes unclear about what 

behaviour was expected of them. One possible limitation of the validity is that it is not certain for every observed 

behaviour whether this behaviour would be shown in the same way outside the test situation. 

4.2 Lessons learned 

In this study, an attempt was made to avoid subjective evaluations of behaviour as far as possible in order to 

make the results robust against subjective influences. With this approach, a large part of the data collected were 

quantitative, and could therefore be easily analysed in the subsequent analysis. For the variables task 

performance, situational awareness, communication, the evaluation was based on several different methods. 

This significantly increased the validity of the evaluation. If subjective assessments were necessary, then these 

assessments were always carried out by several experts. In case of discrepancies, these were discussed (e.g. 

in debriefing) in order to obtain a consistent result. The observation tool was extremely useful for the execution 

of the assessments and the documentation of the measurements. Nevertheless, not all measurements were 

easy to interpret in the evaluation. The measurement of situation awareness turned out to be the measurement 

with the most problems. One reason for the problems was the simple question of what time frame is meant with 

‘next task’. This was interpreted differently by the crew members, which affected the content of the answers. 

Furthermore, the level of detail was not specified. Some operators answered on a very low level of detail (“power 

increase up to 5%”), other on a very high level of detail (“open valve No. 1”). Both answers were correct, but it 

was difficult to assess whether the different operators really had an identical understanding of their tasks. Finally, 

the freeze time point was also very important. If the freeze were only slightly postponed, this could influence the 

answers, e.g., for relevant parameters and process aspects and their assumed course. 

5. Conclusion 

Apart from a few detailed problems in recording situation awareness, the combination of methods for the human 

factors constructs has shown to be very successful. The results helped Finnish radiation and safety authority 

STUK to conclude on operational readiness of the OL3 plant prior to granting the permit for start of nuclear 

operations with the first fuel loading of the reactor. The documentation and evaluation of the data collected using 

the observation tool and tablet have proven to be very useful. Especially the approach to capture as many 

assessments as possible directly and synchronously was very feasible and highly efficient. Therefore, the 

necessity to use audio or video recordings was minimized. For future studies, it would be helpful to have even 

more methods available for assessing situation awareness. The study also showed that - especially for the 

evaluation of situation awareness - careful preparation is necessary to achieve reliable results. Nevertheless, 

the methodology developed provides valuable guidance for future investigations in the context of human 

performance in control rooms.  

References 

Dominguez, C., 1994, Can SA be defined. Situation awareness: Papers and annotated bibliography, 5-15. 

Endsley, M. R., 1995, Measurement of Situation Awareness in Dynamic Systems, Chapter In: Human Factors, 

Vol 37 (1), Human Factors and Ergonomics Society, United States, 65-84. 

IEC-1771, 1995, Nuclear Power Plants - Main Control Room - Verification and Validation of Design, IEC - 

International Electrotechnical Commission, Geneva, Switzerland. 

IEC-60964, 1989, Design for control rooms of nuclear power plants, IEC - International Electrotechnical 

Commission, Geneva, Switzerland. 

Koskinen, H., Norros, L., 2010, Expanding control room – a new frame for designing spatial affordances of 

control places. Research Report, VTT Technical Research Centre of Finland. 

O’Hara, J., Higgins, J., Fleger, S., 2012, Human factors engineering program review model (NUREG-0711), 

Revision 3: Update Methodology and Key Revisions. Brookhaven National Lab (BNL-96812-2012-CP), 

Upton, New York, United States. 

Roscoe, A., Ellis, G., 1990, A Subjective Rating Scale for Assessing Pilot Workload in Flight: A Decade of 

Practical Use, Royal Aerospace Establishment, Defence Technical Information Center, Farnborough, UK. 

Rouse, W. B., & Rouse, S. H., 1983, Analysis and classification of human error. IEEE Transactions on Systems, 

Man, and Cybernetics, (4), 539-549. 

STUK, 2002, YVL Guide 5.5 - Instrumentation systems and components at nuclear facilities, Radiation and 

Nuclear Safety Authority, Helsinki, Finland. 

534


	147leitner.pdf
	Human Factors Assessment of a New Control Room System in a Nuclear Power Plant