Australasian Journal of Educational Technology, 2022, 38(2). 49 An examination of student user experience (UX) and perceptions of remote invigilation during online assessment Lesley T. Sefcik, T. Veeran-Colton Office of the Academic Registrar, Curtin University M. Baird School of Management and Marketing, Curtin University C. Price Curtin Learning and Teaching (now Learning Innovation & Teaching Excellence Centre), Curtin University S. Steyn Codemaster Institute, Perth, Western Australia This study aimed to understand the effects of a custom-developed, artificial intelligence– based, asynchronous remote invigilation system on the student user experience. The study was conducted over 3 years at a large Australian university, and findings demonstrate that familiarity with the system over time improved student attitudes towards remote invigilation. Positive experiences were found to be related to ease of use and convenience for test sitting. The majority of students reported that it was important for the institution to have approaches such as remote invigilation to discourage cheating and they believed that the system was useful in this regard. Perceived technical problems were found to invoke feelings of anxiety with being remotely invigilated, and students suggested that greater clarity on expectations of appropriate behaviour, privacy and data security would help alleviate discomfort and improve the system. Implications for practise or policy: • Educators can improve the student user experience of remote invigilation by ensuring that students are provided the opportunity to practise and become familiar with using remote invigilation software before any summative assessment task. • Administrators should provide clear policy guidance about the management of student data collected during remotely invigilated assessment tasks. Keywords: remote invigilation, invigilation, online tests, online learning, artificial intelligence, user experience Introduction Online education (e-learning) is a fundamental part of tertiary education with universities rapidly growing their repertoire of courses (Jefferies et al., 2017). For example, Swinburne University of Technology (Australia) reported an exponential increase in online undergraduate enrolments from 3 students in 2011 to 30,164 students in 2012 (Greenland & Moore, 2014). In 2018, Open Universities Australia, a popular online education platform with 13 university partners, reported a student base of up to 350,000 (Cameron, 2018). This increase has been magnified with the current rapid transition to e-learning necessitated by the SARS- COV2 global pandemic. However, this form of education presents both benefits and challenges with regard to assessment (Alessio et al., 2017). Like exams and assessments completed in person in traditional testing venues, online assessments (e-assessments) are vulnerable to cheating (Fask et al., 2014). Cheating undermines academic integrity and threatens the reputation of educational qualifications (Fendler & Godbey, 2016; McCabe et al., 2012). Cheating methods used online sometimes differ from those in traditional testing environments (Vegendla & Sindre, 2019). Impersonation and preventing students from accessing unauthorised assistance or materials are challenges for e-learning that must be managed (Boyd Davis et al., 2016; Fenu et al., 2018). Australasian Journal of Educational Technology, 2022, 38(2). 50 Parnther (2020) suggested that “academic integrity is the cornerstone of the learning process” (p. 25). Thus, maintaining academic integrity in rapidly growing e-learning environments is a critical issue for higher education stakeholders (Alessio et al., 2017; Baird et al., 2019; Fenu et al., 2018; Ullah et al., 2016; Woldeab et al., 2017). To address this issue and mitigate cheating risks, institutions are adopting remote invigilation (also known as remote proctoring) methods for e-assessments as one strategy to better assure integrity (Boyd Davis et al., 2016). Remote invigilation systems Online remote invigilation systems are generally provided as a software as a service solution. This involves students downloading specific software onto their computers, which allows webcam, microphone and desktop feeds to be viewed or recorded by a third-party service provider while they are completing an assessment. Some systems utilise lock-down browsers to help prevent access to unauthorised materials (Jefferies et al., 2017). The provider then makes the recordings or reports available to the educators as evidence of assessment integrity. Assessments can be completed from the students’ preferred location or within a designated testing centre (Atoum et al., 2017; Milone et al., 2017). Institutions are increasingly requiring the use of this type of software to facilitate sitting for e-assessments remotely (outside of normal invigilated testing environments in traditional campus venues) to ensure integrity (Mellar et al., 2018). These services generally operate in one of two ways: (a) real-time invigilation with a live invigilator or (b) asynchronous monitoring, which records the assessment process, and recordings are reviewed post-hoc by either a live invigilator or by artificial intelligence (AI) (Atoum et al., 2017; Lilley et al., 2016; Vegendla & Sindre, 2019). Real-time remote invigilation Real-time live remote invigilation entails that a human invigilator monitors student behaviour through their webcam feed while they are in the process of sitting for their e-assessment (Lilley et al., 2016). The invigilator is able to see and hear the student through their webcam, and the student generally is required to show the invigilator the testing environment. Real-time live remote invigilation requires that a student complete an assessment at a designated time in which they meet online with the invigilator. A disadvantage to live remote invigilation is that it is an expensive and resource-intensive service that is not easily scalable (Atoum et al., 2017; Fenu et al., 2018). Invigilators may be monitoring tens of students simultaneously (Jefferies et al., 2017). This type of invigilation may also limit flexibility for the student to sit the assessment at a time that suits them best. Asynchronous remote invigilation For asynchronous live remote invigilation, a live invigilator reviews the recording of the assessment process post-hoc and provides a report to the educator. For automatic or asynchronous remote invigilation with AI, the AI reviews the recordings either post-hoc or concurrent with the assessment process and provides a summary for the educator to review. The main benefit of asynchronous remote invigilation for students is increased flexibility as they can complete the assessment without having to schedule a session with an invigilator. Automatic invigilation systems may also be more cost-efficient by eliminating the need for numerous human invigilators and reducing workload by narrowing the scope of review (Atoum et al., 2017). Since asynchronous AI or automatic remote invigilation is nascent, literature exploring experiences with this system is limited. Studies appear to focus on the effect of automatic remote invigilation on academic integrity (Bedford et al., 2011; Fayomi et al., 2015; Hylton et al., 2016) or student performance (Alessio et al., 2017; Boyd Davis et al., 2016; Rios & Liu, 2017). A few studies have identified student or staff experiences with the system (Bedford et al., 2009; Karim et al., 2014; Lilley et al., 2016; Milone et al., 2017). These studies highlight a lack of nuanced understanding of the finer characteristics within remote invigilation systems that can influence user experience (UX). This study drew broadly from literature where remote invigilation is performed by AI or by humans that may be comparable to the data collected for this study. A deeper understanding of student experiences during remote invigilation may enhance the quality and delivery of online education. Australasian Journal of Educational Technology, 2022, 38(2). 51 Study objectives The use of remote invigilation as a risk-mitigation strategy to assure the integrity of e-assessment affects those who use the system. This study aimed to understand the effects of a custom-developed asynchronous AI-facilitated remote invigilation system on the student UX (Hassenzahl & Tractinsky 2006). The key research questions were: • How do the system characteristics, users’ internal state and contextual factors influence the student UX of an asynchronous AI remote invigilation system during online assessment? • How can the student UX be improved? This study contributes to a growing body of literature evaluating the impact and experience of remote invigilation in online education. Materials and methods System overview Curtin Remote Invigilation System (CRIS) functions as an asynchronous AI remote invigilation system. CRIS helps verify student identity through face matching; records the audio, video and screen of the student work environment; and analyses student movements during online examinations in order to flag behaviour that may indicate academic dishonesty. We (the authors of this paper) are the developers of CRIS. The system uses an avatar to guide students through the process of using CRIS. (1) Students install the software by downloading the required extension for their internet browser. (2) Students access their e-assessment through their institution’s learning management system (LMS). (3) CRIS is triggered when the student attempts to begin their assessment and the avatar describes the purpose of using CRIS and the set-up process. Students using CRIS for the first time receive the full instruction suite. Returning student users receive a condensed version of the set-up instructions from the avatar. (4) Students provide their name and student identification number and give access to their webcam and microphone feeds. (5) Next, students provide a screenshot of their identification card for identity verification purposes. (6) Students then share their computer desktop screen feed. (7) CRIS populates a specific password in the LMS giving the student access to their assessment. This prevents students from accessing the assessment without using remote invigilation. (8) Students sit their assessment with CRIS recording the session. (9) Students complete their assessment and then submit it to the LMS. (10) CRIS uploads the assessment recording while the avatar explains the process to the students. (11) CRIS closes automatically when the student recording has uploaded successfully to the server dashboard. (12) CRIS analyses the student video and uses customised face detection algorithms to flag particular video frames for review. Flagged video frames from the student recording are concatenated to produce a streamlined summary for the educator to review post assessment. (13) Staff members are able to review the recordings within the dashboard. Analytical framework The analytical framework developed for the study was inspired by the need to understand how student users of CRIS experience the system and how this experience influences their attitudes and feelings. The concept of UX, underpinned this framework. Although there is no explicit or widely accepted definition for UX, authors generally agree that it is a multifaceted concept, which is variously defined and conceptualised across disciplines (e.g., Gómez-López et al., 2019; Sauer et al., 2020; Simonsen, 2018). According to Datig (2015), behaviours and emotions are central constructs in UX research. Hence, UX provides an analytical lens through which to understand how students interact with CRIS as a technology and how certain behaviours and emotions may transpire as a result. Australasian Journal of Educational Technology, 2022, 38(2). 52 Hassenzahl and Tractinsky (2006) defined UX as: A consequence of a user’s internal state (predispositions, expectations, needs, motivation, mood, etc.), the characteristics of the designed system (e.g. complexity, purpose, usability, functionality, etc.) and the context (or the environment) within which the interaction occurs (e.g. organisational/social setting, meaningfulness of the activity, voluntariness of use, etc.). (p. 95) This definition was applied to this research and specifically focused on the student user’s internal state, characteristics of CRIS and the context or environment in which students interact with the system. Data collection and analysis The study was conducted at Curtin University, a large Australian university, between 2017 and 2019, providing data from 6 semester study periods. Prior to data collection, Curtin University Human Research Ethics Committee approved the study (HREC number 2017-0114). The study required participants who had experience using CRIS; therefore, convenience and purposive sampling was used to target individuals who met this criterion and were accessible to us (Morse, 2004). e-survey A mixed-method e-survey was designed for data collection. Mixed methods present a pragmatic approach to research that draws on the strengths of both qualitative and quantitative strategies (Shorten & Smith, 2017). This allowed the survey to be time efficient whilst allowing respondents the opportunity to express their views. At the end of each semester, the e-survey was open to students who had completed online assessments using CRIS. Following analysis of initial data, additional questions were incorporated (Questions 19–28, Appendix). Drawing from the UX construct, survey questions were grouped into four main themes (Figure 1). These themes provided a framework to guide the data analysis and systematically present the findings of the survey. Figure 1. Survey questions grouped into themes Australasian Journal of Educational Technology, 2022, 38(2). 53 Data analysis Data was input into IBM SPSS version 26 and analysed using frequencies, cross-tabulations and inferential statistics to explore the relationships between variables. Cross-tabulations helped determine how categories may influence choices, whilst inferential statistics highlighted statistically significant relationships within the data set. As the survey contained mostly ordinal and nominal data categories, non-parametric statistical tests such as the chi-square test for independence and Wilcoxon signed-rank tests were best suited to the data set as these do not require the data to be normally distributed (Hoskin, 2012; McHugh, 2013; Pereira & Leslie, 2010). For a relationship to be statistically significant in this study, the significance level (p value), must be p < 0.05 (McCrum-Gardner, 2008). Inferential statistics were run on specific questions that were answered by the entire sample (N = 253 participants), with no missing values. Chi-square test for independence The chi-square test of independence is best suited to evaluating the statistical significance of the relationship between independent variables (Gilbert & Prion, 2016). It should be noted that the study sample is a non- probability convenience sample; hence, the condition for a random sample for the chi-square test has been violated. However, using a chi-square test for independence is not unusual in convenience samples; hence, the test is applicable to the study, though results cannot be generalised beyond the survey sample (McHugh, 2013). Wilcoxon signed-rank test Other relationships of interest in the study involved students being surveyed regarding two points of time in the data set, for example, before and after taking part in the study or becoming familiar with CRIS. The Wilcoxon signed-rank test is best suited to exploring the relationship between match-paired, ordinally ranked data, where the mean difference between the variables is tested (McCrum-Gardner, 2008; Woolson, 2008). It tests the null hypothesis that the pair-wise differences in the set will have a distribution centred around zero (Woolson, 2008). Within the data set, two sets of questions were applicable to this type of statistical significance testing: Questions 6–7 and 10–11. Qualitative responses For the two open-ended questions (Questions 16 & 28, Appendix) that generated qualitative data, a template analysis approach was adopted (Brooks & King, 2012; King, 2012). Question 16 explored how the student experience could be improved and was answered by the entire sample (N = 253). Question 28 asked if students thought that any questions or topics should be included in the resources or covered by the avatar. This was answered by 70% (n = 177) of the sample. Qualitative data was thematically analysed in NVivo 12 Pro, with Questions 16 and 28 forming the central concept or initial priori parent codes and nodes (Braun & Clarke 2013). The data in each parent code and node was then analysed for meaning conveyed through asking questions of the data, which included: • “What is the student referring to, a topic or issue within CRIS?” • “What words are being used to describe things?” • “What emotion is conveyed through their language?” • “Was the experience negative or positive?” We recognise that developing meaning and themes was based on our personal interaction with the data (Varpio et al., 2017). To ensure reliability, we developed a code book with descriptions of each code to allow us to evaluate the content of the template. The qualitative data analysis was reviewed to ensure that quotes were appropriate, impactful and matched the codebook descriptions. Where appropriate, responses were coded to multiple relevant themes. The subsequent Results and Discussion sections are structured according to the broad survey themes shown in Figure 1 using the UX model. Where results are represented as part of a sample of 70% (n = 177), this refers to the additional Questions 19 to 28 from 2018 Semester 2 to 2019 Semester 2 that were not answered by the entire sample. Australasian Journal of Educational Technology, 2022, 38(2). 54 Results Demographics A total of 253 students completed the e-survey between 2017 Semester 1 and 2019 Semester 2. Figure 5 shows the breakdown of participants for each year and semester. The sample was made up of mostly females (57.3%; n = 145) and 41.5% of males (n = 105); 1.2% of the sample preferred not to disclose their gender. The highest number of respondents was recorded for the year 2019 making up 60.9% of the total sample size (Figure 2). This is a result of increased usage over time with study expansion. The three largest age groups represented were 18–24 (43.1%, n = 109), 25–34 (32%, n = 81) and 35–44 (14.2%, n = 36). Other age groups represented were under 18 (1.2%, n = 3), 45–54 (5.9%, n = 15), 55–64 (1.2%, n = 3), 65–74 (0.8%, n = 2) and 1.6% (n = 4) preferred not to disclose their age. Assessments varied from 10 minutes in duration to more than 90 minutes, with some having a specific start time and others having a window of completion. Assessments ranged in type from short low-stakes e-tests to higher-stakes mid- semester examinations. Participants were located in the disciplines of public health, business, economics, finance, law, human biology and construction management and generally ranged from first- to third-year undergraduates, with a fewer number of postgraduate students. Overall, there was a wide variety of assessment types, disciplines and student groups involved in the study. Familiarity with remote invigilation It was important to understand student familiarity with remote invigilation, as this can influence their experience and comfort level with the system. Familiarity was determined by asking questions related to comfort, improved ability, speed of use and perceived ease of use. Perceived ease of use is related to improved ability and speed using the system. To gauge the impact of familiarity on various factors (shown in Figure 2), students were asked a range of questions. Figure 2. The influence of familiarity on different factors The majority of students (75.1%, n = 190) reported not having heard of remote online invigilation before participating in the study. This trend was consistent across all years. Other students (24.9%, n = 63) had heard of remote invigilation. Of those 63 students who had heard of remote invigilation, 60% (n = 38) knew what it meant and 40% (n = 25) did not. Australasian Journal of Educational Technology, 2022, 38(2). 55 Most students (82.6%, n = 209) had no prior experience using remote invigilation in their studies. This trend was observed across all years. From those who did have prior experience, only 3.6% (n = 9) had used it eight or more times, while 12.3% (n = 31) used it up to three times and 1.6% (n = 4) had used it between four and five times. Leading from this, it was important to understand how comfortable students felt with the idea of being remotely invigilated before and after participating in the study (Table 1). Table 1 Comfort level with the idea of remote invigilation before and after participating in the study Comfort level Before participation After participation % n % n Extremely comfortable 9.9 25 18.6 47 Comfortable with minor reservations 19.8 50 25.7 65 Neutral 24.9 63 20.9 53 Somewhat uncomfortable 31.2 79 19 48 Extremely uncomfortable 14.2 36 15.8 40 Total 100 253 100 253 Prior to participating in the study, the two most prominent comfort levels were somewhat uncomfortable, (31.2%, n = 79), followed by neutral (24.9%, n = 63); (see Table 1). After participating, this trend shifted towards feelings of being comfortable with minor reservations (25.7%, n = 65) and neutral (20.9%, n = 53). The number of students who felt extremely uncomfortable increased marginally from 14.2% (n = 36) to 15.8% (n = 40) after study participation. More notably, there was an increase in students feeling extremely comfortable from 9.9% (n = 25) to 18.6% (n = 47) after taking part in the study. Before evaluating the relationship between ease of use, usefulness and attitude in more depth, it is important first to understand how easy or difficult students found CRIS to use; particularly, how familiarity affected ease of use. To explore this, students were asked how easy or difficult they found CRIS to use at first, as well as after they became familiar with the system (Table 2). Table 2 Ease of use when first using CRIS compared to after becoming familiar with the system Ease of use At first After becoming familiar % n % n Extremely easy 23.3 59 41.9 106 Somewhat easy 36 91 34.4 87 Neither easy nor difficult 16.6 42 13.8 35 Somewhat difficult 18.6 47 7.5 19 Extremely difficult 5.5 14 2.4 6 Total 100 253 100 253 Most students (36%, n = 91) found the system somewhat easy to use with a minority (5.5%, n = 14) finding it extremely difficult (Table 2). However, after becoming familiar with the system most students 41.9% (n = 106) found CRIS extremely easy and 34.4% (n = 87) found the system to be somewhat easy. Additionally, most students reported being able to set up CRIS within a relatively short time frame with 35.6% (n = 90) taking 5–10 minutes and 32% (n = 81) taking less than 5 minutes. However, 16.6% (n = 42) did report taking more than 15 minutes to set up CRIS. Overall, the majority of students (75%, n = 192) felt that they became faster at using CRIS over the course of their unit. Australasian Journal of Educational Technology, 2022, 38(2). 56 Perceptions of cheating Student perceptions of cheating formed a broad theme within which four factors were evaluated to better understand their thoughts on the issue (Figure 3). Figure 3. Perceptions of cheating Student perceptions were mostly positive for the efficacy of remote invigilation for deterring cheating. Within the total sample, 20.9% (n = 53) thought it was extremely effective, 30% (n = 76) very effective and 28.1% (n = 71) moderately effective. However, 11.1% (n = 28) thought the system was only slightly effective and 9.9% (n = 25) not effective at all. Students (N = 177) appeared to be divided on how important they thought it was for the university to have approaches such as remote invigilation to discourage cheating. Approximately 33.3% (n = 59) thought it was extremely important and 33.3% (n = 59) believed that it was somewhat important. A minority reported it was not very important (3.4%, n = 6) or not at all important (5.6%, n = 10). However, when asked to consider the impact and likelihood of potential cheating in relation to the aptness of the assessment process, including the use of CRIS, the majority of students (67.2%, n = 119) believed that processes were appropriately balanced with the impact of cheating. Others (23.2%, n = 41) believed that processes were excessive in comparison to the impact of cheating and a minority (9.6%, n = 17) believed that the processes were inadequate. Students (N = 177) indicated that they understood the expectations of appropriate student behaviour while test sitting and being remotely invigilated. Most students reported this to be extremely clear (48.6%, n = 86) and somewhat clear (36.2%, n = 64). The remaining students found expectations to be neither clear nor unclear (4.5%, n = 8), somewhat unclear (8.5%, n = 15) and extremely unclear (2.3%, n = 4). UX and efficacy Students (N = 253) provided insights from their UX. For example, students were required to perform a familiarisation activity prior to commencing their e-assessment. Most students (67.2%, n = 170) completed the task and found it to be useful. Others (20.6%, n = 52) thought it was unnecessary and 7.9% (n = 20) completed the check but thought that it did not matter, while a minority (7.9%, n = 20) did not do the check but wished they had. Only 15 (5.9%) students out of the entire sample indicated that they did not have the Australasian Journal of Educational Technology, 2022, 38(2). 57 necessary technology to use CRIS, which included a strong internet connection, a microphone, a webcam or a combination of these. The theme of efficacy explored how effective students found various elements of CRIS to be (Figure 4). Figure 4. Efficacy of various elements of CRIS Students (N = 177) generally agreed that the CRIS avatar was either extremely effective (24.9%, n = 44), very effective (29.9%, n = 53) or moderately effective (28.2%, n = 50) at helping them use the system correctly, whilst 7.9% (n = 14) found it slightly effective and 9% (n = 16) found it to be not effective at all. Of the 177 students, approximately 35% (n = 62) accessed at least one additional resource, which included the information sheet (n = 40), frequently asked questions (n = 41) or set-up instructions (n = 57). Students were asked to rate the resource efficacy and most students found the additional resources accessed to range between extremely effective to moderately effective. This indicates that additional resources were useful in assisting students with using the system; however, a small number of students found these resources to be slightly effective or not effective at all, and this highlights that there is room for improvement. Attitude To understand how the experience of using the system affected their attitudes, students (N = 253) were asked if their attitudes had become more positive, more negative or if they had remained the same. Overall, the majority of students (49%, n = 124) reported that their attitude had not changed while 30% (n = 76) had become more positive about remote invigilation. The remainder (20.9%, n = 53) reported that they had become more negative about the process (Figure 5). Australasian Journal of Educational Technology, 2022, 38(2). 58 Figure 5. Change in attitudes towards remote invigilation over survey period from 2017 Semester 1 to 2019 Semester 2 We wanted to understand if there was a relationship between students getting quicker at using CRIS after becoming familiar with the system and their attitude towards remote invigilation after completing their unit. Here the null hypothesis is that there is no relationship between efficiency at using the system and attitude. A cross-tabulation with a chi-square test was carried out in SPSS version 26. The results of the chi-square test [X2 (2, N = 253) = 36.8, p < 0.001] indicate that the null hypothesis is rejected and the alternative hypothesis that there is a significant association between the variables of speed and attitude is accepted. Figure 6 shows that students who reported becoming quicker at using the system after becoming familiar with it, were generally more positive and less negative towards remote invigilation than those that did not become quicker. Figure 6. Becoming quicker at using CRIS versus attitude towards remote invigilation graphed by proportion of total within attitude category Interestingly, even though 75.9% of students (n = 192) had become faster at using CRIS, within this group 50% (n = 96 out of 192) indicated that their attitude towards CRIS had not changed. This may indicate that whilst time efficiency of CRIS is important, other considerations of UX should be made in improving the system and encouraging more positive attitudes towards CRIS. 0% 10% 20% 30% 40% 50% 60% 70% 2017-S1 (n = 31) 2017-S2 (n = 25) 2018-S1 (n = 20) 2018-S2 (n = 23) 2019-S1 (n = 51) 2019-S2 (n = 103) Total (N = 253) P e rc e n t o f re sp o n se s Year, semester (S) and number of responses (n) I have become more positive about remote proctoring My attitude has not changed 93% 7% 77% 23% 47% 53% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Yes (n = 192) No (n = 61) P ro p o rt io n o f re sp o n se s (% ) Q13 Did you get quicker at using the remote invigilation software over the course of the unit? I have become more positive about remote proctoring (n = 76) My attitude has not changed (n = 124) I have become more negative about remote proctoring (n = 53) Q14 How has your attitude towards remote invigilation changed over the course of the unit? Australasian Journal of Educational Technology, 2022, 38(2). 59 Comfort Questions 6 and 7 explored student levels of comfort with the idea of being remotely invigilated before and after participating in the study (N = 253). A Wilcoxon signed-rank test indicated that post-test ranks were statistically significantly higher than pre-test ranks; Z = 4216, p < 0.001 (Figure 7). Questions 10 and 11 explored difficulty levels with using CRIS before and after becoming familiar with the system. A Wilcoxon signed-rank test indicated that post-test ranks were statistically significantly higher than pre-test ranks; Z = 5566, p < 0.001 (Figure 8). Figure 7. Difference between post and prior levels of comfort Figure 8. Difference between levels of difficulty before and after becoming familiar with CRIS. Overall results from the Wilcoxon signed-rank tests show that post ranks showing comfort levels improving and difficulty levels easing is statistically significant. These findings suggest that CRIS became easier to use over time and student discomfort at being remote invigilated also dissipated with continued use. To further explore this relationship, cross-tabulation between difficulty level after becoming familiar with CRIS and attitude towards remote invigilation was performed (Figure 9). Australasian Journal of Educational Technology, 2022, 38(2). 60 Figure 9. Difficulty level after becoming familiar with CRIS versus attitude towards remote invigilation Figure 9 demonstrates that students who found CRIS to be extremely easy to use after becoming familiar with the system had mostly become more positive towards remote invigilation. This suggests that ease of use positively influenced attitudes towards remote invigilation. Interestingly, the 18–24 age group demonstrated the greatest proportional levels of discomfort with the idea of being remotely invigilated after using the system compared to other age groups (Figure 10). Figure 10. Discomfort level with being remotely invigilated before and after becoming familiar with CRIS by age group. A missing bar indicates all participants in the category selected neutral or comfortable responses to using the system. 55 17 3 1 42 56 19 7 9 14 13 11 6 0 10 20 30 40 50 60 Extremely easy Somewhat easy Neither easy nor difficult Somewhat difficult Extremely difficult N u m b e r o f re sp o n se s (n ) After you became familiar with the remote invigilation software, please describe how easy or difficult it was to use. I have become more positive about remote proctoring My attitude has not changed I have become more negative about remote proctoring 0 % 6 1 % 3 5 % 3 9 % 2 0 % 3 3 % 1 0 0 % 2 5 % 0 % 4 6 % 3 1 % 2 5 % 1 3 % 3 3 % 0 % 2 5 % 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% U n d e r 1 8 ( n = 3 ) 1 8 - 2 4 ( n = 1 0 9 ) 2 5 - 3 4 ( n = 8 1 ) 3 5 - 4 4 ( n = 3 6 ) 4 5 - 5 4 ( n = 1 5 ) 5 5 - 6 4 ( n = 3 ) 6 5 - 7 4 ( n = 2 ) I p re fe r n o t to s a y ( n = 4 ) P ro p o rt io n o f u n c o m fo rt a b le s tu d e n ts o u t o f to ta l w it h in a g e g ro u p Age group (number surveyed in category) Uncomfortable before Uncomfortable after Australasian Journal of Educational Technology, 2022, 38(2). 61 Qualitative responses Students were given an opportunity to recommend enhancements for CRIS after having used the system. Students were asked, “What could be done to improve your experience of being remotely invigilated during your online tests?” (Question 16, Appendix). References (N = 274) were coded, and the themes of emotional responses, technical issues and varied suggestions contained the most coded references. Responses that made no suggestions for improvements to the system were coded as neutral. The following subsections explore data within themes that yielded the most valuable insights. Positive experiences and opinions Students (n = 14) provided positive experiences/opinions where factors such as convenience and ease of use were highlighted as being important. One student commented, “the process was simple and effective” and another stated, “it was very convenient for me to be able to use this, as it meant I could take the test after hours, at my leisure, and without driving to a venue”. Negative experiences and opinions Students (n = 15) also conveyed negative experiences with CRIS. Some students found that the process of being recorded adversely affected their ability to concentrate and complete their task with confidence, for example, “I did not enjoy this at all – I found myself concentrating more on the system focusing on where I was looking and was very uncomfortable…I felt this method hindered my concentration and performance”. Privacy issues Some students (n = 15) flagged privacy related issues as a feature to address to improve their experience of being remotely invigilated. One student commented, “I think it’s a bit concerning that data (video, mic, etc) is recorded of me doing tests on some server. [It] feels a bit like an invasion of privacy.” Suggestions for improving this centre around having, “a clear and simple privacy policy. Where is the data stored and for how long[?] Who has access to it?” A few students did not grasp that the system used asynchronous AI remote invigilation instead of a human proctor, for example, “I did not find the experience very pleasant…the thought of an unknown person monitoring me was disconcerting” and “[the] camera made me feel uncomfortable because I couldn't see who was watching me.” Discontinue use of the system Some students (n = 17) did not agree with the concept of the system, did not like it or felt it did not suit the type of assessment they were completing and suggested it be discontinued. One student commented, “I think the anxiety behind being recorded isn’t something that can be fixed with remote invigilation and I believe that the tests should be undertaken in the given time for our allocated workshops.” Technical issues Students (n = 35) commented on technical issues such as system lag, problems with installing and downloading the software, glitches with uploading recordings and issues with hardware (camera and microphone). Some student responses highlighted that technical issues caused them to feel emotions of uneasiness, worry and stress, for example, “It wasn't working for me a few times which made my tests quite stressful as I would spend most of the time trying to get the video to work and less time preparing for the test.” Comments in this theme also inferred that students were concerned that CRIS would incorrectly flag their behaviours as cheating, due to the perceived (but not necessarily actual) technical issue they were experiencing: I had this issue of the tracking losing my eyes because of my display size and in the back of my mind I was thinking if it would alert my tutor of possible cheating, when I wasn't. Just the lack of understanding on that perspective made me very uneasy. Another student commented: I also got nervous when there was no highlighted box around my face, maybe my lighting wasn't 100% adequate, but if I moved my head slightly to write on my allowed blank piece of paper to calculate a sum, it lost recognition. I was somewhat worried that this would cause a breach of some sort! Australasian Journal of Educational Technology, 2022, 38(2). 62 The following quote describes a context where the themes of privacy, technical issues and emotional responses intersect: Due to privacy issues I did not want to install remotely invigilated software on my laptop. I still fully understand the need for CRIS, as some method is necessary to prevent cheating. I chose to use a … Library laptop, for these e-tests. I found the first e-test stressful, before I had the opportunity to start it, as the Library staff I spoke to (24 hours before the e-test deadline) were not in the least prepared, information-wise … I was eventually given a suitable laptop. When I tried to undertake the e-test, CRIS was not installed on the laptop and I was already stressed from previous events. Suggested UX improvements Students (n = 108) provided suggestions for improving CRIS including providing greater clarity on expectations of appropriate behaviour, simplifying the process, narrowing the scope of usage and improving the software with a dedicated application rather than a browser extension. Some student suggestions were not practical, such as allowing breaks during the assessment, as this would compromise assessment integrity. Students (n = 85) also made suggestions for topics that should be covered by the avatar and additional resources. Students (n = 27) commented on the efficacy of the CRIS avatar stating the avatar was satisfactory as is and did not need to cover any additional information. The topic that students (n =12) provided the most recommendations for was recording and uploading. Examples of suggestions include providing “more guidance as to when recording would start/stop” and “what to do if you need to stop the recording, or if something goes wrong with your assessment?” A recommendation for a more detailed troubleshooting section yielded nine responses. Clarity on expectations of appropriate behaviour including accessing notes and blank paper was reiterated from suggestions made in responses to Question 16 (Appendix), indicating that students consider this aspect important. Discussion The aim of this research was to determine how the system characteristics, users’ internal state and contextual factors influence the student UX of asynchronous AI remote invigilation to better understand how this experience can be improved. Understanding the UX during remote invigilation has become critically important given the rapid transition to online education and assessment that has occurred globally starting in 2020 due to the SARS-CoV-2 pandemic. To put this in context, our study began in 2017 with a sample size in the hundreds. Due to changes necessitated by the pandemic, in Semester 2 of 2020, we used remote invigilation for more than 16,000 centrally scheduled examination sittings. This number did not include school-scheduled mid-semester tests and low-stakes e-assessments that were rapidly converted to online formats during the year. In 2021, this use of online invigilation increased again, and these changes only represent that of one educational institution. Therefore, understanding how institutions can make this large-scale transition easier for students is paramount. To answer the research questions and align the results of this study with the wider scope of remote invigilation literature, the following discussion is broken down into examining system characteristics, which comprises of usefulness and ease of use; users’ internal state; and context and UX interactions. System characteristics Usefulness Previous research has indicated that two-thirds of students surveyed in a large study were not overly concerned about contract cheating from their peers (Bretag et al., 2018). Bretag et al. suggested that students in that study might not recognise the impacts that this could have on the reputation of their degree or the potential risks to public safety of unqualified graduates. In our research, two thirds of the 177 students (n = 118) indicated that they thought it was extremely to somewhat important for the university to have approaches such as remote invigilation to discourage cheating. Only 9% of the sample here reported it was not very important or not at all important. This suggests that students may view directly cheating on online tests differently to the way that they view contract cheating. Student perceptions were mostly positive for the efficacy of remote invigilation with 79% of students surveyed reporting that they believed remote invigilation was extremely to moderately effective for deterring cheating. This is comparable to results Australasian Journal of Educational Technology, 2022, 38(2). 63 reported by Milone et al. (2017), where most students (70.3%) felt similarly and agreed that “the use of invigilation reduces cheating and is fairer by keeping all students on a level playing field” (p. 113). Although it is outside of the research parameters set for this study, in practise, we have found this to be true. During 2020, we identified a large rise in the use of real-time examination help websites such as Chegg.com. Through targeted investigation of these misconduct cases, we found that units that used remote invigilation during their online examinations did not have reported incidents of student cheating by real- time help websites. This suggests that remote invigilation may be helpful with this form of e-cheating. In this study, students were asked about their attitudes towards remote invigilation after using the system. Approximately, 30% had become more positive, 21% had become more negative about the idea and half of students surveyed were neutral. These results are similar to attitudes towards remote invigilation reported in another study (Bedford et al., 2009). Similarly, Lilley et al. (2016) reported that 9 out of 10 students felt that remote invigilation should be used and extended to other units. Despite SARS-CoV-2 induced changes to assessment, some studies have found minimal impact on student test performance due to remote invigilation (Hall et al., 2021; Hope et al., 2021). Topps (2021) also found that the validity of test results was not compromised by remote invigilation. This type of information is important to convey to students to reassure them that online invigilation is useful and generally has little impact on test performance. Ease of use This research determined that familiarity with the system was a key component that reduced difficulty levels for students (Table 2). Students who found CRIS to be extremely easy to use after becoming familiar with the system had mostly become more positive towards remote invigilation (Figure 9). We argue that these results are largely due to the use of a video-based avatar that guided students through the set-up process in a stepwise fashion; therefore, this is recommended as one method to help students effectively learn to use a novel remote invigilation system. Also, providing formative practise with the remote invigilation system for students before summative assessment is essential in ensuring familiarity with any system. The broadly positive findings within the theme of familiarity are consistent with findings reported by Bedford et al. (2009), where usefulness and perceived ease of use were key factors influencing positive UX. This also bears similarity to Davis (1989), who asserted that perceived usefulness and ease of use are “fundamental determinants” in the acceptance and adoption of new technologies (p. 319). We found that approximately 6% of students did not have the required equipment to use the remote invigilation system. This is an area that requires policy and pragmatic considerations for rapid deployment of these types of technologies. Users’ internal state Overall, feelings towards the idea of being remotely invigilated improved after participating in the study. However, anxiety resulting from being recorded was reported as a recurrent theme in the qualitative responses. It also emerges as a key driver for some student suggestions to discontinue the use of the software. Students commented on the impact of being recorded on their concentration and performance. Others have reported similar findings wherein participants expressed concerns regarding safety and security, anxiety about being watched by a stranger, self-consciousness and apprehension about whether the invigilation service would protect their data and privacy as stipulated by laws (Eaton & Turner, 2020; Karim et al., 2014; Lilley et al., 2016; Richardson, 2020; Woldeab & Brothen, 2019). It was interesting to note that 18–24-year-old participants in this study were the least comfortable with remote invigilation; we suggest that this age group may be more keenly aware of their online privacy rights due to the rapid increase in what Dawson (2021b) calls surveillance culture within their schooling. Even though this research utilised asynchronous automatic remote invigilation facilitated by AI, some students still expressed discomfort with the idea of someone watching them. This theme supports the construct within the UX model that a user’s internal state is a key factor influencing user experience more broadly. Therefore, understanding how to make students feel more at ease is paramount to improving UX and the efficacy of the system as a whole. We argue that this can be accomplished through a multifaceted approach to providing comprehensive information, appropriate and timely guidance and required formative practise for students. In this case, a comprehensive help website that contains all information that students need for the remote invigilation system is provided on the main university website. Students also are provided with timely email, chat, telephone and walk-in service options with staff trained specifically to help with remote invigilation issues. This consistent university-wide approach to student support is essential Australasian Journal of Educational Technology, 2022, 38(2). 64 for students to feel comfortable with remote invigilation. It is possible that privately owned and managed invigilation software such as CRIS may evoke fewer privacy concerns for students compared to those run by external companies (Richardson, 2020); however, this is an area that would benefit from further research. Context and UX interactions In this study, contextual factors were found to intersect mostly with positive UX. These were largely based on convenience, effectiveness and ease of use. This concurs with findings reported by Milone et al. (2017), who surveyed undergraduate students who were able to choose between being remotely invigilated in real time or attending a testing centre to sit for an assessment. The majority of students (79%) who answered the post-exam survey had used remote invigilation, and most students (89%) expressed a positive attitude towards the system. Our research demonstrated that negative UX were generally caused by perceived technical problems or anxiety about being remotely invigilated. This is comparable to other findings where technical issues experienced by students negatively impacted the UX and satisfaction levels (Elsalem, 2020; Milone et al., 2017). There is emerging evidence to suggest that remote e-exams and online invigilation may be more stressful for students than face-to-face examinations; however, more research is needed in this area (Eaton & Turner, 2020; Elsalem et al., 2020). This emphasises the need to understand student anxiety triggers and potential technical problems and then have comprehensive support structures in place to ease negative emotional responses. We suggest that an effective approach to remote invigilation needs to achieve a balance between usefulness for cheating deterrence and detection and ease of use for de-stressing the process for students and educators. This entails that comprehensive policies are constructed to provide students and educators with information about (a) where and when remote invigilation should be used; (b) who is responsible for advising students of system requirements and for providing opportunities for system check tests before summative assessment; (c) assessment limitations; (d) requirements for identity verification; (e) processes for reviewing invigilation records; (f) alternative arrangements and exemption processes; and, (g) privacy and record management. We agree with Dawson’s (2021a) 10 practical suggestions for online invigilated exams and contend that humans should always be part of the decision-making process regarding any suspected academic integrity breach rather than relying on AI (Dawson, 2021b). We argue that this balance can be achieved with appropriate policy, guidance and support structures, which will facilitate familiarity between the student and the remote invigilation system. Conclusions This study employed a UX model to understand the student experience of an asynchronous remote invigilation system with AI. The first two components of the UX model are explored in detail; however, there was less opportunity to examine how context, assessment length and the physical environment where the student interacts with CRIS, influenced UX. This presents an opportunity for future work. It is important to note that over the 3-year research period, the system did improve (i.e., the addition of further support information, frequently asked questions) and more units commenced using the software. The research is limited by the use of a single, custom-developed asynchronous remote invigilation system used at one university. However, our results show a high level of consistency with other studies where issues of privacy and technical aspects were highlighted as being important to the UX, emotional reaction and acceptance of remote invigilation technology (Bedford et al., 2009; Eaton & Turner, 2020; Karim et al. 2014; Lilley et al. 2016; Milone et al. 2017). Perceived technical problems negatively influenced student emotions; therefore, prioritising strategies to address this may improve attitudes and experience towards remote invigilation. We suggest that comprehensive policies and effective communication is needed to ensure students understand remote invigilation processes, particularly with regard to privacy considerations. As most students in the study had not heard of and had no prior experience with remote invigilation, the concept of familiarity emerged as a significant factor that positively influenced perceived ease of use, comfort level and student UX in general. We anticipate that further practice will encourage familiarity and consequently more positive attitudes towards remote invigilation. Australasian Journal of Educational Technology, 2022, 38(2). 65 Declaration of interest This study was made possible through funding by Curtin University Innovation Grants through Curtin Learning and Teaching, Office of the Deputy Vice Chancellor Academic. The funding area was not involved in study design; in the collection, analysis and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. Some information presented in this paper has been adapted for use as part of Universities Australia Academic Integrity Best Practice Principles Case Study entitled Online Invigilation at Curtin University. At the time of writing, the authors have no conflicts of interest for the remote invigilation software in relation to commercialisation or payment for services. The authors of the paper are the developers of the software. Acknowledgements We would like to acknowledge the Engineering Institute of Technology, early adopters of this software, as well as the support of Professor Jon Yorke and Professor Jill Downie, who helped make this project possible. References Alessio, H., Malay, N., Maurer, K., Bailer, A., & Rubin, B. (2017). Examining the effect of proctoring on online test scores. Online Learning, 21(1). https://doi.org/10.24059/olj.v21i1.885 Atoum, Y., Chen, L., Liu, A., Hsu, S., & Liu, X. (2017). Automated online exam proctoring. IEEE Transactions on Multimedia, 19(7), 1609–1624. https://doi.org/10.1109/tmm.2017.2656064 Baird, M., Sefcik, L., Steyn, S., & Price, C. (2019). Learning analytics leading to remote invigilation for eTests: A case study. In D. Ifenthaler, D.-K. Mah, & J. Y.-K. Yau (Eds.), Utilizing learning analytics to support study success (pp. 295–312). Springer. https://doi.org/10.1007/978-3-319-64792-0_16 Bedford, W., Gregg, J., & Clinton, S. (2009). Implementing technology to prevent online cheating: A case study at a small southern regional university (SSRU). MERLOT Journal of Online Learning and Teaching, 5, 2, 230–238. https://jolt.merlot.org/vol5no2/gregg_0609.pdf Bedford, D. W., Gregg, J. R., & Clinton, M. S. (2011). Preventing online cheating with technology: A pilot study of remote proctor and an update of its use. Journal of Higher Education Theory and Practice, 11(2), 41–58. http://m.www.na-businesspress.com/JHETP/BedfordWeb.pdf Boyd Davis, A., Rand, R., & Seay, R. (2016). Remote proctoring: The effect of proctoring on grades. Advances in Accounting Education: Teaching and Curriculum Innovations, 18, 23– 50. https://doi.org/10.1108/S1085-462220160000018002 Braun, V., & Clarke, V. (2013). Successful qualitative research: A practical guide for beginners (1st ed.). Sage. Bretag, T., Harper, R., Burton, M., Ellis, C., Newton, P., Saddiqui, S., Rozenberg, P., & van Haeringen, K. (2018). Contract cheating: A survey of Australian university students, Studies in Higher Education, 1–20. https://doi.org/10.1080/03075079.2018.1462788 Brooks, J., & King, N. (2012, April 18–20). Qualitative psychology in the real world: The utility of template analysis [Paper presentation]. The British Psychological Society Annual Conference, London. Cameron, N. (2018, June 6). What Open Universities Australia did to enact a complete business and technology transformation. CMO. https://www.cmo.com.au/article/641962/what-open-universities- australia-did-enact-complete-business-technology-transformation/ Datig, I. (2015). Walking in your users’ shoes: An introduction to user experience research as a tool for developing user-centered libraries. College & Undergraduate Libraries, 22(3–4), 234–246. https://doi.org/10.1080/10691316.2015.1060143 Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008 Dawson, P. (2021a). Strategies for using online invigilated exams. Online learning good practice TEQSA. Tertiary Education Quality and Standards Agency. https://www.teqsa.gov.au/sites/default/files/strategies-for-using-online-invigilated-exams.pdf Dawson, P. (2021b). Defending assessment security in a digital world: Preventing e-cheating and supporting academic integrity in higher education. Taylor & Francis. https://doi.org/10.4324/9780429324178 https://doi.org/10.24059/olj.v21i1.885 https://doi.org/10.1109/tmm.2017.2656064 https://doi.org/10.1007/978-3-319-64792-0_16 https://jolt.merlot.org/vol5no2/gregg_0609.pdf http://m.www.na-businesspress.com/JHETP/BedfordWeb.pdf https://doi.org/10.1108/S1085-462220160000018002 https://doi.org/10.1080/03075079.2018.1462788 https://www.cmo.com.au/article/641962/what-open-universities-australia-did-enact-complete-business-technology-transformation/ https://www.cmo.com.au/article/641962/what-open-universities-australia-did-enact-complete-business-technology-transformation/ https://doi.org/10.1080/10691316.2015.1060143 https://doi.org/10.2307/249008 https://www.teqsa.gov.au/sites/default/files/strategies-for-using-online-invigilated-exams.pdf https://doi.org/10.4324/9780429324178 Australasian Journal of Educational Technology, 2022, 38(2). 66 Eaton, S. E., & Turner, K. L. (2020). Exploring academic integrity and mental health during COVID-19: Rapid review. Journal of Contemporary Education Theory & Research, 4(2) 35–41. https://doi.org/10.5281/zenodo.4256825 Elsalem, L., Al-Azzam, N., Jum’ah, A. A., Obeidat, N., Sindiani, A. M., & Kheirallah, K. A. (2020). Stress and behavioral changes with remote E-exams during the Covid-19 pandemic: A cross-sectional study among undergraduates of medical sciences. Annals of Medicine and Surgery, 60, 271–279. https://doi.org/10.1016/j.amsu.2020.10.058 Fask, A., Englander, F., & Wang, Z. (2014). Do online exams facilitate cheating? An experiment designed to separate possible cheating from the effect of the online test taking environment. Journal of Academic Ethics, 12(2), 101–112. https://doi.org/10.1007/s10805-014-9207-1 Fayomi, O., Amodu, L., Ayo, C., Idowu, O., & Iyoha, F. (2015) E-invigilation: Panacea to examination malpractice in Nigeria. In L. G. Chova, A. López Martínez, & I. Candel Torres (Eds.), Proceedings of the 8th International Conference of Education, Research and Innovation (pp. 2849–2858). International Academy of Technology, Education and Development. https://library.iated.org/view/FAYOMI2015EIN Fendler, R. J., & Godbey, J. M. (2016). Cheaters should never win: Eliminating the benefits of cheating. Journal of Academic Ethics, 14(1), 71–85. https://doi.org/10.1007/s10805-015-9240-8 Fenu, G., Marras, M., & Boratto, L. (2018). A multi-biometric system for continuous student authentication in e-learning platforms. Pattern Recognition Letters, 113, 83–92. https://doi.org/10.1016/j.patrec.2017.03.027 Gilbert, G. E., & Prion, S. (2016). Making sense of methods and measurement: The chi-square test. Clinical Simulation in Nursing, 12(5), 145–146. https://doi.org/10.1016/j.ecns.2015.12.013 Gómez-López, P., Simarro, F., & Bonal, M. (2019). Analysing the UX scope through its definitions. In J. Abascal, M. Arrue, N. Garay-Vitoria, & M. D. Lozano (Eds.), Interacción '19—Proceedings of the XX International Conference on Human Computer Interaction (pp. 1–4). Association for Computing Machinery. https://doi.org/10.1145/3335595.3335648 Greenland, S. J., & Moore, C. (2014). Patterns of student enrolment and attrition in Australian open access online education: A preliminary case study. Open Praxis, 6(1), 45–54. https://doi.org/10.5944/openpraxis.6.1.95 Hall, E. A., Spivey, C., Kendrex, H., & Havrda, D. E. (2021). Effects of remote proctoring on composite exam performance among Doctor of Pharmacy students. American Journal of Pharmaceutical Education, 85(8) 824–828. https://doi.org/10.5688/ajpe8410 Hassenzahl, M., & Tractinsky, N. (2006). User experience: A research agenda. Behaviour & Information Technology, 25(2), 91–97. https://doi.org/10.1080/01449290500330331 Hope, D., Davids, V., Bollington, L., & Maxwell, S. (2021). Candidates undertaking (invigilated) assessment online show no differences in performance compared to those undertaking assessment offline. Medical Teacher, 43(6), 1–14. https://doi.org/10.1080/0142159X.2021.1887467 Hoskin, T. (2012). Parametric and nonparametric: Demystifying the terms. Mayo Clinic. https://www.mayo.edu/research/documents/parametric-and-nonparametric-demystifying-the- terms/doc-20408960 Hylton, K., Levy, Y., & Dringus, L.P. (2016). Utilizing webcam-based proctoring to deter misconduct in online exams. Computers & Education, 92–93, 53–63. https://doi.org/10.1016/j.compedu.2015.10.002 Jefferies, A., Barton, K., Meere, J., Pyper, A., Yip, P. Y., & Peramunagama, S. (2017). Trialling online proctoring for e-assessments: Early outcomes from the Erasmus+ OP4RE Project. In A. Mesquita & P. Peres (Eds.), Proceedings of the 16th European Conference on e-Learning (pp. 221–228). Academic Conference Publishing International. Karim, M. N., Kaminsky, S. E., & Behrend, T. S. (2014). Cheating, reactions, and performance in remotely proctored testing: An exploratory experimental study. Journal of Business and Psychology, 29(4), 555–572. https://doi.org/10.1007/s10869-014-9343-z King, N. (2012). Doing template analysis., In G. Symon & C. Cassell (Eds.), Qualitative organizational research: Core methods and current challenges (pp. 77–101). Sage. https://doi.org/10.4135/9781526435620.n24 Lilley, M., Meere, J., & Barker, T. (2016). Remote live invigilation: A pilot study. Journal of Interactive Media in Education, 2016(1), 1–5. https://doi.org/10.5334/jime.408 McCabe, D. L., Butterfield, K. D., & Treviño, L. K. (2012). Cheating in college: Why students do it and what educators can do about it. The Johns Hopkins University Press. McCrum-Gardner, E. (2008). Which is the correct statistical test to use? British Journal of Oral and Maxillofacial Surgery, 46(1), 38–41. https://doi.org/10.1016/j.bjoms.2007.09.002 https://doi.org/10.5281/zenodo.4256825 https://doi.org/10.1016/j.amsu.2020.10.058 https://doi.org/10.1007/s10805-014-9207-1 https://library.iated.org/view/FAYOMI2015EIN https://doi.org/10.1007/s10805-015-9240-8 https://doi.org/10.1016/j.patrec.2017.03.027 https://doi.org/10.1016/j.ecns.2015.12.013 https://doi.org/10.1145/3335595.3335648 https://doi.org/10.5944/openpraxis.6.1.95 https://doi.org/10.5688/ajpe8410 https://doi.org/10.1080/01449290500330331 https://doi.org/10.1080/0142159X.2021.1887467 https://www.mayo.edu/research/documents/parametric-and-nonparametric-demystifying-the-terms/doc-20408960 https://www.mayo.edu/research/documents/parametric-and-nonparametric-demystifying-the-terms/doc-20408960 https://doi.org/10.1016/j.compedu.2015.10.002 https://doi.org/10.1007/s10869-014-9343-z https://doi.org/10.4135/9781526435620.n24 https://doi.org/10.5334/jime.408 https://doi.org/10.1016/j.bjoms.2007.09.002 Australasian Journal of Educational Technology, 2022, 38(2). 67 McHugh, M. L. (2013). The chi-square test of independence. Biochemia Medica, 23(2), 143–149. https://doi.org/10.11613/BM.2013.018 Mellar, H., Peytcheva-Forsyth, R., Kocdar, S., Karadeniz, A., & Yovkova, B. (2018) Addressing cheating in e-assessment using student authentication and authorship checking systems: teachers’ perspectives. International Journal for Educational Integrity, 14, Article 2. https://doi.org/10.1007/s40979-018- 0025-x Milone, A. S., Cortese, A. M., Balestrieri, R. L., & Pittenger, A. L. (2017). The impact of proctored online exams on the educational experience. Currents in Pharmacy Teaching and Learning, 9(1), 108–114. https://doi.org/10.1016/j.cptl.2016.08.037 Morse, J. M. (2004). Purposive sampling. In M. S. Lewis-Beck, A. Bryman, & T. Futing Liao (Eds.), The SAGE Encyclopedia of social science research methods (pp. 884–885). Sage. https://doi.org/10.4135/9781412950589 Parnther, C. (2020). Academic misconduct in higher education: A comprehensive review. Journal of Higher Education Policy and Leadership Studies, 1(1), 25–45. https://doi.org/10.29252/johepal.1.1.25 Pereira, S. M. C., & Leslie, G. (2010). Testing differences between two samples of continuous data. Australian Critical Care, 23(3), 160–166. https://doi.org/10.1016/j.aucc.2010.06.002 Richardson, H. (2020, July 16). University students at UQ raise concerns about online exam monitoring service ProctorU. ABC News. https://www.abc.net.au/news/2020-07-16/uq-privacy-issues-concerns- student-exams/12454964 Rios, J. A., & Liu, O. L. (2017). Online proctored versus unproctored low-stakes internet test administration: Is there differential test-taking behavior and performance? American Journal of Distance Education, 31(4), 226–241. https://doi.org/10.1080/08923647.2017.1258628 Sauer, J., Sonderegger, A., & Schmutz, S. (2020). Usability, user experience and accessibility: Towards an integrative model. Ergonomics, 63(10), 1207–1220. https://doi.org/10.1080/00140139.2020.1774080 Shorten, A., & Smith, J. (2017). Mixed methods research: expanding the evidence base. Evidence-Based Nursing, 20(3), 74–75. https://doi.org/10.1136/eb-2017-102699 Simonsen, J. G. (2018). User experience. In N. Kent & J. Kirakowski (Eds.), The Wiley Handbook of human computer interaction set (pp. 198–211) John Wiley and Sons Ltd. Topps, M. (2021). Remote proctoring provides candidates a safe way to continue the path to licensure. Canadian Medical Education Journal, 12(1), 113–114. https://doi.org/10.36834/cmej.71437 Ullah, A., Xiao, H., & Barker, T. (2016). A classification of threats to remote online examinations. In Proceedings of the IEEE 7th Annual Information Technology, Electronics and Mobile Communication Conference (pp. 1–7). IEEE. https://doi.org/10.1109/IEMCON.2016.7746085 Varpio, L., Ajjawi, R., Monrouxe, L. V., O'Brien, B. C., & Rees, C. E. (2017). Shedding the cobra effect: problematising thematic emergence, triangulation, saturation and member checking. Medical Education, 51(1), 40–50. https://doi.org/10.1111/medu.13124 Vegendla, A., & Sindre, G. (2019). Mitigation of cheating in online exams: Strengths and limitations of biometric authentication In A. Kumar (Ed.), Biometric authentication in online learning environments (pp. 47–68). IGI Global. https://doi.org/10.4018/978-1-5225-7724-9.ch003 Woldeab, D., & Brothen, T. (2019). 21st century assessment: Online proctoring, test anxiety, and student performance. International Journal of E-Learning & Distance Education, 34(1) 1–10. http://www.ijede.ca/index.php/jde/article/view/1106/1727 Woldeab, D., Lindsay, T., & Brothen, T. (2017). Under the watchful eye of online proctoring. In I. D. Alexander & R. K. Poch (Eds.), Innovative learning and teaching: Experiments across the disciplines (pp. 152–166). University of Minnesota Libraries Publishing. https://open.lib.umn.edu/innovativeteaching/chapter/under-the-watchful-eye-of-online-proctoring/ Woolson, R. F. (2008) Wilcoxon signed-rank test. In R. D’Agostino, L. Sullivan, & J. Massaro (Eds.), Wiley Encyclopedia of clinical trials (pp 1–3). Wiley. https://doi.org/10.1002/9780471462422.eoct979 https://doi.org/10.11613/BM.2013.018 https://doi.org/10.1007/s40979-018-0025-x https://doi.org/10.1007/s40979-018-0025-x https://doi.org/10.1016/j.cptl.2016.08.037 https://doi.org/10.4135/9781412950589 https://doi.org/10.1016/j.aucc.2010.06.002 https://www.abc.net.au/news/2020-07-16/uq-privacy-issues-concerns-student-exams/12454964 https://www.abc.net.au/news/2020-07-16/uq-privacy-issues-concerns-student-exams/12454964 https://doi.org/10.1080/08923647.2017.1258628 https://doi.org/10.1080/00140139.2020.1774080 https://doi.org/10.1136/eb-2017-102699 https://doi.org/10.36834/cmej.71437 https://doi.org/10.1109/IEMCON.2016.7746085 https://doi.org/10.1111/medu.13124 https://doi.org/10.4018/978-1-5225-7724-9.ch003 http://www.ijede.ca/index.php/jde/article/view/1106/1727 https://open.lib.umn.edu/innovativeteaching/chapter/under-the-watchful-eye-of-online-proctoring/ https://doi.org/10.1002/9780471462422.eoct979 Australasian Journal of Educational Technology, 2022, 38(2). 68 Corresponding author: Lesley T. Sefcik, Lesley.Sefcik@curtin.edu.au Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY- NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Sefcik, L. T., Veeran-Colton, T., Baird, M., Price, C., & Steyn, S. (2022). An examination of student user experience (UX) and perceptions of remote invigilation during online assessment. Australasian Journal of Educational Technology, 38(2), 49-69. https://doi.org/10.14742/ajet.6871 mailto:Lesley.Sefcik@curtin.edu.au https://creativecommons.org/licenses/by-nc-nd/4.0/ https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.6871 Australasian Journal of Educational Technology, 2022, 38(2). 69 Appendix Questions grouped into themes: A. Perceptions of cheating B. Familiarity with remote invigilation C. User experience and efficacy D. Attitude Theme No. Question N/A 1 Research consent N/A 2 Please select your age group N/A 3 Please select the gender you identify as B 4 Before being part of this study, had you heard of remote invigilation (remote online invigilation)? B 5 Before being part of this unit, how many times had you used remote invigilation in your studies? B 6 Before you participated in the project, please describe how you felt about the idea of being remotely invigilated. B 7 After you participated in the project, please describe your level of comfort in relation to being remotely invigilated. C 8 You were asked to conduct a remote invigilation equipment check before undertaking any online tests. Which of the following statements best matches your experience? C 9 Did you have the necessary technology required for remote invigilation during your online tests? (video camera, microphone, internet connection) B 10 When you first started using the remote invigilation software, please describe how easy or difficult it was to use. B 11 After you became familiar with the remote invigilation software, please describe how easy or difficult it was to use. B 12 Please indicate the amount of time it took to initially get started with the remote invigilation software (including setting up the software and doing the equipment check). B 13 Did you get quicker at using the remote invigilation software over the course of the unit? D 14 How has your attitude towards remote invigilation changed over the course of the unit? A 15 How effective do you think remote invigilation is for discouraging cheating? C 16 What could be done to improve your experience of being remotely invigilated during your online tests? N/A 17 No question N/A 18 No question A 19 How important it is to you that Curtin has approaches (like remote invigilation) in place to discourage cheating? A 20 Considering the likelihood and impact of potential cheating in the assessments in this unit, do you feel that the assessment process (including the use of remote invigilation) are appropriate? A 21 How clear were the expectations for appropriate student behaviour (while test taking with remote invigilation)? C 22 Did you access the additional linked resources provided on the CRIS landing page? C 23 Which resources did you access? Please select all that apply. C 24 How effective were the step by step ‘CRIS Set Up Instructions’ for helping you correctly set up the system? C 25 How effective was the FAQs (Frequently Asked Questions) document to answer your questions about CRIS? C 26 How effective was the 'What is This?' information sheet to understand how CRIS works? C 27 How effective was the CRIS Avatar for helping you correctly use the system? C 28 What questions/topics would you like to have been covered by the avatar or in the resources? Introduction Remote invigilation systems Real-time remote invigilation Asynchronous remote invigilation Study objectives Materials and methods System overview Analytical framework Data collection and analysis e-survey Data analysis Chi-square test for independence Wilcoxon signed-rank test Qualitative responses Results Demographics Familiarity with remote invigilation Perceptions of cheating UX and efficacy Attitude Comfort Qualitative responses Positive experiences and opinions Negative experiences and opinions Discontinue use of the system Technical issues Suggested UX improvements System characteristics Usefulness Ease of use Users’ internal state Context and UX interactions Conclusions Declaration of interest Acknowledgements References Appendix