Paper title (Paper Title style) Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 1 Factor structures associated with online student engagement in campus-based blended and online distance education settings Graeme Pye Deakin University, Geelong, Australia graeme.pye@deakin.edu.au Dale Holt Deakin University, Geelong, Australia Scott Salzman Deakin University, Warrnambool, Australia Abstract This research is strictly grounded in the work undertaken by Coates regarding traditional campus-based settings of student engagement and was applied to the online settings positioned within the domain settings of blended online learning design and practice in an Australian higher education business context. Utilising an online student survey instrument, across two consecutive academic years, undergraduate commerce students were invited to reflect upon their learning engagement experience through the lens of a common learning management system (LMS) a resource accessible to both campus-based and off-campus student cohorts. Subsequent analysis of the research reconfirms the existence of student engagement constructs of Coates in the blended online setting, but also unexpectedly revealed a previously unknown construct relating to Assessment. This new student engagement construct, Assessment, is identified as being a significant motivational factor relevant to student engagement in the context of the blended online learning environment of this higher education business undergraduate commerce course and is the focus of this exposition. Keywords: Student engagement, learning management systems, blended learning, online education, distance education. 1 Introduction Online learning is now central to many variant educational environments in global higher education. Online learning supports campus-based blended education by providing access to the online learning environment. Online also delivers distance education resources to students unable to attend campus and is a major delivery mode integrated with the development of global Massive Open Online Course (MOOC) platforms. It also bridges environments designed to meet the needs of both campus-based and distance education students studying the same subjects and programs in multi-campus and multi-modal settings. Whatever the concerns and criticisms levelled against the institution-wide selection and deployment of Learning Management Systems with their set-piece forms and functions, they have become the mainstay platform for developing, delivering, and administering the online dimension of many variant higher educational environments. The generic term ‘Learning Management System’ (LMS) has been used instead of the institution’s own branding of its online learning system. There is no suggestion that the LMS alone is somehow automatically undertaking the various learning activities and resource delivery covered here. Therefore, it is presumed that Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 2 academics design and utilise the LMS with students to create a value-added learning experience and hence, human agency animates the system. The motivation for this research is to revisit the specific key constructs of Coates’s (2006) student engagement research, which was broadened via the extensive Australasian Survey of Student Engagement (AUSSE) survey, the focus of Coates’s (2010) research. The intent here is to investigate if these specific student engagement constructs are prevalent in online learning supports and campus-based blended education environments within the contemporary online learning environment. The findings may further inform the online learning environment design of an institutional LMS, to ensure that the student engagement constructs are active and prevalent as design features for the contemporary online learning environment. This research builds upon Coates’s (2006) work on the validity of identified student engagement constructs that Coates’s research recognised. To this end, this research project undertook a major surveying of individual student experiences of online engagement. The online student engagement data collected offered in multiple periods has been statistically analysed to test whether the constructs initially identified by Coates’s (2006) are still apparent, valid, and of use for ongoing survey investigations. Since Coates’s original research, it is now timely to undertake this independent analysis of online student engagement constructs initially utilising those identified by Coates (2006). Specifically, this research was undertaken in the context of student engagement research around the inclusive on-campus learning, blended with online distance education, in the major field of higher business commerce education, and by extension, associated major professional fields of study. The structure of this document begins with the discussion relating to the review of the underpinning and relevant student engagement literature, including the research methodology as applied. Next, the data collection approach is described, and the subsequent data analysis is undertaken utilising statistical analysis of relevant data to tease out the findings and their relevance. Finally, the latent student engagement constructs that became apparent during the analysis are described, which is then followed by conclusions and a scope for future research. 2 Literature Review This research builds upon the previous research of Pye et al. (2015, 2018) and is allied with the selection and use of a LMS, as the enduring interest in investigating the student experience in higher education. One rich vein of research in this domain has related to theorising, conceptualising, and enhancing the practice of a phenomenon named ‘student engagement’ (Kuh, 2003; Kuh, Kinzie, Schuh, Whitt & Associates, 2005). Student engagement relates to the time and effort students devote to educationally purposeful activities pursuant to achieving valued learning outcomes (Coates & McCormick, 2014). Student engagement integrates significant research based on the student experience in higher education settings, in key domains covering quality of effort (Pace, 1979; 1995), principles for good practice in undergraduate education (Chickering & Gamson, 1987), student learning involvement principles (Astin, 1985), and student integration into university life and persistence in study (Tinto 1993; 1997). Student engagement theory and policy development manifested itself in the National Survey of Student Engagement (NSSE) in the United States in the late 1990s, and was then adopted in Canada, Australia, New Zealand, South Africa, China, Ireland (Coates & Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 3 McCormick, 2014) and the United Kingdom (Howson & Buckley, 2017), as a major international initiative. Student engagement is a high-potential vehicle for assuring and improving the quality of higher education at the sectoral and institutional levels, was implemented through the development of the Australasian Survey of Student Engagement (AUSSE) in the Australian and New Zealand context (Coates, 2010). This research draws upon, recognises, reinforces, and extends previous learning design theories and practices examining various dimensions of interactivity in enhancing quality online learning environments (see, for example, Bonk & Zhang, 2008; Garrison and Anderson, 2003; Salmon, 2011). Coates (2006) work laid the theoretical, conceptual, and empirical foundation for the development and implementation of AUSSE. His empirically based, large-scale survey approach added a new dimension, illuminating the student experience in contemporary educational settings, both in relation to online learning technologies used in campus settings and student engagement in general, irrespective of the mode of course delivery. Given the growing influence of online learning on the campus/residential student learning experience, Coates’s (2006) expression of six student engagement constructs to encompass fundamental on-campus student engagement. Are applied here to extend understanding and the implications of how ‘online’, as enabled through the mainstream use of LMSs as it applies to (re)shaping the educational experience of students. This research seeks to take the six-student engagement constructs that Coates (2006) identified in campus-based student learning settings and investigate whether these same student engagement constructs are apparent within the blended learning settings of a higher education business discipline setting, utilising a contemporary online and on-campus combined learning delivery environment. This research study positions and opens a new domain of specialised study of student engagement in online and blended learning environment within the discipline of business education. 2.1 Student Engagement The development and validation of Coates’s (2006) student engagement constructs formed the foundation of his student survey instrument to elicit a structured conversation between learners and universities alike. Coates developed a structured, methodical approach for deriving the qualities of campus-based and online engagement; and then validated these as constructs through the operationalisation of a student engagement questionnaire. As outlined by Coates (2006, pp.71-80), ‘the student engagement instrument development began with the framing of constructs, scales and items found in the US NESSE, and was shaped by various theoretical considerations.’ This framework of constructs and items was adjusted to include the findings of major online learning research. Semi-structured interviews with students were conducted to explore student experiences engaging with online learning systems. Students were sampled from different Australian universities and fields of study and the analysis of the student interviews helped in the drafting of the scales and items to measure key qualities of online and campus student engagement. Subject and technical experts reviewed draft items and scales, and this provided the basis for developing the actual student engagement questionnaire. The draft instrument was later reviewed by a student focus group who were invited to complete the survey and provide feedback on its meaning and comprehensibility. The survey instrument was then trialled with students from several universities. It constituted a landmark study of student engagement in online and blended learning settings in Australian Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 4 higher education which had not been previously attempted, or subsequently replicated or extended in the Australian higher education sector. 2.2 Research Modelling Approach Coates (2006) used the statistical technique of congeneric measurement modelling to ascertain the identified qualities as being reflected in the validity of the generated latent online student engagement constructs, their sets of associated survey items and in calibrating the scales. The technique is described as providing ‘…a means of determining the strength of relationship between each item and the targeted latent scale, the extent to which the items from a common scale, and the existence of off-dimensional cross-loadings between items’ (Coates, 2006, p.72). The six online student engagement scales used by Coates in his survey instrument covered: 1. online active learning; 2. online social interaction; 3. online collaboration; 4. online teaching; 5. online academic relevance, and 6. online contact with staff. These were seen as relatively distinct qualities and properties of the online student experience. Therefore, it is important to investigate this knowledge gap as there is little research that elucidates the motivational factors of student engagement. This research utilises Coates’s (2006) student engagements constructs to investigate if they are evident in the online learning supports and campus-based blended education environment within a tertiary business education setting. 3 Research Design The development of the current student engagement survey instrument by Pye et al. (2015) selected from Coates’s (2006) work, the six online constructs for the study of student engagement in online and blended learning settings in commerce business education: 1. Online active learning: key ways in which students use online systems to enhance learning. 2. Online social interaction: use of online learning systems to support general forms of social interaction. 3. Online collaboration: students’ use of online systems to work collaboratively with peers. 4. Online teaching: whether teachers use online learning systems to promote effective learning. 5. Online academic relevance: use of online systems to enhance the relevance and context of study. 6. Online contact with staff: extent and ease of use of online systems by students in contacting staff. The items for each construct were the same as those used by Coates (2006) for each of the adopted scales. These items for each construct are listed below in Table 1. The student engagement survey used these six key latent constructs identified by Coates (2006), and the associated activities indicating quality and quantity of time and effort related to each. Thus, previously established, and validated items were chosen to measure each construct. One further (a seventh) student engagement construct identified by Pye et al. (2015) was added which related student engagement to various activities associated with assessment. Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 5 3.1 The Assessment Construct Assessment redesign has been a cornerstone of major institutional curriculum review and renewal projects in Australian higher education (see Lawson, 2015; Oliver, 2013). The strategic importance of relevant, innovative, and efficient assessment design and feedback approaches has been a major focus of higher education research and development (Boud, 2010; Nicol, 2010; Boud & Molloy, 2013; Henderson, Boud, Molloy, et al., 2016; Small & Attree, 2016; Winstone, Nash, Rowntree & Parker, 2017). The domain of online assessment is therefore considered as a potentially significant student engagement construct by this investigation. In the 2014 survey design, inter alia, items were included around student engagement with online assessment feedback: ‘Online assignment feedback helped me improve my understanding of the topic’; and ‘Online assignment feedback helped me prepare for subsequent work’. In recognition of the specific importance of online assessment, feedback rubrics in the digital learning sphere, the two 2014 survey items were refined into the following three, in a revised version of the 2015 survey: • ‘Access to the online assessment rubric before the assignment submission deadline, helped me with my assignment task’. • ‘After my assignment was marked, the online assessment Rubric clearly explained the marks I had received’. • ‘After my assignment was marked, the online assessment Rubric feedback comments helped me to improve my learning’. The inclusion of these refined items around online assessment is consistent with recent work emphasising the importance of well-designed assessment rubrics for advancing student learning (Parkin, Hepplestone, Holden, Irwin & Thorpe, 2012; Jonsson, 2014; Mohoney, 2016; Dawson, 2017). The best design of online assessment rubrics has come into sharper focus in fields like business education, which attract large and diverse student populations in Australian higher education (Atkinson & Lim, 2013). It is surmised that Coates (2006) did not focus on online assessment because of the technical limitations of the associated LMS assessment functions, given the time his research was undertaken in the mid-2000s. The LMS has now negotiated their experimentation and passage into mainstream use, with the last ten years seeing considerable maturation in the effective use of various LMS features and assessment functions. One feature, the online assessment rubric, was not a feature available in early LMSs. However, Palmer and Holt (2009; 2010; 2012 and 2014) reported the maturation in the importance and satisfaction attributed to different LMS functions relating to the one original unchanged system, from both teacher and student perspectives. Their historic findings (Palmer & Holt, 2014) in using a large, repeated cross- sectional and quantitative survey spanning the period 2004-2011, reported that online assessment was still a major area requiring greater improvement in the student learning experience, an area now addressed by online assessment rubrics. Additionally, Pye et al. (2018), have reported significant variations in online student engagement in contemporary blended and distance education environments by gender, age, and cultural background. These substantive findings aside, it is timely to return to a statistical analysis of the robustness of Coates’s (2006) original work, along with the addition of Assessment to the six original online student engagement constructs. Such an analysis of the now seven student engagement constructs would suggest the potential to extend the Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 6 application of student engagement into the contemporary world of higher education teaching and learning practices, academic policy development and research. 3.2 Research Limitations The design of this online student engagement survey encompasses both satisfaction and frequency of use approaches to understanding student-learning experiences in the online and blended domain of business commerce education. The hybrid design was adjudged as the best approach for investigating student engagement at the micro unit/subject level of the business commerce student experience. While not germane to this meta-analysis, the survey has drawn upon the online student engagement constructs and their descriptors, and associated items as reported by Coates (2006). The item response scales used levels of satisfaction rather than frequency of usage. This avoids the use of uniform frequency of use response scales that can lead to multiple student interpretations unless such anchor points clearly specify the level of usage in question. Uniform response scales across a broad range of constructs and items can create syntactical problems between response scale descriptor, and item wording. The interpretation issue was examined by Howson and Buckley (2017) in the development of the UK national student engagement survey where student responses to ‘how often/to what extent’ survey items can be seen to depend on varying student expectations and on the priority and regularity of the use of the activities in question. Frequency of use positively correlates with student learning value and satisfaction; higher levels of satisfaction can also positively relate to higher levels of usage. This matter requires further methodological consideration relating to online student engagement, which is outside the scope of this paper. Suffice to say that the online and blended student experience across all three dimensions of importance, satisfaction, and frequency of use, are all of vital significance. In addition, the observed advent of learning analytics and associated applications can now make it less necessary to rely on students’ self-reported usage of key learning system features and activities, at least to the extent students are learning online. 4 Data Collection and Analysis Closed-ended questions allowed respondents to express their level of agreement with each of the items associated with the six key latent constructs proposed by Coates, as well as the additional proposed student engagement construct of Assessment from Pye et al. (2015). Students were asked to rank their level of agreement with each of the items using a Likert scale represented using 2 ‘strongly disagree’ through to 6, ‘strongly agree’”. For all responses sought, participants were given the option to indicate that a response was 1, ‘not applicable’. For modelling and summary purposes, all ‘not applicable’ responses were removed, and all other ordinal responses were rescaled from 1, ‘strongly disagree’ through to 5, ‘strongly agree’. Data collection was undertaken anonymously using self-selection sampling via an online survey of student participants utilising the online survey tool ‘Opinio’ administered through the Strategic Intelligence and Planning Unit at Deakin University. Data collection occurred at the end of each teaching period during 2014 and 2015 across commerce subjects in accounting and information systems. The research project and survey instrument received Ethics approval from the Faculty of Business and Law’s Ethics Committee. The total number of responses was 569 out of a total population of 7353 surveyed, equating to a 7.74% overall response rate. There is no shortage of recommendations regarding the required sample size to conduct factor analysis, ranging from as few as three, to as many as 10 times the number of variables in the Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 7 data set (MacCallum et al., 1999). Our number of observations to variables ratio is 19 observations per variable. The limitations associated with self-selected sampling methods are noted. However, as this exploratory research is conducted within one educational institution only, and as the investigation attempts to consider online learning at this institution, a self- selection sampling method provides the most efficacious outcome. A summary of the constructs, their items and their mean score is presented in Table 1 and IBM SPSS ver. 22.0 was employed for the factor analysis modelling. Latent Constructs Questions No’s Mean Std. Dev Active Learning I used online materials to improve my learning 539 4.19 0.81 I used online material to make Classroom learning more meaningful 511 3.91 0.95 I identified expected academic standards using the [LMS] unit site 537 4.00 0.91 I found that online materials challenged me to learn 539 3.47 1.09 Social Interaction Teaching staff participated in online discussions 508 4.09 0.85 I found it easy to explain my ideas in online discussions 445 3.60 1.05 I had helpful online discussions with other students 420 3.44 1.15 I met new people when using the [LMS] 424 2.69 1.26 Collaboration I used the [LMS] with other students around the campus (on-campus students only) 378 3.45 1.11 I used the [LMS] to work with other students outside the class (on-campus students only) 374 3.11 1.23 I used the [LMS] to do academic work with other students 450 3.24 1.22 I used the [LMS] to communicate with other students 464 3.21 1.22 Teaching Staff used [LMS] to clarify what was required to do well 503 4.04 0.87 Staff used {LMS] in ways that improved the overall teaching 500 3.97 0.88 Academic staff made an effort to communicate with students online 501 4.20 0.78 Academic staff used [LMS] to provide students with extra assistance 500 4.05 0.89 Teaching staff used [LMS] to tailor activities for students 494 3.63 1.04 Teaching staff used [LMS] to discuss interesting issues 494 3.60 1.05 Assessment Assignments were easy to locate online 499 4.26 0.83 Tests were easy to locate online 468 4.19 0.89 Assignments were easy to submit online 499 4.23 0.90 Online assignment submission gave me more time to complete assessable work 497 4.13 0.95 Access to the online assessment rubric before the assignment submission deadline, helped me with my assignment task 343 4.21 0.86 After my assignment was marked, the online assessment rubric clearly explained the marks I had received 424 3.81 1.08 Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 8 After my assignment was marked, the online assessment rubric feedback comment helped me to improve my learning 421 3.73 1.11 Relevance Using [LMS] made my study seem more relevant 494 3.75 1.01 Using [LMS] made me feel part of the University 495 3.75 1.13 Using online materials helped me put my study in real- world contexts 494 3.61 1.09 Contact with Staff I used [LMS] to contact academic staff 435 3.76 1.09 I found it easy to communicate with staff online 432 3.83 0.99 I had individual contact with academic staff online 391 3.54 1.19 Table 1. Descriptive Statistics for All Items of Each Latent Construct Note: Assessment item 5 (italicised in the table above) was as mentioned, added to teaching period one in the 2015 iteration of the survey and is not included in subsequent analysis. All items were evaluated for reliability and internal consistency before investigating factor structures. It was necessary to assess the internal consistency of the factors to identify whether or not the proposed items were all measuring the same underlying construct (Pallant, 2005). 4.1 Data Consistency Analysis Cronbach’s alpha was used to determine if the assumption of reliability (internal consistency) had been met. Cronbach’s alpha was calculated separately for the items of each construct. Consideration of consistent and reliable variance was then considered using an estimate of the combined composite score for all items associated with each construct. Table 2 presents the results of the test for reliability using Cronbach’s alpha. Construct Cronbach’s Alpha Cronbach’s Alpha Based on Standardised Items Number of Items Active Learning 0.789 0.796 4 Social Interaction 0.791 0.793 4 Collaboration 0.868 0.868 4 Teaching 0.881 0.884 6 Assessment 0.850 0.856 6 Relevance 0.874 0.875 3 Contact with Staff 0.852 0.854 3 Table 2. Reliability Statistics The assumption of reliability appears to have been met in Table 2 results. According to Hinton, McMurray and Brownlow (2004), reliability is high when values ranging between 0.00 (more error) and 1.00 (no error), are closer to 1.00. A Cronbach’s alpha score below 0.60 is considered unacceptable, 0.60 to 0.65 undesirable yet worth investigating further, 0.65 to 0.70 acceptable, 0.70 to 0.80 good, above 0.80 very good. Reliability relates to the amount of random error from the measurement process that might be evident within the scores. Highly reliable scores are precise, reproducible, and consistent across multiple testing instances when a measure produces similar results under consistent application. Assumptions associated with convergent and discriminant validity were assessed. Convergent validity assesses whether or not the construct items in effect cluster together to form that construct, whereas discriminant validity assesses if constructs are different from one Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 9 another. Both convergent and discriminant validity were assessed using the pattern component matrix of the separate question items and the extracted factors (Table 4). An initial analysis utilised ANOVA, t, and z tests to examine the potential for demographic differences both between and within the various demographic characteristics measured. However, this had no bearing on the results of the factor analysis undertaking and are therefore not presented. 4.2 Exploratory Factor Analysis An Exploratory Factor Analysis was undertaken to test if Coates’ (2006) hypothesised six constructs as well as the new hypothesised ‘Assessment’ construct provided a reasonable fit to the data, exploratory factor analysis was used to estimate the model and assess its fit to the 569 student responses. Exploratory factor analysis was used to determine if the thirty student engagement items could be scaled into a reduced number of constructs that could provide a more meaningful basis for understanding how the activities associated with student engagement might be structured (Sass & Schmitt, 2010; Schmitt, 2011). Factors were extracted using Principal Components using the Varimax rotation method with Kaiser normalisation. For the dataset of students (n=569), Bartlett's test of sphericity is significant (p< 0.000), indicating that dimension reduction exploratory factor analysis will provide a reasonable solution. Further, a Kaiser-Meyer-Olkin (KMO) indicator of 0.913 also supports this analytical approach. Factor analysis is a modelling approach that allows a number of unobserved latent variables to be estimated that purportedly help explain the variability in the original, observed data. It is widely known that ‘retaining too few or too many factors’ has serious ramifications regarding the interpretation and the stability of the resulting factor patterns (Preacher, Zhang, Kim & Mels, 2013). In fact, Hubbard and Allen (1989) (cited in Preacher et al., 2013) describe it as ‘one of the thorniest problems a researcher faces’. A population’s factor structure for any set of latent constructs can only be hypothesised, that is, there is no one true model (Cudeck & Henly, 2003). As it is impossible to identify the true and correct number of factors to extract. The researchers therefore discuss and present those factors considered worthwhile to retain, specifically, the factors identified by Coates (2006) and Pye et al. (2015). Total Variance Explained Components Initial Eigenvalues Rotation Sums of Squared Loadings Total % of Variance Cumulative % Total % of Variance Cumulative % 1 12.151 40.503 40.503 4.805 16.016 16.016 2 2.439 8.131 48.634 3.92 13.066 29.082 3 1.641 5.471 54.105 3.281 10.935 40.017 4 1.493 4.975 59.08 2.941 9.803 49.82 5 1.422 4.738 63.818 2.364 7.881 57.701 6 1.249 4.163 67.982 2.218 7.394 65.096 7 1.054 3.512 71.494 1.919 6.398 71.494 Extraction Method: Principal Component Analysis. Table 3. Eigenvalues and the Proportion of Variance Explained by Each of the Extracted Factors Factors with an Eigenvalue of greater than 1.00 were retained and these factors explained 71.5% of the variance in the data set Kaiser (1960) (Table 3). Other factor models were examined but they did little to improve the amount of variance explained. A Scree Plot Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 10 Graph of the Eigenvalues and the extracted components is presented as Figure 1. The percentage of variance explained by each extracted factor was high, with each factor explaining between 6.4% and 16.0% of the total variation in the observed data. Figure 1. Scree plot of Eigenvalues and components extracted The resulting factor structure is presented in Table 4. Factor loadings of greater than |0.300| are presented (Hair, Anderson, Tatham & Black, 1998; 2009). Factor loadings above 0.572 are bolded. Seven distinct factors were extracted, except for the hypothesised construct of Relevance, the extracted factors confirm five of the constructs defined by Coates (2006). The latent constructs, in order of extraction were: Collaboration (Construct 1); Teaching (Construct 2); Assessment a (Construct 3); Active Learning (Construct 4); Contact with Staff (Construct 5); Assessment b (Construct 6) and Social Interaction (Construct 7). The items associated with Coates’ hypothesised construct of Relevance, were not distinctly separable and revealed cross-loading with several other extracted factors. This indicates that in the online learning environment, Relevance does not seem to be a separate construct. Rather, it disappears into the other hypothesised constructs of mostly Collaboration (1), but it does also cross-load with Teaching (2) and Active Learning. Notably, Active Learning (4) was the only other item that did not significantly load, and similarly, exhibited cross-loading with Collaboration (1). Using a cut-off criterion of 0.500, all assessment items loaded with an estimated factor. However, two distinct factors were extracted for the hypothesised construct of ‘Assessment’ (6). The hypothesised overarching themes of each of the new constructs, based on the separation of items associated with Assessment, are presented in Table 5 and cluster as follows. These results indicate that student engagement aspects of Assessment in the online environment reveal two distinct latent constructs: one associated with access and submission of assessment items, and the other revealed as post assessment feedback or discussion. Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 11 Rotated Component Matrix 1 2 3 4 5 6 7 Collaboration 2 0.815 Collaboration 3 0.767 Collaboration 1 0.740 Social Interaction 1 0.730 Collaboration 4 0.714 Teaching 2 0.310 0.730 Teaching 1 0.722 Teaching 3 0.708 0.396 Teaching 4 0.698 Teaching 5 0.451 0.539 Teaching 6 0.447 0.572 Assessment 2 0.838 Assessment 1 0.826 Assessment 3 0.817 Assessment 4 0.586 Active Learning 2 0.821 Active Learning 1 0.783 Active Learning 3 0.749 Contact with Staff 7 0.833 Contact with Staff 8 0.728 Contact with Staff 9 0.305 0.689 Assessment 7 0.817 Assessment 6 0.815 Social Interaction 2 0.385 0.701 Social Interaction 1 0.361 0.692 Social Interaction 3 0.476 0.613 Active Learning 4 0.435 0.466 Relevance 1 0.415 0.452 0.340 0.373 Relevance 2 0.483 0.32 0.323 0.340 Relevance 3 0.413 0.419 0.383 0.435 Table 4. Factor structure matrix. Extraction Method: Principal Components. Rotation Method: Varimax with Kaiser Normalisation Factors Correlated Significantly With Assessment a Access and Submission Assessment item 1: Assignments were easy to locate online Assessment item 2: Tests were easy to locate online Assessment item 3: Assignments were easy to submit online Assessment item 4: Online assignment submission gave me more time to complete assessable work Assessment b Post Assessment Feedback Assessment item 5: After my assignment was marked, the online assessment rubric clearly explained the marks I had received Assessment item 7: After my assignment was marked, the online assessment rubric feedback comment helped me to improve my learning Table 5. Separation and clustering of Assessment item 4.3 tableStudent Engagement Summary Findings In summary overall, except for the construct ‘Relevance’, all other constructs identified by Coates (2006) appear to populate a discrete and yet distinct factor structure space. Furthermore, two new latent constructs associated with Assessment seem apparent. The Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 12 student engagement constructs revealed in the online learning environment using exploratory factor analysis are: 1. Collaboration. 2. Teaching. 3. Assessment a: Access and Submission. 4. Active learning. 5. Contact with staff. 6. Assessment b: Post Assessment Feedback. 7. Social interaction. These findings serve to first, reconfirm that the initial six constructs of student engagement as identified by Coates (2006) remain apparent in the traditional campus-based environment as expected, but are also evident in the online settings positioned within the domain settings of blended online learning environment. Secondly, tertiary practitioners would consider that the Assessment construct has always been an ad hoc or anecdotal motivating factor influencing student engagement within the learning environment. However, and notably this research confirms and evidences the existence of the previously unidentified new construct of Assessment as being an apparent motivating construct of student engagement. Furthermore, the Exploratory Factor Analysis undertaken indicates the presence of two new, important student engagement latent constructs. To this end, the Assessment item revealed two additional distinct factors ‘access and submission’ and ‘post assessment feedback’. Generally, the results of this analysis agree with the constructs as defined by Coates (2006) for student engagement, in that they exist in an online learning context, and that they are also apparent and relevant in an online and on-campus (location learning) blended learning context. However, the separation of the items for Assessment into two distinctly separate latent constructs implies that student’s perspectives regarding this theme are also markedly important latent factors motivating student engagement. These results indicate that not only is ‘Assessment’ a distinctly important aspect of online student engagement, but that the hypothesised construct of ‘Assessment’ has two markedly important features. That is, within the context of the ‘Assessment’ items presented, ‘access and submission’, as well as ‘post assessment feedback’ are identified as distinctly important characteristics of student experience in the online engagement features of assessment. 5 Conclusion It was established that Coates (2006) student engagement constructs do likewise coalesce within the domain settings of blended online learning design and practice in an Australian higher education business context. The analysis of the survey instrument responses confirms this through the results of the Exploratory Factor Analysis. Although anecdotally expected that assessment was a student motivator, the discovery and confirmation of the ‘Assessment’ construct was unexpected and highlights the strong motivational factor that ‘Assessment’ has on student engagement and subsequent learning. Furthermore, the exceptions confirming the validity of all online student engagement constructs (and their associated items) related to the construct of ‘Relevance’, which is not deemed as important or as apparent as initially thought, Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 13 while the ‘Assessment’ construct, breaks down into principally two factors: ‘Access and submission’ and ‘Post Assessment Feedback’. Overall, the analysis confirms the robustness of conceptualising the phenomenon of online student engagement across all discretely identified constructs. Business faculties specifically, and universities in general, have committed to the systematic use of various types of student evaluation instruments to assure and improve the educational quality of their subject and course offerings. Such student evaluation instruments could be designed specifically to measure the student experience perspectives from different types of learning contexts and learning activities. The extension of the student engagement survey to include ‘Assessment’ constructs provides a theoretically sound and practical method for illuminating the student experience as applied to integrated online and blended learning environments, which are now the designed norm in the sector. Current and changing external environmental factors make it imperative to consider all aspects of the student experience in an increasingly online driven learning world. This in turn reflects dramatic shifts in national and global workplace settings, which is shaped by ongoing developments in the global digital economy. Student engagement continues to increasingly be the point of focal importance in formal education, workplace learning and workplace practices. Capturing and responding to these learning experience constructs through the prism of student engagement research, approaches and methods is of major relevance to increasing student engagement and learning enthusiasm. These global trends have gone into hyper-drive with the advent of the COVID-19 pandemic and the imperative to take university studies online on an unprecedented scale in the context of the wide-spread shutdown of university campuses across the world. The aftermath will see online education anchored into its prime role in supporting all forms of teaching, learning and workplace training. In relation to Assessment feedback and the use of rubric design, further research and development is needed relating to the use and value of feedback mechanisms enabled by LMSs. Rubrics and actioning feedback have dominated assessment practices in traditional higher education. For example, rubrics have been applied to the use of traditional forms of assessment tasks, i.e., essays, problem sets and project assignments requiring written student submissions. With the move towards shorter, modularised higher education courses, graduate professional skills development programs and workplace training endeavours, with an increasing variety of forms of oral, live, group and multimedia response formats, ‘rubric’ design and LMS feedback tools need further elaboration and refinement to enhance student engagement. Online feedback tools need to be broadened to include media responses other than text, adaptations for different types of oral and written student responses, both individual and group, and need to be deployed rapidly and efficiently to improve future student learning. These will be increasingly augmented by live and virtual assessment task debriefing sessions with quick mobilisation of assessment feedback regimes to operate in truncated education and training study periods, adapted to the needs and circumstances of students’ work and life preferences. Future research proposes to expand to include undergraduate students from across all university faculties and other higher education establishments. This will significantly broaden the research focus and test the veracity of student engagement constructs in differing higher education discipline contexts, including postgraduate student cohorts. Student engagement, as an overarching concept and its influential measurement in the context of the student Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 14 experience of online and blended learning environments. Continues to hold great promise in advancing learning theory, integration into educational policy, and extending student engagement motivational practice in the field. References Astin, A.W. (1985). Achieving Educational Excellence: A Critical Analysis of Priorities and Practices in Higher Education. Jossey-Bass: San Francisco. Atkinson, D. & Lim, S.L. (2013). Improving assessment processes in Higher Education: Student and teacher perceptions of the effectiveness of a rubric embedded in a LMS. Australasian Journal of Educational Technology. 29(5), pp. 651-666. Bonk, C.J. & Zhang, K. (2008). Empowering online learning: 100 + activities for reading, reflecting, displaying, & doing. John Wiley & Sons: San Francisco. Boud, D. (2010). Student Assessment for Learning In and After Courses. Final Report for Senior Fellowship. Australian Learning and Teaching Council (ALTC): http://www.olt.gov.au/resource-student-assessment-learning-and-after-courses-uts- 2010 Boud, D. & Molloy, E. (2013). (Eds.). Feedback in Higher and Professional Education Understanding it and doing it well. Routledge: London. Chickering, A.W. & Gamson, Z.F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 39(7), pp. 3-7. Coates, H. (2006). Student engagement in campus-based and online education: university connections. Routledge: London. Coates, H. (2010). Development of the Australasian survey of student engagement (AUSSE). Higher Education, 60, pp. 1-17. Coates, H. & McCormick, A.C. (Eds.) (2014). Engaging University Students International Insights from System-Wide Studies. Springer: Singapore. Coates, H. & McCormick, A.C. (2014). Introduction: Student Engagement – A Window into Undergraduate Education. In H. Coates & A.C. McCormick (Eds.) (2014). Engaging University Students International Insights from System-Wide Studies. Springer: Singapore, pp.1-12. Cudeck, R., & Henly, S. J. (2003). A realistic perspective on pattern representation in growth data: Comment on Bauer and Curran (2003). Psychological Methods, 8, pp.378–383. Dawson, P. (2017). Assessment rubrics: towards clearer and more replicable design, research and practice. Assessment & Evaluation in Higher Education, 42(3), pp. 347-360. DOI: 10.1080/02602938.2015.1111294 Garrison, D.R. & Anderson, T. (2003). E-Learning in the 21st Century. A Framework for Research and Practice.Routledge Falmer: London. Hair, J., Anderson, R., Tatham, R. and Black, W. (1998). Multivariate Data Analysis (5th Edition). Prentice Hall: New Jersey. Hair, J, F. (Jr), Black, J. F., Babin, J. & Anderson R. E. (2009). Multivariate Data Analysis (7th Edition). Prentice Hall: New Jersey. ISBN-10: 0138132631 • ISBN-13: 9780138132637 Henderson, M., Boud, D., Molloy, E., Dawson, P., Phillips, M., Ryan, T., Walker, H. & Mahoney, P. (2016). Feedback for Learning Closing the assessment loop. Federal Australian Government Office for Learning and Teaching (in progress). Project website: https://feedbackforlearning.org/ Hinton, P. R., McMurray, I. & Brownlow, C. (2014). SPSS Explained (2nd Edition). Routledge Taylor and Francis: East Sussex, UK. Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 15 Howson, C.K. & Buckley, A. (2017). Development of the UK engagement survey. Assessment & Evaluation in Higher Education, 42(7), pp. 1132-1144. DOI: 10.1080/02602938.2016.1235134 Jonsson, A. (2014). Rubrics as a way of providing transparency in assessment. Assessment & Evaluation in Higher Education, 39(7), pp. 840-852. DOI: 10.1080/02602938.2013.875117 Kaiser, H.F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, pp. 141-151. Kuh, G.D. (2003). What we’re learning about student engagement from NSSE. Change, 35(2), pp. 24-31. Kuh, G.D., Kinzie, J., Schuh, J.H., Whitt, E.J., & Associates (2005). Student success in college: Creating conditions that matter. Jossey-Bass: San Francisco. Lawson, R. (2015). Curriculum design for assuring learning – leading the way. Final Report for Senior Fellowship. Office for Learning and Teaching (OLT): http://www.olt.gov.au/resource-curriculum-design-assuring-learning-leading-way- 2015 MacCallum, R. C., Widaman, K., Zhang, Shaobo. (1999) Sample Size in Factor Analysis. Psychological Methods, 4(1), pp. 84-99 DOI:10.1037/1082-989X.4.1.84 Nicol (2010). Re-engineering Assessment Practices in Higher Education: http://www.reap.ac.uk/Home.aspx Oliver, B. (2013). Graduate attributes as a focus for institution-wide curriculum renewal: innovations and challenges. Higher Education Research & Development, 32(3), pp.450-63. Pallant, J. (2005). SPSS Survival Manual. A step by step guide to data analysis using SPSS for Windows (Ver 12.0). Allen and Unwin Press: Crows Nest, Australia. Pace, C.R. (1979). Measuring Outcomes of College: Fifty Years of Findings and Recommendations for the Future. Jossey Bass: San Francisco. Pace, C.R. (1995). From Good Practices to Good Products: Relating Good Practices in Undergraduate Education to Student Achievement. Paper presented at the Association for Institutional Research, Boston. Palmer, S., & Holt, D. (2009). Staff and student perceptions of an online learning environment: Difference and development. Australasian Journal of Educational Technology, 25(3), pp. 366- 381. Palmer, S., & Holt, D. (2010). Students’ perceptions of the value of the elements of an online learning environment: Looking back in moving forward. Interactive Learning Environments, 18(2), pp. 135-151. Palmer, S. & Holt, D. (2012). Trajectories of engagement: a repeated cross-sectional investigation of student perceptions of an online learning environment. Research in Learning Technology, 20, pp. 253-265. Palmer, S. & Holt, D. (2014). Development of student and academic staff perceptions of the elements of an online learning environment over time. Australasian Journal of Educational Technology, 30(4), pp. 375-389. Parkin, H.J., Hepplestone, S., Holden, G., Irwin, B. & Thorpe, L. (2012). A role for technology in enhancing students’ engagement with feedback. Assessment & Evaluation in Higher Education, 37(8), pp. 963-973. DOI: 10.1080/02602938.2011.592934 Pye, G., Holt, D., & Salzman, S., Bellucci, E., Lombardi, L. (2015). Engaging diverse student audiences in contemporary blended learning environments in Australian higher business education: Implications for design and practice. Australasian Journal of Information Systems, 19, pp. 1-20. Australasian Journal of Information Systems Pye et al. 2022, Vol 26, Research Article Factor structures associated with online student engagement 16 Pye, G., Holt, D. & Salzman, S. (2018). Investigating different patterns of student engagement with blended learning environments in Australian business education: Implications for design and practice. Australasian Journal of Information Systems, 22, pp. 1-23. Preacher, K. J., Zhang, G., Kim, C. & Mels, G. (2013). Choosing the Optimal Number of Factors in Exploratory Factor Analysis: A Model Selection Perspective. Multivariate Behavioral Research, 48, pp. 28–56, ISSN: 0027-3171 print/1532-7906 online. DOI: 10.1080/00273171.2012.710386. Salmon, G. (2011). E-moderating. The Key to Teaching and Learning Online. Routledge: New York. Sass, D.A., Schmitt, T.A. (2010). A Comparative Investigation of Rotation Criteria within Exploratory Factor Analysis. Multivariate Behavioral Research, 45 (1), pp. 73-103. Schmitt, T.A. (2011). Current Methodological Considerations in Exploratory and Confirmatory Factor Analysis. Journal of Psychoeducational Assessment, 29(4), pp. 304-321. Small, F. & Attree, K. (2016). Undergraduate student responses to feedback: expectations and experiences, Studies in Higher Education, 41(11), pp. 2078-2094. DOI: 10.1080/03075079.2015.1007944 Tinto, V. (1993). Leaving College: Rethinking the Causes and Cures of Student Attrition. University of Chicago Press: Chicago. Tinto, V. (1997). Classrooms as communities: exploring the educational character of student persistence. Journal of Higher Education, 68(6), pp. 599-623. Winstone, N.E., Nash, R.A., Rowntree, J. & Parker, M. (2017). ‘It’d be useful, but I wouldn’t use it’: barriers to university students’ feedback seeking and recipience. Studies in Higher Education, 42(11), pp. 2026-2041. DOI: 10.1080/03075079.2015.1130032 Copyright: © 2022 authors. This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 Australia License, which permits non- commercial use, distribution, and reproduction in any medium, provided the original author and AJIS are credited. doi: https://doi.org/10.3127/ajis.v26i0.2667 http://creativecommons.org/licenses/by-nc/3.0/au/ 1 Introduction 2 Literature Review 2.1 Student Engagement 2.2 Research Modelling Approach 3 Research Design 3.1 The Assessment Construct 3.2 Research Limitations 4 Data Collection and Analysis 4.1 Data Consistency Analysis 4.2 Exploratory Factor Analysis 4.3 tableStudent Engagement Summary Findings 5 Conclusion