Microsoft Word - [1] 6369-Article Text-21729-1-11-20201211.docx Australasian Journal of Educational Technology, 2020, 36(6). 1 Combining self-reported and observational measures to assess university student academic performance in blended course designs Feifei Han, Robert Ellis Griffith University This study combined the methods from student approaches to learning and learning analytics research by using both self-reported and observational measures to examine the student learning experience. It investigated the extent to which reported approaches and perceptions and observed online interactions are related to each other and how they contribute to variation in academic performance in a blended course design. Correlation analyses showed significant pairwise associations between approaches and frequency of the online interaction. A cluster analysis identified two groupings of students with different reported learning orientations. Based on the reported learning orientations, one-way ANOVAs showed that students with understanding orientation reported deep approaches to and positive perceptions of learning. The students with understanding orientation also interacted more frequently with the online learning tasks and had higher marks than those with reproducing orientation, who reported surface approaches and negative perceptions. Regression analyses found that adding the observational measures increased 36% of the variance in the academic performance in comparison with using self-reported measures alone (6%). The findings suggest using the combined methods to explain students’ academic performance in blended course designs not only triangulates the results but also strengthens the acuity of the analysis. Implications for practice or policy:  Using combined methods of measuring learning experience offers a relatively more comprehensive understanding of learning.  Combining self-reported and observational measures to explain students’ academic performance not only enables the results to be triangulated but also strengthens the acuity of the analysis.  To improve student learning in blended course design, teachers should use some strategies to move students from a reproducing learning orientation towards an understanding orientation and encourage active online participation by highlighting the importance of learning online. Keywords: student approaches to learning research, learning analytics research, self-reported measures, observational measures, academic performance, blended course designs Introduction Ongoing advances in Internet and computer technology have created new challenges in appraising the learning outcomes in university courses (Mestan, 2019). Positive effects have been reported for various forms of technology-enhanced learning, such as online tutorials, social networking sites, computer- supported collaborative learning, web conferencing, webinars, e-portfolio, digital games, simulations and virtual worlds (Kurvinen et al., 2020). Widespread online learning is most commonly used in combination with co-present face-to-face delivery, resulting in a blended course design, which can be defined as a systematic combination of co-present and technologically mediated interactions between students, teachers and learning resources in the pursuit of learning outcomes (Bliuc et al., 2007). In such designs, the learning experienced by students differs from that in solely face-to-face contexts, as the course activities require students to constantly move back and forth between physical and virtual spaces and to actively seek to understand how different components are related to and complementary with each other (Han & Ellis, 2020). This not only creates new ways of conceptualising but also requires new ways of assessing student learning experience (Iskander et al., 2010). Australasian Journal of Educational Technology, 2020, 36(6). 2 Research into experiences of student learning in higher education tends to be primarily based on either self- reported measures or observational evidence. Each approach has attracted some criticisms. The two sources of data are rarely combined into integrated research methodologies, which allow a type of triangulation and elaboration of the findings. Driven by the idea that the unit of research into university learning is increasingly a combination of people and technologies, this study combined self-reported methods in student approaches to learning (SAL) research (Trigwell & Prosser, 2020) and observational methods in learning analytics research (Buckingham Shum & Crick, 2012). To be more specific, the study involved 335 first-year computer science students in an Australian university and examined (a) the associations between students’ self-reported measures (approaches and perceptions) and observational measures (interaction with the online learning tasks) of their learning experience and academic performance; and (b) the contributions of self-reported and observational measures to the overall academic performance. In the following section, we review relevant studies in SAL and learning analytics research and then outline research that has adopted a combined approach using self-reported and observational evidence. Literature review SAL research SAL research is a well-recognised, research-based framework for describing experiences of student learning and explaining variation in learning outcomes in higher education (Biggs & Tang, 2011). This area of research has systematically identified key elements that explain why some students are more successful than others. SAL researchers have used interviews, open-ended questionnaires or Likert-scale questionnaires to investigate topics such as how students go about learning (approaches to learning), how they perceive learning (perceptions of learning) and how these elements are related to each other and to student learning outcomes (Trigwell & Prosser, 2020). Across a range of academic disciplines, two broad categories of approaches have been found: deep and surface approaches. Deep approaches to learning, which are directed towards meaningful understanding of the subject matter, have features of being proactive, engaging and reflective. They focus on the critical aspects of the object of study in context (Marton, 2014). In contrast, surface approaches, which tend to be formulaic and mechanistic and to rely heavily on the teaching staff and other students, seek outcomes such as simply fulfilling course requirements and passing exams (Nelson Laird et al., 2014). Students’ approaches to learning have been found to relate to their perceptions of the learning and teaching environment, confirming that a student’s approach to learning is likely to vary in different learning contexts (Entwistle, 2009). Research has reported that deep approaches are positively related to students’ perceptions of high-quality teaching, clear teaching goals and teachers’ encouragement of independence in learning (positive perceptions), whereas surface approaches are linked with perceptions of inappropriate workload and the mismatch between assessment tasks and learning objectives (negative perceptions). In addition, positive relations have been observed between deep approaches, positive perceptions and relatively better academic performance; and between surface approaches, negative perceptions and relatively poorer academic performance (Lizzio et al., 2002; Wilson et al., 1997). In blended course design, students’ perceptions of the blended learning environment, the approaches they adopt in both face-to-face and online components of the course and their academic learning outcomes have also been shown to be related. Students who report perceiving an integration between face-to-face and online learning and perceiving the online workload to be appropriate also report using deep approaches to learning and to using online learning technologies. They also tent to achieve a relatively higher level of academic performance. In contrast, those who do not perceive how online learning is integrated into the course design tend to adopt surface approaches to learning and to using online learning technologies and obtain relatively poor outcomes in the course (Ellis & Bliuc, 2016). While self-reported measures in SAL research are an important means for understanding students’ reported cognitions in learning, such assessments largely represent how students believe what they experience when learning, based on only one category of data. To more comprehensively assess experiences of student learning, this study offers a more robust methodology by extending the investigative lens to include observational methods commonly used in the learning analytics research. Australasian Journal of Educational Technology, 2020, 36(6). 3 Learning analytics research The increase in the use of educational technology and the emphasis on the capacity to collect detailed data of student interactions in experiences of learning has led to the widespread adoption of digital traces created online by learners in learning management systems (LMSs) and the like (Baker & Siemens, 2014). The large amount of digital traces of students’ online learning, known as observational measures (Pardo et al., 2017), including frequencies, durations and time-stamped sequences of students’ interaction with various online learning tasks and tools, can be stored and retrieved from LMSs (Lockyer et al., 2013). With the assistance of sophisticated data mining techniques and algorithms, when combined with additional datasets, such as students’ demographic information, the digital traces and logs can be used to increase our understanding of student experiences of learning and to improve the quality of such experiences (Siemens & Gašević, 2012). This recognition has contributed to an emerging research area known as learning analytics, which is placed at the intersection of a number of disciplines, such as learning sciences, computer science, psychology and business intelligence (Baker & Siemens, 2014). The outcomes of this area of research are used to predict attrition and retention in learning (Dawson et al., 2017), advise students for careers and employability (Bettinger & Baker, 2013), detect at-risk students early on in courses and study programmes (Krumm et al., 2014), describe study strategies and tactics (Jovanović et al., 2017), find out students’ online collaborative patterns (Kaendler et al., 2015), offer feedback about learning (Tanes et al., 2011), monitor students’ emotions and affect in learning (Ocumpaugh et al., 2014), assist learning design (Rienties & Toetenel, 2016), inform decision-making processes and educational policy (Ferguson et al., 2016) and most often to explain students’ academic performance (Jovanović et al., 2017). Despite the wide use of learning analytics in addressing the above-mentioned challenges in education and the results it has produced, researchers have pointed out that only a limited number of learning analytics studies take into account theoretically established educational theories, with the majority of the published work primarily taking empirical approaches to research design (Boyd & Crawford, 2012). This has limited its potential to produce insightful ideas of students’ intentions underlying their behaviours and strategies and to precisely locate problems in learning, as well as generate actionable knowledge to enhance student experiences and learning outcomes (Long & Siemens, 2011). To improve the insights, learning analytics research should be well grounded in educational theories and employ complementary methods, such as self-report. Through a combination of methods and data sources, a more comprehensive understanding of the phenomenon under study can be produced, and the research results will become more robust when interpreting patterns and trends in the data in order to understand experiences of learning and academic learning outcomes (Han et al., 2020). Combining self-reported and observational measures to understand learning experience Although the self-reported methods used in SAL research have been criticised for lacking objectivity in their ability to represent how students learn in reality, observational measures in learning analytics research are typically not able to reveal the intent underlying student learning (Han et al., 2020). Buckingham Shum and Crick (2012) also pointed out that relying solely on empirical methods without theoretical guidance offers limited useful information for locating learning problems and guiding pedagogical reforms and learning designs. To address these challenges, recent research has combined self-reported and observational methods to investigate the experiences of student learning in blended course designs (Ellis et al., 2017; Pardo et al., 2017). For instance, using self-reported data from questionnaires and observational data from digital footprints of students in LMSs, Han and Ellis (2017) found, on the one hand, students’ self-reported positive perceptions of the course learning environment were related to higher online participation rates indicated by the frequency of online learning events (the number of times a student used an online tool) and higher course marks; and on the other hand, negative perceptions of the learning environment were related to lower online participation rates and relatively lower levels of academic performance as measured by course marks. In a more recent study, self-reported approaches and perceptions in the learning environment were complemented by observational data, which were the digital traces of students’ learning technologies in their online learning (Han et al., 2020). The self-reported measures identified understanding and reproducing learning orientations. Using a hidden Markov model and agglomerative sequence clustering, Australasian Journal of Educational Technology, 2020, 36(6). 4 the digital trace data observed four types of learning orientations, which differed mainly on the percentage of summative and formative learning activities with which the students were engaged. Significant associations were found between the learning orientations detected by the self-reported and observational measures respectively. Students reporting an understanding learning orientation tended to engage in more formative learning activities than those reporting a reproducing orientation. As to whether self-reported and observational measures uniquely explain differences in students’ academic performance, there is considerable contestation in the literature. Some research has reported that adding observational measures can increase a large variance explained in students’ learning performance more so than by using only self-reported measures (Ellis et al., 2017; Pardo et al., 2017). Other studies have found that after introducing observational measures, self-reported measures either contributed only indirectly to the learning outcomes via the observed online interaction (Pardo et al., 2016) or their contribution was non- significant (Tempelaar et al., 2015). For instance, in a large-scale study of 151 online learning modules involving 11,256 students, the regression analyses showed that students’ duration of online engagement explained 11% of variance of students’ retention of the programmes, which was used as an indicator of their academic performance, whereas their self-reported satisfaction of the modules was a non-significant predictor (Tempelaar et al.). Clearly further research is required to investigate if self-reported and observational measures offer complementary information for explaining students’ academic performance. To add to prior research by testing the associations among self-reported and observational sources of data, this study addressed the following research questions: (1) What are the associations amongst self-reported (approaches and perceptions) and observational (interaction with the online learning tasks) measures of students’ learning experience and their academic performance? (2) To what extent do self-reported (approaches and perceptions) and observational (interaction with the online learning tasks) measures of students’ learning experience (the independent variables) contribute to their academic performance (the dependent variable)? Method Recruitment and participants We recruited the participants from first-year undergraduates who were enrolled in a foundational computer science course in a large research-intensive Australian university. We strictly followed the ethical requirements of the institution. Before the study, we explained to the students the research purpose, the voluntary nature of participation, and the essential written consent to invite participation. A total of 335 out of 365 students returned written consent forms and agreed to participate. Their ages were between 17 and 31 (M = 19.66, SD = 2.05). The research context The course was semester-long and compulsory. The course delivered topics for operating and designing digital computers, such as computer architecture, central processing unit, digital logic design, machine language, microprocessor assembly language, the Internet and servers. A major aim of the course was to enhance students’ ability to learning through inquiry into contemporary issues in computing engineering. The course was designed using the flipped approach, in which “students begin to learn the course subject by themselves and complete pre-class activities by benefitting from computer technologies in order to take the advantage of active in-class tasks efficiently” (Ozudogru & Aksu, 2020, p. 28). Building on these principles, the course comprised both face-to-face and online learning components. The face-to-face component consisted of face-to-face lectures (2 hours/week), tutorials (2 hours/week) and laboratory practice (3 hours/week). The lectures covered the key concepts and their links to practical issues. Tutorials offered students opportunities to work in collaboration to discuss questions related to the key concepts and their applications in real contexts. In the laboratory practice, students had to solve problem sequences and work collaboratively on a pre-agreed project. The online learning component was self-paced independent study in a bespoke LMS, which held a range of learning materials: learning resources, study kit, exercise sequences, multiple-choice questions and multiple-choice questions embedded in videos. The multiple- Australasian Journal of Educational Technology, 2020, 36(6). 5 choice questions tested students’ understanding of theoretical concepts, whereas the multiple-choice questions embedded in videos tested students’ abilities to apply theories into practice. Measures and instruments The study used self-reported and observational measures of student learning experience as well as their academic performance. Self-reported measures We used a questionnaire consisting of six 5-point Likert scales (1 = strongly disagree, 5 = strongly agree) to collect self-reported measures. We developed the scales using the literature in SAL research (Biggs & Tang, 2011; Wilson et al., 1997) and used them in studies in blended learning contexts (Han & Ellis, 2020). The six scales are:  Deep approaches to learning through inquiry (4 items, α = .68): describe approaches to learning through inquiry being characterised by taking initiatives, being proactive, and involving reflection (e.g., “I often pursue independent pathways when researching something”).  Surface approaches to learning through inquiry (7 items, α = .68): involve approaches to inquiry that are formulaic, highly dependent upon teaching staff, and without much reflection of the learning (e.g., “I only use the directions my teacher gives me when researching something for a task”).  Deep approaches to using online learning technologies (6 items, α = .74): reflect that using technologies in learning in a meaningful way, such as assisting inquiry, deepening understanding of the concepts, and developing essential skills (e.g., “I find I use the learning technologies in this course to further my research into a topic”).  Surface approaches to using online learning technologies (5 items, α = .76): limit using learning technologies and use them only to satisfy course requirements (e.g., “I only use the online learning technologies in this course to fulfil course requirements”).  Perceptions of the integrated learning environment (7 items, α = .86): assess students’ perceptions of the level of integration between face-to-face and online components (e.g., “The ideas we reviewed online helped with the assessment of the course”).  Perceptions of the online workload (6 items, α = .77): evaluate students’ perceptions of level of appropriate amount of work required for online learning (e.g., “The workload for the online activities was too heavy”) (negatively worded). Observational measures The observational measures were captured using the learning analytic algorithms designed by the course convener, which recorded the frequency of students’ online interactions with different learning activities in the LMS:  The learning resources directed students to an overview of compulsory and supplementary course materials, including files, videos, links to webpages, which were categorized by different topics covered in the course.  The study kit provided students with tools to organise online learning activities according to their ratings of the confidence and their perceptions of the level of easiness. This allowed students to place the activities under one of four tabs (confident/easy, confident/challenging, unconfident/easy, unconfident/challenging) for them to review and to access activities quickly.  The exercise sequences required students to solve and provide answers to a sequence of practical questions in mini-case studies based on different topics. The exercise sequences recorded student responses as either correct or incorrect.  Multiple-choice questions assessed students’ understanding of concepts in the course formatively. Apart from providing answers, an option for showing the answer without attempting the question was also built in. Hence, students’ interactions with the multiple-choice questions were recorded as one of the following: a correct response, an incorrect response and showing the answer. Australasian Journal of Educational Technology, 2020, 36(6). 6  Multiple-choice questions embedded in videos enabled students to view videos which covered issues and practice procedures in computer system design. Formative quizzes were embedded in the videos to test students’ understanding of the video contents. Similar to the method adopted for the multiple-choice questions, students’ interactions were also recorded as one of the three: a correct answer, an incorrect answer and showing the answer. Academic performance Students’ course marks were used as an indicator of their academic performance in the course. The course marks were made up of scores of the following assessment tasks: timely submission of preparation online learning tasks before each lecture and tutorial (5%); actively involved in the discussions in the tutorials (5%); four laboratory reports submission on problem-solving tasks in the laboratory practice (5%); an oral presentation, a written report and a demonstration of the implementation of the computer design project in the laboratory (15%); closed-book mid-term examination (20%) and final (40%) examination. Students’ academic performance ranged from 25 to 98 out of 100 (M = 67.28, SD = 14.43). Data collection and analysis The collection of the self-reported measures took place in one of laboratory sessions in the 12th week of the semester. The questionnaire completion took approximately 15 minutes. The collection of the observational measures and academic performance data were at the end of the semester. In order to answer the first research question, which concerns the associations between self-reported and observational measures of students’ learning experience and their academic performance, correlation analyses were conducted to examine the pairwise associations amongst variables. Then, a hierarchical cluster analysis using the self-reported measures was conducted. Based on the cluster membership, one-way ANOVAs were performed to investigate if the observational measures and course marks differed by the clusters. To provide an answer to the second research question, which sought to understand the contributions of self- reported and observational measures of students’ learning experience (i.e., the independent variables) to their academic performance (i.e., the dependent variable), two regression models were constructed. The first model included only self-reported measures, whereas in the second model, the academic performance was regressed on both self-reported and observational measures. Results Results of the first research question – the associations amongst the self-reported and observational measures of students’ learning experience and their academic performance Results of correlation analyses, hierarchical cluster analysis and one-way ANOVAs were used to answer the first research question. Results of correlation analyses The results of correlation analyses are presented in Table 1. It shows that frequency to access to study kit was negatively related to the surface approaches to using online learning technologies (r = -.12, p < .05), which also negatively associated with frequency of correctly answered multiple-choice questions embedded in videos (r = -.12, p < .05). The frequency of incorrectly answered exercise sequences had a positive association with the surface approaches to learning through inquiry (r = .13, p < .05) but a negative association with the deep approaches to learning through inquiry (r = -.12, p < .05) and perceptions of the online workload (r = -.17, p < .01). The frequency of multiple-choice questions with showing the answer positively correlated with perceptions of the integrated learning environment (r = .14, p < .05) but negatively associated with perceptions of the online workload (r = -.12, p < .05). Between self-reported measures and academic performance, while the course mark had a negative association with surface approaches to using online learning technologies (r = -.11, p < .05), it had a positive correlation with perceptions of the integrated learning environment (r = .14, p < .01) and the online workload (r = .20, p < .01). Between observational measures and performance, we found that the course mark was positively related to learning resources (r = .41, p < .01), study kit (r = .26, p < .01), frequency of correctly answered exercise sequences (r = .38, p < .01), frequency of correctly answered multiple-choice questions (r = .30, p < .01) and frequency of correctly answered multiple-choice questions embedded in videos (r = .30, p < .01), whereas it was negatively related to frequency of incorrectly answered exercise Australasian Journal of Educational Technology, 2020, 36(6). 7 sequences (r = -.16, p < .01). In general, we found that the correlation coefficients between observational learning experience and academic performance were stronger than those between self-reported learning experience and performance. Table 1 Results of correlation analyses Variables 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Self-reported measures 1 DAI -.14** .35** -.09 .25** .06 -.04 .03 -.04 -.12* -.07 -.13* -.06 -.05 -.09 -.03 .07 2 SAI --- .01 .56** -.06 -.23** .04 -.09 -.01 .13* -.09 -.06 .03 -.02 .01 .08 -.10 3 DAT --- --- -.12* .50** .05 .02 .09 .06 .02 -.03 -.05 .04 .02 .03 .06 .03 4 SAT --- --- --- -.23** -.33** .00 -.12* -.04 .10 -.08 -.01 .06 -.12* -.08 .01 -.11* 5 INTER --- --- --- --- .15** .05 -.09 .11 -.05 .09 .01 .14** .04 -.01 .06 .14** 6 POW --- --- --- --- --- -.03 .09 .02 -.17** .02 -.02 -.12* .02 .00 -.10 .20** Observational measures 7 LR --- --- --- --- --- --- .26** .60** .23** .60** .45** .24** .64** .47** .29** .41** 8 SK --- --- --- --- --- --- --- .20** 0 .16** .14* -.01 .30** .26** .09 .26** 9 ES_C --- --- --- --- --- --- --- --- .53** .65** .55** .10 .71** .60** .16** .38** 10 ES_I --- --- --- --- --- --- --- --- --- .26** .43** .22** .26** .43** .25** -.16** 11 MCQ_C --- --- --- --- --- --- --- --- --- --- .88** .24** .65** .53** .00 .30** 12 MCQ_I --- --- --- --- --- --- --- --- --- --- --- .25** .53** .61** -.02 .09 13 MCQ_S --- --- --- --- --- --- --- --- --- --- --- --- .02 -.03 .52** .01 14 VMCQ_C --- --- --- --- --- --- --- --- --- --- --- --- --- .87** .32** .30** 15 VMCQ_I --- --- --- --- --- --- --- --- --- --- --- --- --- --- .27** .10 16 VMCQ_S --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- -.02 Academic performance 17 CM --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- Note. DAI = deep approaches to learning through inquiry, SAI = surface approaches to learning through inquiry, DAT = deep approaches to using online learning technologies, SAT = surface approaches to using online learning technologies, INTER = perceptions of the integrated learning environment, POW = perceptions of the online workload, LR = learning resources, SK = study kit, ES_C = correctly answered exercise sequences, ES_I = incorrectly answered exercise sequences, MCQ_C = correctly answered multiple-choice question, MCQ_I = incorrectly answered multiple- choice question, MCQ_S = multiple-choice question with showing the answer, VMCQ_C = correctly answered multiple-choice questions embedded in videos, VMCQ_I = incorrectly answered multiple-choice questions embedded in videos, VMCQ_S = multiple-choice questions embedded in videos with showing the answer, and CM = course marks. **p < .01, *p < .05. Results of the hierarchical cluster analysis and one-way ANOVAs To facilitate interpretation, we converted the means of the six scales into z scores (M = 0, SD = 1). We used z scores in the hierarchical cluster analysis with Ward’s method. Based on the increasing value of the squared Euclidean distance between clusters, we retained a two-cluster solution with Cluster 1 having 115 students and Cluster 2 having 220 students. We then applied one-way ANOVAs to examine if self-reported and observational learning experience and academic performance differed between the clusters. Because there were sub-categories within frequency of access to exercise sequences, multiple-choice questions and multiple-choice questions embedded in videos, we first calculated the percentages of sub-categories (correctly answered, incorrectly answered and showing the answer), which we then used in the one-way ANOVAs. The results of one-way ANOVAs based on cluster membership are displayed in Table 2. Australasian Journal of Educational Technology, 2020, 36(6). 8 Table 2 Results of one-way ANOVAs based on cluster membership Measures 1 understanding (n = 115) M SD 2 reproducing (n = 220) M SD F p η2 Self-reported measures DAI 0.55 0.77 -0.29 0.99 .00 .16 SAI -0.65 0.69 0.35 0.96 97.63 .00 .23 DAT 0.62 0.70 -0.34 0.98 87.04 .00 .21 SAT -0.73 0.75 0.38 0.90 128.34 .00 .28 INTER 0.66 0.67 -0.36 0.97 100.70 .00 .23 POW 0.56 1.00 -0.27 0.88 61.12 .00 .16 Observational measures LR 897.01 379.36 892.34 507.14 0.01 .93 .00 SK 32.44 36.59 28.86 29.33 0.86 .36 .00 ES_C 50.90% 13.53% 45.00% 12.33% 16.16 .00 .05 ES_I 49.10% 13.53% 55.00% 12.33% 16.16 .00 .05 MCQ_C 56.74% 12.09% 53.62% 11.37% 5.34 .02 .02 MCQ_I 31.32% 7.35% 32.41% 8.44% 1.33 .25 .00 MCQ_S 11.94% 10.47% 13.98% 10.88% 2.66 .10 .01 VMCQ_C 54.40% 13.06% 49.56% 12.20% 10.59 .00 .03 VMCQ_I 33.46% 10.25% 35.15% 8.81% 2.33 .13 .01 VMCQ_S 12.14% 10.82% 15.29% 12.57% 4.86 .03 .02 Academic performance CM 70.55 14.10 65.57 14.33 9.22 .00 .03 Note: DAI = deep approaches to learning through inquiry, SAI = surface approaches to learning through inquiry, DAT = deep approaches to using online learning technologies, SAT = surface approaches to using online learning technologies, INTER = perceptions of the integrated learning environment, POW = perceptions of the online workload, LR = learning resources, SK = study kit, ES_C = correctly answered exercise sequences, ES_I = incorrectly answered exercise sequences, MCQ_C = correctly answered multiple-choice question, MCQ_I = incorrectly answered multiple- choice question, MCQ_S = multiple-choice question with showing the answer, VMCQ_C = correctly answered multiple-choice questions embedded in videos, VMCQ_I = incorrectly answered multiple-choice questions embedded in videos, VMCQ_S = multiple-choice questions embedded in videos with showing the answer, and CM = course marks. **p < .01, *p < .05. The results of one-way ANOVAs show that all the self-reported measures of learning experience significantly differed between the two clusters: deep approaches to learning through inquiry: F(1, 334) = 63.10, p < .01, η2 = .16; surface approaches to learning through inquiry: F(1, 334) = 97.63, p < .01, η2 = .23; deep approaches to using online learning technologies: F(1, 334) = 87.04, p < .01, η2 = .21; surface approaches to using online learning technologies: F(1, 334) = 128.34, p < .01, η2 = .28; perceptions of the integrated blended learning environment: F(1, 334) = 100.70, p < .01, η2 = .23; and perceptions of the online workload: F(1, 334) = 61.12, p < .01, η2 = .16. Compared with the 220 students in Cluster 2, the 115 students in Cluster 1 reported adopting deep approaches to learning through inquiry (M = 0.55), deep approaches to using online learning technologies (M = 0.62), positive perceptions of the integration between face-to-face and online components (M = 0.66) and perceiving the amount of online work required in the course to be appropriate (M = 0.56). In contrast, the Cluster 2 students reported surface approaches to learning through inquiry (M = 0.35), surface approaches to using online learning technologies (M = 0.38) and holding more negative perceptions of the integrated learning environments (M = -0.36) and the online workload (M = -0.27) than the Cluster 1 students. From the patterns of the self-reported approaches and perceptions, Cluster 1 students’ experience appeared to be oriented towards gaining a deep understanding of the subject matter through adopting deep approaches to inquiry, using technologies in learning and holding positive perceptions of the learning environment. The positive associations amongst these variables reported by the students in Custer 1 are referred to as this group reporting an understanding learning orientation, whereas Cluster 2 students were categorised as Australasian Journal of Educational Technology, 2020, 36(6). 9 reporting a reproducing learning orientation, characterised by surface approaches to inquiry and learning technologies and negative perceptions of the learning environment. For the observed interaction with the online learning tasks, one-way ANOVAs showed the majority of the measures differed between the two clusters: percentage of frequency of correctly answered exercise sequences, F(1, 334) = 16.16, p < .01, η2 = .05; percentage of frequency of incorrectly answered exercise sequences, F(1, 334) = 16.16, p< .01, η2 = .05; percentage of frequency of correctly answered multiple- choice questions, F(1, 334) = 5.34, p < .05, η2 = .02; percentage of frequency of correctly answered multiple-choice questions embedded in videos, F(1, 334) = 10.59, p < .01, η2 = .03; and percentage of frequency of multiple-choice questions embedded in videos with showing the answer, F(1, 334) = 4.86, p < .05, η2 = .02. Understanding students were observed to have relatively higher percentage of correctly answered exercise sequences, correctly answered multiple-choice questions and correctly answered multiple-choice questions embedded in videos, whereas reproducing students were observed to have a relatively higher percentage of incorrectly answered exercise sequences and percentage of multiple-choice questions embedded in videos with showing the answer. Furthermore, understanding and reproducing students also differed in terms of the academic performance in the course F(1, 334) = 9.22, p < .01, η2 = .03, with understanding students (M = 70.55) obtaining higher course marks than reproducing students (M = 66.57). These results demonstrate that the self-reported and observational measures of students’ learning experience and their academic performance were aligned with each other. Using the criteria provided by Cohen (1988), we found that all the values of the effect size for the self-reported measures were large, whereas those values of the effect size for the observational measures as well as students’ academic performance were small. Such results were understandable, as the population sample was clustered based on the self-reported measures rather than the observational measures or the academic performance. Results of the second research question – contributions of the self-reported and observational measures of students’ learning experience to their academic performance To examine the benefit of combining self-reported and observational variables in explaining academic performance, we constructed two regression models: in the first regression model, we only used self- reported measures to examine the contribution to the academic performance, because past research has consistently shown that the prediction of approaches and perceptions to the academic performance (Ellis et al., 2017; Trigwell et al., 2012). Observing that the three variables – surface approaches to using online learning technologies, perceptions of the integrated learning environment and perceptions of the online workload – had significant correlations with the academic performance (see Table 1), we entered the three variables into the first regression equation. In the second regression model, apart from the three self- reported variables, we added the observational variables which were significantly correlated with the academic performance (i.e., learning resources, study kit, frequency of correctly answered exercise sequences, frequency of incorrectly answered exercise sequences, frequency of correctly answered multiple-choice questions and frequency of correctly answered multiple-choice questions embedded in videos). The results of the hierarchical regression analyses are presented in Table 3. Table 3 shows that the first regression model explained around 6% of the variation in the academic performance, and this effect was rather small (F(3, 332) = 5.77, p < .01, f2 = .06). Of the three predictors, only perceptions of the online workload made a significant contribution to academic performance (β = 2.96, p < .01). In the second regression model, introducing the six observational variables explained an additional 36% of the variation in academic performance, and this R² change was significant (F(6, 349) = 30.57, p < .01, f2 = .56). The second regression model demonstrates that in addition to the significant contribution made by perceptions of the online workload (β = .12, p < .01), frequency of access to the learning resources (β = .22, p < .01), frequency of access to the study kit (β = .13, p < .05), frequency of correctly answered exercise sequences (β = .65, p < .01) and frequency of incorrectly answered exercise sequences (β = .55, p < .01) were additional significant predictors of academic performance. Altogether the self-reported and observational measures of students’ learning experience had a large effect in explaining the academic performance and they accounted for approximately 42% of the variance in academic performance (F(9, 346) = 23.52, p < .01, f2 = .72). Australasian Journal of Educational Technology, 2020, 36(6). 10 Table 3 Results of the hierarchical regression analyses Measures B SE B β t adjusted R2 ∆R2 p f2 Model 1 .06** --- SAT 1.13 -.05 -0.79 1.13 .43 .06 INTER 1.21 .10 1.65 1.21 .10 POW 1.17 .18 2.96 1.17 .00 Model 2 .36** .56 .42** .72 SAT 0.12 .90 0.01 0.14 .89 INTER 0.86 .97 0.04 0.89 .38 POW 2.27 .93 0.12 2.46 .02 LR 0.01 .00 0.22 3.41 .00 SK 0.06 .02 0.13 2.60 .01 ES_C 0.04 .01 0.61 7.45 .00 ES_I -0.04 .01 -0.55 -9.60 .00 MCQ_C -0.01 .01 -0.08 -1.12 .26 VMCQ_C -0.02 .01 -0.10 -1.31 .19 Note. SAT = surface approaches to using online learning technologies, INTER = perceptions of the integrated learning environment, POW = perceptions of the online workload, LR = learning resources, SK = study kit, ES_C = correctly answered exercise sequences, ES_I = incorrectly answered exercise sequences, MCQ_C = correctly answered multiple- choice question, and VMCQ_C = correctly answered multiple-choice questions embedded in videos. **p < .01. Discussion The current study investigated the associations between undergraduates’ self-reported approaches to learning, approaches to using online learning technologies, perceptions of the blended learning environment, observed frequency of students’ interaction with online learning tasks recorded by the LMS and their academic performance in a blended course design. It also examined the extent to which the self- reported and observational measures jointly contributed to students’ academic performance. The association between students’ learning experience and their academic performance The associations across the self-reported and observational sources of data arising in the same experience of learning aligned with each other. The results concerning the pairwise associations of the self-reported approaches and perceptions were consistent with previous findings in SAL research (e.g., Ellis & Bliuc, 2016; Trigwell et al., 2012). The identification of the two clusters of students also confirmed the existence of understanding and reproducing learning orientations, which clearly distinguished how learning through inquiry and using online learning technologies were approached and how these elements were perceived in terms of their integration (Ellis et al., 2007; Han et al., 2020). The pairwise associations between students’ self-reported approaches and perceptions and the frequencies of their online participation are also reasonable, as those perceiving that the face-to-face and online parts were well integrated tended to participate in online learning more frequently, whereas those perceiving the online workload was inappropriate were less frequently engaged online. However, all the pairwise associations between the self- reported and observational measures were rather weak, which might indicate that the self-reported and the observational measures of students’ learning experience were only partially aligned, hence requiring multiple measurements to be combined in order to represent a more comprehensive picture of student learning. These results reveal the benefits of using the combined methods to triangulate the results and to strengthen the analyses. The critical aspects of the different experiences of learning, understanding and reproducing orientations provided some reasons why some experiences of learning are relatively more successful than others. There was generally a consistency and alignment across self-reported and observational measures and academic performance reflected in the subgroups identified in the population sample. Approximately one third of the students reported adopting deep approaches to inquiry and using online learning technologies, perceiving the online learning as being well integrated with the course and perceiving the online workload as being of an appropriate level. At the same time, this grouping of students was also observed to participated in online Australasian Journal of Educational Technology, 2020, 36(6). 11 learning tasks relatively more frequently and performed relatively better in the course. In comparison, two thirds of the students reported that they opted for surface approaches, that they did not perceive the connections between their online learning and the course and that they considered the online workload as being too high. Their observational digital traces also indicated that they had relatively less frequent interactions with the online learning tasks and obtained significantly lower course marks. These consistent patterns between the self-reported and observational measures support the claim of combining both to more comprehensively understand why the experiences of some students are more successful than others in the same course. The contributions of students’ learning experience to their academic performance The results of regression analyses show that the learning experience measured by self-reported and observational methods complemented each other, as the two methods made unique contributions to students’ learning outcomes. In contrast to studies which did not find any significant prediction of students’ academic performance by self-reported measures (Tempelaar et al., 2015), our results show that even after adding the observed learning experience, students’ self-reported cognitions still significantly predicted their academic performance in the course, albeit having a small effect that explained only 6% of the variance. However, the small percentage explained by the self-reported methods is consistent with previous studies (Ellis et al., 2017; Pardo et al., 2017). Corroborating with their findings, we found that including observational measures predicted a large amount of the variance in students’ learning outcomes, highlighting the importance of measuring the observational learning experience. These results indicate that using self-reported measures of student learning experience to explain their academic performance alone may present only a partial picture of students’ experiences of learning. Implications and conclusion The limitations of the study should be pointed out so that they can be considered in the future research. First, we included only a limited number of self-reported measures of students’ learning experience. Future studies may consider other important self-reported aspects in the experiences of learning, such as students’ conceptions of learning and their perceptions of teaching quality. Second, the observational measures in the study captured only the frequency of students’ online interactions with the online learning tasks, which does not represent the complexity of students’ online interactions. Future research should aim to use other observational measures in addition to frequency, including but not limited to duration of the online interaction and the time-stamped trace data of sequences of the online interaction (Mirriahi et al., 2018; Winne et al., 2017). Through applying advanced process-mining methods to analysing these types of digital trace data, the more dynamic nature of students’ online learning can be revealed (Jovanović et al., 2017; Sonnenberg & Bannert, 2019). Notwithstanding these limitations, the results offer some valuable insights into why the quality of student experiences of learning in blended course designs vary, which may be useful in improving student learning. In blended course designs, students whose learning orientation is mostly about reproducing facts without much deep reflection present an ongoing challenge for teachers. The results reported here suggest some strategies to help move these students towards an understanding learning orientation. A fragmented perception of the purpose of the online environment – one that disintegrated with the course assessment or classroom tasks – is associated with formulaic strategies in learning. To break down such perceptions and strategies, a systematic design approach to each activity describing the role of the online tasks and resources would seem to be essential. Directly and explicitly explaining the links between face-to-face and online elements in the course, particularly around assessment activities, should be emphasised, as this will enable students to understand the holistic intent of activities in the blended learning environment when they move between the classroom and the online tasks. In addition, the teaching staff should discuss the purpose and intent of using online learning technologies in facilitating learning and model effective ways of using various online learning tools or invite students with successful experience in using online tools to do so. Pairing students with impoverished approaches and perceptions to collaborate with those with deep approaches and a clear understanding of why and how they are engaging online is also likely to make a difference in their experiences of learning. To signal the importance of online learning activities, teachers may also consider giving at least equal weight to participation in the online aspects of assessment where feasible, as this will help raise students’ awareness of the importance of participating in both face-to-face and online aspects of the learning tasks. Australasian Journal of Educational Technology, 2020, 36(6). 12 The inclusion of online aspects in university programmes and courses and in the student experience offers both significant benefits and challenges for a quality university education. The online aspects enhance, elaborate and enrich the resources, activities and connective potential of learning, but students often perceive the online part as separate from their classroom experiences. To achieve benefits from the online part of the student experience, university teachers’ approaches to course design and to teaching need to use evidence-based knowledge to help reduce the possible fragmentation students may experience. This study aims to contribute some ideas to that knowledge base for improving students’ experiences of learning in blended course designs. Acknowledgements The authors are pleased to acknowledge the financial support of the Australian Research Council through grant DP150104163. References Baker, R., & Siemens, G. (2014). Educational data mining and learning analytics. In R. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 253–272). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.016 Bettinger, E., & Baker, R. (2013). The effects of student coaching: An evaluation of a randomized experiment in student advising. Educational Evaluation and Policy Analysis, 36(1), 3–19. https://doi.org/10.3102/0162373713500523 Biggs, J., & Tang, C. (2011). Teaching for quality learning at university: What the student does. McGraw-Hill. https://doi.org/10.1111/teth.12173 Bliuc, A.-M., Goodyear, P., & Ellis, R. A. (2007). Research focus and methodological choices in studies into students’ experiences of blended learning in higher education. The Internet and Higher Education, 10(4), 231–244. https://doi.org/10.1016/j.iheduc.2007.08.001 Boyd, D., & Crawford, K. (2012). Critical questions for big data. Information, Communication & Society, 15(5), 662–679. https://doi.org/10.1080/1369118X.2012.678878 Buckingham Shum, S., & Crick, R. (2012). Learning dispositions and transferable competencies: Pedagogy, modelling and learning analytics. In S. Buckingham Shum, S. Dawson, & C. Haythornthwaite (Eds.), Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 92–101). Association for Computing Machinery. https://doi.org/10.1145/2330601.2330629 Cohen, J. (1988). Statistical power analysis for the behavioural sciences. Erlbaum. https://doi.org/10.4324/9780203771587 Dawson, S., Jovanovic, J., Gašević, D., & Pardo, A. (2017). From prediction to impact: Evaluation of a learning analytics retention program. In X. Ochoa & I. Molenaar (Eds.), Proceedings of the International Conference on Learning Analytics and Knowledge (pp. 474–478). Association for Computing Machinery. https://doi.org/10.1145/3027385.3027405 Ellis, R., & Bliuc, A.-M. (2016). An exploration into first‐year university students’ approaches to inquiry and online learning technologies in blended environments. British Journal of Educational Technology, 47(5), 970–980. https://doi.org/10.1111/bjet.12385 Ellis, R., Han, F., & Pardo, A. (2017). Improving learning analytics: Combining observational and self- report data on student learning. Educational Technology & Society, 20(3), 158–169. https://drive.google.com/file/d/1hZwMtIdZGlOjkmXKHf8-NdPWDkHdsm8w/view Entwistle, N. (2009). Teaching for understanding at university: Deep approaches and distinctive ways of thinking. Palgrave Macmillan. https://doi.org/10.1111/j.1467-9647.2011.00722.x Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittlemeier, J., Rienties, B., Ullmann, T., & Vuorikari, R. (2016). Research evidence on the use of learning analytics: Implications for education policy. Joint Research Centre. https://doi.org/10.2791/955210 Han, F., & Ellis, R. (2017). Variations in coherence and engagement in students’ experience of blended learning. In H. Partridge, K. Davis, & J. Thomas. (Eds.), Proceedings of 34th International Conference on Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education (pp. 268–275). University of Southern Queensland. http://2017conference.ascilite.org/wp- content/uploads/2017/11/Full-HAN_Feifei3.pdf Australasian Journal of Educational Technology, 2020, 36(6). 13 Han, F., & Ellis, R. (2020). Initial development and validation of the Perceptions of the Blended Learning Environment Questionnaire. Journal of Psychoeducational Assessment, 38(2), 168–181. https://doi.org/10.1177/0734282919834091 Han, F., & Ellis, R. (2020). Personalised learning networks in the university blended learning context. Comunicar, 62(1), 19–30. https://doi.org/10.3916/C62–2020–02 Han, F., Pardo, A., & Ellis, R. (2020). Students’ self-report and observed learning orientations in blended university course design: How are they related to each other and to academic performance? Journal of Computer Assisted Learning, 36(6), 969–980. https://doi.org/10.1111/jcal.12453 Iskander, M., Kapila, V., & Karim, M. (2010). Technological developments in education and automation. Springer. https://doi.org/10.1007/978-90-481-9151-2 Jovanović, J., Gašević, D., Pardo, A., Dawson, S., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher Education, 23, 74–85. https://doi.org/10.1016/j.iheduc.2017.02.001 Kaendler, C., Wiedmann, M., Rummel, N., & Spada, H. (2015). Teacher competencies for the implementation of collaborative learning in the classroom: A framework and research review. Educational Psychology Review, 27(3), 505–536. https://doi.org/10.1007/s10648–014–9288–9 Krumm, A., Waddington, R., Teasley, S., & Lonn, S. (2014). A learning management system-based early warning system for academic advising in undergraduate engineering. In J. Larusson & B. White (Eds.), Learning analytics: From research to practice (pp. 103–119). Springer. https://doi.org/10.1007/978-1-4614-3305-7_6 Kurvinen, E., Kaila, E., Laakso, M., & Salakoski, T. (2020). Long term effects on technology enhanced learning: The use of weekly digital lessons in mathematics. Informatics in Education, 19(1), 51–75. https://doi.org/10.15388/infedu.2020.04 Lizzio, A., Wilson, K., & Simons, R. (2002). University students’ perceptions of the learning environment and academic outcomes: Implications for theory and practice. Studies in Higher Education, 27, 27–52. https://doi.org/10.1080/03075070120099359 Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. https://doi.org/10.1177/0002764213479367 Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 31–40. https://er.educause.edu/-/media/files/article-downloads/erm1151.pdf Marton, F. (2014). Necessary conditions of learning. Routledge. https://doi.org/10.1080/00220671.2015.1092729 Mestan, K. (2019). Create a fine blend: An examination of institutional transition to blended learning. Australasian Journal of Educational Technology, 35(1), 70–84. https://doi.org/10.14742/ajet.3216 Mirriahi, N., Jovanovic, J., Dawson, S., Gašević, D., & Pardo, A. (2018). Identifying engagement patterns with video annotation activities: A case study in professional development. Australasian Journal of Educational Technology, 34(1), 57–72. https://doi.org/10.14742/ajet.3207 Nelson Laird, T., Seifert, T., Pascarella, E., Mayhew, M., & Blaich, C. (2014). Deeply affecting first-year students’ thinking: Deep approaches to learning and three dimensions of cognitive development. Journal of Higher Education, 85(3), 402–432. https://doi.org/10.1080/00221546.2014.11777333 Ocumpaugh, J., Baker, R., Gowda, S., Heffernan, N., & Heffernan, C. (2014). Population validity for Educational Data Mining models: A case study in affect detection. British Journal of Educational Technology, 45(3), 487–501. https://doi.org/10.1111/bjet.12156 Ozudogru, M., & Aksu, M. (2020). Pre-service teachers’ achievement and perceptions of the classroom environment in flipped learning and traditional instruction classes. Australasian Journal of Educational Technology, 36(4), 27–43. https://doi.org/10.14742/ajet.5115 Pardo, A., Han, F., & Ellis, R. A. (2016). Exploring the relation between self-regulation, online activities, and academic performance: A case study. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 422–429). Association for Computing Machinery. https://doi.org/10.1145/2883851.2883883 Pardo, A., Han, F., & Ellis, R. A. (2017). Combining university student self-regulated learning indicators and engagement with online learning events to predict academic performance. IEEE Transactions on Learning Technologies, 10(1), 82–92. https://doi.org/10.1109/TLT.2016.2639508 Rienties, B., & Toetenel, L. (2016). The impact of learning design on student behaviour, satisfaction and performance: A cross-institutional comparison across 151 modules. Computers in Human Behavior, 60, 333–341. https://doi.org/10.1016/j.chb.2016.02.074 Australasian Journal of Educational Technology, 2020, 36(6). 14 Siemens, G., & Gašević, D. (2012). Learning and knowledge analytics [Editorial]. Educational Technology & Society, 15(3), 1–2. https://drive.google.com/file/d/1fu8JL6t8pwfGSkAnktZ4AEWChPjRnbdI/view Sonnenberg, C., & Bannert, M. (2019). Using process mining to examine the sustainability of instructional support: How stable are the effects of metacognitive prompting on self-regulatory behavior? Computers in Human Behavior, 96, 259–272. https://doi.org/10.1016/j.chb.2018.06.003 Tanes, Z., Arnold, K., King, A., & Remnet, M. (2011). Using signals for appropriate feedback: Perceptions and practices. Computers & Education, 57(4), 2414–2422. https://doi.org/10.1016/j.compedu.2011.05.016 Tempelaar, D., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for feedback generation: Learning analytics in a data-rich context. Computers in Human Behavior, 47, 157–167. https://doi.org/10.1016/j.chb.2014.05.038 Trigwell, K., Ellis, R., & Han, F. (2012). Relations between students’ approaches to learning, experienced emotions and outcomes of learning. Studies in Higher Education, 37(7), 811–824. https://doi.org/10.1080/03075079.2010.549220 Trigwell, K., & Prosser, M. (2020). Exploring university teaching and learning: Experience and context. Palgrave Macmillan. https://doi.org/10.1007/978-3-030-50830-2 Wilson, K., Lizzio, A., & Ramsden, P. (1997). The development, validation and application of the Course Experience Questionnaire. Studies in Higher Education, 22(1), 33–53. https://doi.org/10.1080/03075079712331381121 Winne, P., Nesbit, J., & Popowich, F. (2017). nStudy: A system for researching information problem solving. Technology, Knowledge and Learning, 22(3), 369–376. https://doi.org/10.1007/s10758-017- 9327-y Corresponding author: Feifei Han, feifei.han@griffith.edu.au Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY- NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Han, F., & Ellis, R. (2020). Combining self-reported and observational measures to assess university student academic performance in blended course designs. Australasian Journal of Educational Technology, 36(6), 1-14. https://doi.org/10.14742/ajet.6369