Australasian Journal of Educational Technology, 2021, 38(1). 182 A literature review of questionnaires for the assessment of online learning with a specific focus on the factors and items employed Ying Zhou, Xin An, Xiuting Li, Lewei Li, Xue Gong, Yushun Li, Beijing Normal University Ching Sing Chai Chinese University of Hong Kong Jyh-Chong Liang, Chin-Chung Tsai National Taiwan Normal University Studies measuring online learning have adopted different perspectives, resulting in different approaches to their assessment of online learning. However, when we consider the literature from a wider angle, there may be complimentary or contrasting relationships. This study performed content analysis on a total of 44 studies that used questionnaires with good reliability and validity from the database of the Social Science Citation Index. Through encoding the 241 factors and 1,171 items used in the questionnaires, we found relationships among them that could be supported by classical theoretical frameworks or our general knowledge of instruction. The content measured by these questionnaires can be divided into five categories according to Li’s (2001) theory of instructional elements: (1) learning environment, (2) learner, (3) teacher, (4) learning content, and (5) learning interaction. Following the initial analysis, this study used the unified theory of acceptance and use of technology (UTAUT) (Oye et al., 2014) to classify 58 factors that reflect the learning environment. In addition, the combination of Marzano and Kendall’s (2007) new taxonomy of educational objectives and learning outcomes provides useful information for 111 factors used to assess learners. Implications for policy and practice: • An online learning questionnaire database was established with a good structure based on the classification of the factors and items of these questionnaires. • Based on the results, teachers can more accurately design online teaching without feeling lost or taking detours. Keywords: online learning, questionnaires, learning environment, learning interaction, literature review, content analysis Introduction The Internet is an important technology that hosts abundant, accessible resources for today’s students to learn and grow. This technology affords convenient communication and tools for collaborative knowledge construction. Based on internet technologies, online learning has emerged as an essential route to lifelong learning. Online learning is now widely adopted as a way of teaching due to the COVID-19 pandemic. Nonetheless, this sudden change has led to learning inequality and barriers, affecting the quality of education students receive. Online learners face many challenges, such as isolation in cyberspace, low- quality courses, and malicious information or service providers. These have drawn many researchers to explore online learning. Due to the rich research results of online learning, several reviews have been published. Recent review studies in the field of educational technology have respectively focused on the enablers and barriers to learning health sciences education online (Regmi & Jones, 2020), the experience of university students on the autism spectrum in online learning (Adams et al., 2019), online learning retention and the effective strategies for enhancing online learning retention (Muljana & Luo, 2019), the acceptance of mobile learning and its influencing factors (Alghazi et al., 2020), the factors predicting satisfaction and Australasian Journal of Educational Technology, 2021, 38(1). 183 perceived learning outcomes in e-learning (Yunusa & Umar, 2021), and the relationship between mobile learning and self-regulated learning (Palalas & Wark, 2020). Furthermore, current review studies obtained their research findings through analysing the research results, theoretical frameworks, or models of the reviewed articles that used diverse research methods. Due to the rise of online learning in recent years, a number of questionnaire surveys on online learning have been developed, but concerns about the quality of the surveys have been raised. There is still a lack of a careful review of the quality of questionnaires used in online learning. We may get to the root of the problems after examining the semantic relationships among the factors or variables from different questionnaires, that is, failure to delve into factors and items may lead to unreliable results. For example, some factors have the same name but use different items with different connotations and applications. These differences could be due to contextual reasons or they could arise from the theoretical frameworks or differences in the researchers’ perspectives. For instance, Gay (2016) used time series, course preparation, implementation, and completion to study online learning and develop their questionnaire, whereas González-Gómez et al. (2012) measured online learning through the critical elements of instructional design: objectives and content, teaching methods, teaching tools, and teacher. Our study referred to Chang et al.’s (2015) approach that aimed to create a framework by which researchers examined factors of questionnaires or scales and classified them into different dimensions based on learning theories. When the dimensions cannot encompass all factors, researchers need to reconsider the dimensions and theories. Therefore, to analyse 156 factors from 22 studies measuring technology-supported learning environments, Chang et al. put forward a six dimension framework incorporating technical, content, cognitive, metacognitive, social, and affective dimensions, verified through several learning theories. The 156 factors were then categorised into the six dimensions. Our study specifically looked at the internet- supported learning environment. We reviewed questionnaires, surveys, or scales, that measure online learning through semantic analysis of their factors and items based on relevant theories. Methodology Article retrieval and collection To ensure that the review of the literature was comprehensive, two researchers with 10 years of online learning survey experience, the first author and the corresponding author, developed a basic retrieval model (Table 1) with three retrieval formulas. We used the Web of Science Core Collection and specified the Social Sciences Citation Index (SSCI), because they include papers perceived to be of higher quality. Moreover, we selected the education-related categories. Many other studies have also searched and reviewed papers from this database (e.g., Hwang & Fu, 2019; Zou et al., 2020). First, formula 1 required that the article titles include the common keywords related to online learning, such as “online learning” and “Internet learning”; and formula 2 required that the article titles include “questionnaire” and/or “survey”. A total of 183 articles were retrieved. Nine articles were filtered out because they did not contain the application of the questionnaire. By analysing the first retrieval results, we found some new retrieval keywords in article titles, such as “difference” or “measurement” which implied that the relevant articles might also use questionnaires. Therefore, formula 3 was developed, and a second retrieval using formula 1 and formula 3 was conducted. However, since the keywords were used in a broader sense, numerous studies using various research methods other than questionnaire surveys flooded in. For example, in one study, the title of which contains three keywords, “online” and “learning” from formula 1 and “assessment” from formula 3, the researchers created 15 short tasks to assess students’ civic online reasoning ability (McGrew et al., 2018). Fifty-three articles conducting questionnaire surveys were manually identified from 670 returns. Forty were retained after deleting 13 articles which were already included in retrieval 1. Eventually, 214 (174 plus 40) retained articles were narrowed down to just 44 (Appendix 1) because the others did not provide a complete list of questionnaire items. In addition, two other experts with rich educational research experience reviewed and approved of the retrieval process, retrieval results, and the preliminary data analysis idea. Australasian Journal of Educational Technology, 2021, 38(1). 184 Table 1 The basic retrieval model Note. Title (TI) Formula 1: TI = (online learning OR on-line learning OR Internet* learning OR web* learning OR net* learning OR e-learning OR digital* learning OR distan* learning OR mobile learning) Formula 2: TI = (questionnaire* OR instrument* OR survey* OR inventory* OR relation* OR scale* OR factor*) Formula 3: TI = (differen* OR measur* OR assess* OR investigat* OR develop* OR preference* OR perception* OR explor*) Coding scheme and process Content analysis was used to analyse and encode the 44 retrieved articles. The coding process was divided into two stages. Stage one involved the general systematic coding of essential research information in the 44 articles, and stage two explored the relationships among those measurements by reviewing the factors and their items related to online learning questionnaires. In stage one, the study developed the basic information coding table (BICT), which included nearly 30 columns, such as article number, publication year, name of questionnaire, learning context, learning approach, data analysis method, sample size, all factors, and items. All articles were read and encoded twice by the appointed two of five coders. Each coder independently read approximately 18 articles on average. The agreement between the pairs of coders attained more than 90%, and differences were resolved through discussion. In stage two, we investigated related theories based on the preliminary reading of the 44 articles. After assessment and comparison were completed, together with continuing discussion with the five instructional experts, several related theories were determined. As a result, the study developed the factors and items coding table (FICT). Based on the FICT, the coders independently encoded every article through detailed reading and examination of the content of the questionnaire factors and items. To calculate the overall agreement percentage of the FICT, we divided the total number of times in which the coders agreed (241) by the total number of factors (244), obtaining an interrater agreement rate of 98.77%. Article distribution Based on the BICT, we found 44 articles published from 2002 to 2018. The respondents in 31 articles were college students from higher education institutions; 8 articles collected responses from teachers; 3 articles surveyed various unspecified online learners; 1 article involved senior high school students and their teachers; and 1 article questioned college students and teachers. The learning contexts included online courses (22 articles) such as MOOCs, the Internet (13 articles), online platforms/systems/tools (8 articles) such as Moodle, and a game (1 article). A total of 244 factors were extracted from the articles, of which 241 were encoded. These 241 factors used 1,171 items. All but one of the 44 papers examined reliability and validity to varying degrees (Appendix 2). Results Referring to Li’s (2001) seven element theory of instruction – comprising environment, learner, teacher, content, interaction, objective, and approach – the FICT divided the factors of online learning questionnaires into five categories as the first dimension, according to the assessment objects rather than the samples investigated: (1) learning environment, (2) learner, (3) teacher, (4) learning content, and (5) learning interaction. In this study, Li’s objective and approach were encoded under the category teacher, Basic retrieval model Process and number of articles retrieved After Retrieval Manual judgment After filtering Reviewed in this study Retrieval 1: Formulas 1 and 2 183 9 irrelevant articles 174 A total of 44 articles with complete scale items Retrieval 2: Formulas 1 and 3 670 617 irrelevant articles (13 repeated) 40 Australasian Journal of Educational Technology, 2021, 38(1). 185 because the teacher is the course designer; only six factors were included. In general, instructional elements were included in the research field of theories of instruction. Bruner and Press’s (1966) and Gagné’s (1985) theories of instruction are well known to educational researchers, but they did not specifically and directly propose instructional elements. Moreover, instructional elements from other theories are not good matches with the measurement of online learning; for example, Martin et al. (2007) summarised five elements of instruction, objectives, information, practice, examples, and review, based on numerous articles to meet their research needs. Therefore, through this study’s broad examination of the literature, Li’s (2001) seven element theory was recognised to be the most appropriate theory for this work. In the first dimension of the FICT, the general category distribution was as follows: learning environment (21 papers with 58 factors, 21/58), learner (35/111), teacher (15/47), learning content (5/5), and learning interaction (14/20). In Figure 1, the centre pie chart shows the percentage of factors for each of the five elements of the first dimension; the other pie charts reflect the distribution within the elements for the subdimensions. Next, the study will introduce the results of analysis of these five elements and their subdimensions in turn. Figure 1. Factor percentages Note. The (21/58) below “Learning environment” means that this dimension involves 58 factors in 21 papers. The 24.07% reflects the percentage of the factors, that is, 58 divided by 241. The same approach was used for the other dimensions. Learning environment Within the learning environment dimension (21/58), there are 58 factors in the 21 papers measuring the online learning environment. As shown in Table 2, with the exceptions of one mixed factor and two factors about economy, the other 55 factors can be divided into four categories based on the unified theory of acceptance and use of technology (UTAUT) (Oye et al., 2014). The UTAUT includes four main factors: effort expectancy, performance expectancy, facilitating conditions, and social influence. The economic variables were found to exert influence on effort expectancy and performance expectancy (Sarrab et al., 2016). Australasian Journal of Educational Technology, 2021, 38(1). 186 Table 2 Learning environment (21/58) subdimensions 2nd Dimension Definition Effort expectancy (13/19) “The degree of ease associated with the use of the system” (Venkatesh et al., 2003, p. 450) Performance expectancy (11/15) “The degree to which an individual believes that using the system will help him or her to attain gains in job performance” (Venkatesh et al., 2003, p. 447) Facilitating conditions (9/13) “The degree to which an individual believes that an organizational and technical infrastructure exists to support use of the system” (Venkatesh et al., 2003, p. 453) Social influence (8/8) “The degree to which an individual perceives that important others believe he or she should use the new system” (Venkatesh et al., 2003, p. 451) Economy (2/2) The degree to which an individual believes that online learning is affordable (user-defined) Mixed factor (1/1) Mixed measurement of multiple elements from the UTAUT (user-defined) Note. The (13/19) below “Effort expectancy” means there are 13 papers and 19 factors involved in this dimension. This approach is the same as for the other dimensions. Within the effort expectancy dimension (13/19), in addition to the overall perception of the ease of use of the learning system (Abbad et al., 2009), the ease of community building (Lu & Chiou, 2010) is a focus of researchers’ attention in three papers. The performance expectancy dimension (11/15) primarily measures the effect of the system on improving performance (Islam, 2013), effectiveness (Abbad et al., 2009) and productivity (Yuen & Ma, 2008). The research on the facilitating conditions dimension (9/13) focuses on physical infrastructure conditions (Teo, 2010) and the available manual support (Gay, 2016). Research involving the social influence (8/8) dimension focuses on the influence of people (Zhou, 2017) and the cultural environment (Sharma et al., 2017) on learners. The economic (2/2) dimension was only involved in two papers about mobile learning (Sarrab et al., 2016; Sharma et al., 2017). Learner In this study, learners refer to the students as well as the teachers who were assessed as learners. The learner dimension (35/111), identified 111 factors in the 35 studies reviewed, accounting for 46.06% of the 241 total factors. Eighty-one of these 111 factors were classified according to Marzano and Kendall’s (2007) new taxonomy of educational objectives, which describes the inner process of how people study and take action, including the self-system, metacognition system, cognition system, and knowledge system. The first three systems were adopted as the learner subdimensions, whereas the knowledge system was not directly assessed in all questionnaires. The perceptual outcomes of learning, such as learner satisfaction (Gómez- Rey et al., 2016) and knowledge improvement (Fu et al., 2009), were measured by some studies. Therefore, the study also adopted learning outcomes as a learner subdimension. The detailed classification results are given in Table 3. Australasian Journal of Educational Technology, 2021, 38(1). 187 Table 3 Learner subdimensions (35/111) 2nd Dimension 3rd Dimension Definition Self-system (16/47) Motivation (8/17) Motivation can be defined as the individual’s tendency toward and persistency in the effort to achieve goals (Ngan & Law, 2015). Self-efficacy (10/13) Self-efficacy is defined as individual’s judgments and confidence in his or her ability to complete and execute specific domain tasks (Lin, 2018). Emotional response (6/7) Emotional response measures personal affective aspects, including positive and negative responses (user-defined). Task value (1/1) Task value measures learners’ perception of a learning task’s importance or value (Artino & McCoach, 2008). Beliefs about Web- based learning (5/9) The individual’s beliefs or cognition about learning activities, learning effectiveness, and difficulties in the online learning environment (user-defined). Metacognition system (11/26) Metacognitive strategies (10/16) Metacognitive strategies are seen as sequential processes to control cognitive activities and to ensure that a cognitive goal is achieved, including planning, monitoring and regulating learning processes (Chen et al., 2014). Resource management strategies (4/10) Resource management strategies can assist learners in managing the available environment and resources (user-defined). Cognition system (5/8) Cognitive strategies (5/8) Cognitive strategies refer to the learning strategies used in information processing that facilitate the processing and utilisation of information (user-defined). Learning outcome (15/26) Learning acquisition (7/12) Learning acquisition refers to the change in knowledge and ability perceived by learners (user-defined). Psychological outcome (11/14) Psychological outcome refers to learners’ feelings about the learning environment, process, and outcomes (user-defined). Mixed factor (4/4) Mixed measurement of multiple elements from the learner (user- defined) Note. The (16/47) below “Self-system” means there are 16 papers and 47 factors involved in this dimension. This approach is the same as for the other dimensions. Self-system (16/47) Marzano and Kendall (2007) proposed that motivation, which occurs when individuals are willing to learn something or improve their ability, depends on learners’ perceptions of self-efficacy, their perceptions of task value and their emotional responses to the new task. Within the self-system (16/47) dimension, motivation (8/17) and self-efficacy (10/13) received more attention than emotional response (6/7) and task value (1/1), accounting for 63.80% of the total factors in this dimension. According to Marzano and Kendall’s (2007) theory, the self-system also comprises the interrelated factors of learners’ attitudes, beliefs, and emotions in deciding whether to engage in a new task. In addition to emotional response (6/7), which can reflect learners’ attitudes and emotions, nine factors in five studies focused on learners’ beliefs about Web-based learning (5/9). The motivation subdimension included intrinsic motivation (4/9), extrinsic motivation (2/4), and intention (4/4). Motivation can be defined as an individual’s tendency toward and persistency of effort in achieving goals (Ngan & Law, 2015). Similarly, intention measures the individual behavioural tendency to adopt new learning practices or use new platforms (Yuen & Ma, 2008). Therefore, intention was classified as a subdimension of motivation. Ten of the 44 studies developed 13 factors associated with self-efficacy. These 13 factors could be divided into three categories: (1) confidence in computer use (Tekinarslan, 2009), (2) online communication with peers (Hung, 2016), and (3) finishing a task (Lin, 2018). Thus, the construct of self-efficacy may refer to different aspects of online learning. This reflects that multiple forms of self-efficacy are needed for online Australasian Journal of Educational Technology, 2021, 38(1). 188 learning. Compared to self-efficacy (10/13), emotional response (6/7) and task value (1/1) received less attention. In the emotional response dimension, positive responses included positive attitude (Korkmaz, 2012), individual attitude and expectations (Ngan & Law, 2015), and preference for learning style (Chen et al., 2014). Anxiety (Chen et al., 2014) and negative attitude (Korkmaz, 2012) measured the negative responses. The findings indicate the range of emotional responses towards online learning. Metacognitive system (11/26) The metacognitive system involves establishing learning goals and choosing strategies to achieve those goals if the self-system has decided to engage in a given task (Marzano & Kendall, 2007). Metacognitive strategies (10/16) and resource management strategies (4/10) were included in the metacognitive system dimension (11/26). A total of 16 factors in 10 studies measured metacognitive strategies (10/16), including planning strategies (Zheng et al., 2016), monitoring strategies (Hung et al., 2010) and regulating strategies (Zheng et al., 2016), which are roughly consistent with the metacognitive strategies described by Oxford et al. (1989) in language learning, including task planning, self-evaluation, and self-monitoring. Monitoring strategies primarily focused on the control of attention (Hung et al., 2010) and persistence (Jansen et al., 2017). There are seven factors within this category, which is more than those for the other two factor categories. Resource management strategies included time management (Zheng et al., 2016), environmental structuring (Jansen et al., 2017), and help seeking (Kocdar et al., 2018). Cognitive system (5/8) The cognitive system continuously interacts with the metacognitive system in learning processes. It is responsible for effective information processing that includes information retrieval, comprehension, analysis, and knowledge utilisation (Marzano & Kendall, 2007). The information retrieval level was not measured in the reviewed articles, although the other three levels were. Six of eight factors measured the strategies to store and organise information to facilitate comprehension (Yang & Tsai, 2008). Furthermore, analysing and employing (Lin, 2018) were implemented to measure learners’ performance to analyse online resources and use information technology to deal with problems. Cognitive strategies (5/8) were measured less compared to metacognitive strategies (10/16). Learning outcomes (15/26) Unlike in traditional face-to-face courses, teachers in online courses cannot perceive their students' nonverbal cues, such as eye contact, movement, and body posture (Proserpio & Magni, 2012). This difference has encouraged many researchers to pay special attention to exploring online learning outcomes. Kraiger et al. (1993) divided online learning outcomes into three categories: cognitive, skill-based, and affective outcomes. Based on Kraiger et al.’s model, this study found that the 26 factors measuring learning outcomes could be divided into learning acquisition and psychological outcomes. Learning acquisition was composed of cognitive and skill-based outcomes. Learning acquisition (7/12) included knowledge acquisition (4/5) and ability acquisition (5/7). In terms of ability acquisition (5/7), some researchers measured learners’ transfer ability (3/4) and critical ability (1/1), while others measured changes in learners’ abilities or skills (1/2), such as: “Online discussions improve my ability of science reading” (Lee, 2013, p. 351). Psychological outcomes (11/14), which included affective outcomes, concern learners’ feelings about the online learning environment, process, and outcome and included learners’ satisfaction (8/11) and pleasure (3/3). Researchers had previously measured learners’ satisfaction when teachers used new technologies in a variety of learning situations. However, Yunusa and Umar (2021), after reviewing 53 articles about the predictors of e-learning satisfaction, pointed out that studies have subjectively measured users’ satisfaction with e-learning. The study also found that the connotations and extension of the factors that measure satisfaction (8/11), and even their names, are quite different. To build on and develop relevant research, we analysed the factors and items in the questionnaires in detail to evaluate their quality, as shown in Table 4. Table 4 only lists 10 factors since there is a pair of identical factors among the 11 used to investigate teachers and students (Gómez-Rey et al., 2016). Some factors did not contain the word “satisfaction” in their names, but item analysis showed that they measured satisfaction with online learning. This phenomenon may have something to do with different perspectives. The encoding results showed that 10 factors, using a total of 37 items, measured learners’ satisfaction with courses (4/5), learning systems (3/4), and online discussions (1/1). Furthermore, 11 aspects of learners’ satisfaction were derived from semantics analysis of the 37 items of 10 factors as follows: A: overall satisfaction, B: incentive, C: learning experience, Australasian Journal of Educational Technology, 2021, 38(1). 189 D: recommendable, E: goals achievement, F: course format, G: functions, H: learning content, I: service, J: interaction, and K: performance (see Table 4). For example, Lee et al. (2011) measured learner satisfaction with a course through the following four aspects: C: learning experience, D: recommendable, E: goals achievement, and F: course format. The content of measuring satisfaction was multifaceted, and the content of each aspect of satisfaction was also complicated. For example, for C: learning experience (5/7), which included usefulness, effectiveness, playfulness, comfort, and conditioning, Lee et al. (2011) measured the comfort and interest of the learning experience, while Gómez-Rey et al. (2016) measured the usefulness of the learning experience. Table 4 Results of analysis of satisfaction factor items Articles Factor name Measurement objects Items 11 aspects of measurement A B C D E F G H I J K Zhou, 2017 Confirmation of prior learning experience Course/MOOCs 3 1 3 2 Zhou, 2017 Satisfaction with prior learning Course/MOOCs 4 1, 4 2 3 Cheng & Chau, 2016 Course satisfaction Course 4 1 4 2 3 Gómez-Rey et al., 2016 Learner satisfaction Course 4 1 2 3 5 Lee et al., 2011 Course satisfaction Course 5 1, 4 5 2 3 Ngan & Law, 2015 E-effect Learning system/PASS 3 1, 2 3 Liaw, 2008 Perceived satisfaction Learning system/BB 4 1 2 3 4 Liaw, 2008 E-learning system quality Learning system/BB 4 1 3 2 4 Lu & Chiou, 2010 E-learning satisfaction Learning system/Toplearn 3 1, 3 2 Lee, 2013 Efficacy Online discussion 3 2, 3 1 Total number of items in one aspect of measurement 37 6 3 7 3 5 2 2 2 3 3 1 Note. A: Overall satisfaction, B: Incentive, C: Learning experience, D: Recommendable, E: Goals achievement, F: Course format, G: Functions, H: Learning content, I: Service, J: Interaction, K: Performance. In the column for the 11 aspects of measurement, the number represents the serial number of an item within a factor. An unrelated item was removed. Teacher In the 44 retrieved articles, teachers were measured not only as instructors but also as learners whose factors were similar to those of general learners (Kao et al., 2011). Therefore, we assigned the latter to the learner category and retained in this section only the factors regarding teachers’ instruction, the questionnaires of which were answered by either students or teachers. Furthermore, according to the different teaching stages and roles, teacher (15/47) measurements could be divided into instructional readiness (8/14) to adapt to the online environment before teaching, as well as the role of instructional designer (7/13) and instructional supporter (6/13) during teaching (Table 5). Australasian Journal of Educational Technology, 2021, 38(1). 190 Table 5 Teacher (15/47) subdimensions 2nd Dimension 3rd Dimension Instructional readiness (8/14) Instructor characteristics (2/2) Technical readiness (3/5) Lifestyle readiness (2/2) Pedagogical readiness (4/5) Instructional designer (7/13) Learning objective (1/1) Instructional approaches (2/5) Learning contents (2/3) Learning activities (3/3) Instructional assessment (1/1) Instructional supporter (6/13) Technical support (1/1) Emotional support (3/6) Feedback (3/4) Guidance (2/2) Mixed factor (6/7) - Note. The (15/47) below “Teacher” indicates that 15 studies and 47 factors are involved in this dimension; this approach is the same for other dimensions. Instructional readiness (8/14) focuses on the extent of teachers’ will to apply their previous experience to online teaching. Gay (2016) proposed that instructional readiness consists of three parts. This study employs Gay’s classification. First, technical readiness (3/5) evaluates teachers’ competence in accessing the Internet or a specialised network connection, using essential software tools, and seeking or acquiring online help. For example, one item is: “I know how to access the online help desk” (Gay, 2016, p. 209). Unlike UTAUT, one of its items is: “Using the e-learning system (LMS) would allow me to accomplish learning tasks more quickly” (Abbad et al., 2009, p. 20). The essential difference is that UTAUT evaluates the learning environment, while technical readiness assesses the teacher. Second, lifestyle readiness (2/2) assesses the factors from daily life that influence teachers’ satisfaction with or resistance to the online environment, including expertise, the organisational or family culture, administrative instructions, and online rules. Third, pedagogical readiness (4/5) concerns instructors’ online experiences, confidence levels, and attitudes and perceptions of the online platform, as well as their predisposition to embrace and use new technological tools to accomplish a variety of tasks. In addition, instructor characteristics (2/2) formed before teaching could also affect online instructional behaviours, so this element was placed within the instructional readiness dimension as well to assess whether teachers’ intrinsic characteristics, including not only personality traits but also ability traits, are suitable for online teaching (Selim, 2007). Teachers’ roles were divided into instructional designer (8/13) and instructional supporter (6/13), supported by Hung and Chou’s (2015) study which proposed that teachers’ roles include course designer and organiser, assessment designer and technology facilitator, social supporter, and discussion facilitator. The two categories employed in this review conveniently encompass these roles. Regarding teachers as instructional designers (7/13), their competence in instructional design was evaluated through the aspects of ability to design learning objectives (1/1) (Fu et al., 2009), instructional approach (2/5) (Yang & Tsai, 2008), learning content (2/3) (Selim, 2007), learning activity (3/3) (Ellis et al., 2009), and instructional assessment (1/1) (Hung & Chou, 2015). Teachers as instructional supporters (6/13) provided technical support (1/1), emotional support (3/6), feedback (3/4), and guidance (2/2). Technical support (1/1) is perceived as guiding and facilitating learners’ technical ability to find useful information for effective learning (Zhou et al., 2017). Emotional support (3/6) involves instructors’ presence, positive words, and warm behaviours that make students feel encouraged, enabling an intimate relationship between teachers and learners (Kang & Im, 2013). Feedback (3/4) mainly investigates what kind of feedback is received (Fu et al., 2009) and from whom (Yang & Tsai, 2008). Guidance (2/2) integrates all kinds of guidance given by teachers (Kang & Im, 2013). Zhou et al. Australasian Journal of Educational Technology, 2021, 38(1). 191 (2017) found that instructional support – which was referred to as teachers’ online homework guidance (TOHG), comprised five factors derived through exploratory factor analysis: homework requirements, learning resources, learning content and learning methods, internet technology, and Internet protection. Zhou et al.’s paper was not covered in the 44 selected articles because of using the term online homework rather than online learning. Guidance (2/2) generally combined the five factors of TOHG to varying degrees, which led to less-detailed measurements. Perhaps to promote objectivity, the support (6/13) provided by teachers was primarily evaluated through students’ perceptions. Learning content Only five factors measured the learning content (5/5) itself. Using the same method as analysing satisfaction showed that researchers paid more attention to the degree of difficulty of the learning content (Gay, 2016; Gómez-Rey et al., 2016; Yang & Tsai, 2008). Other research concerns included the format of learning content (Yang & Tsai, 2008), the connection between learning content and learning objectives (Gómez- Rey et al., 2016), whether the learning content was updated in a timely way (Gay, 2016), whether the learning content had a clear organisational structure (Gay, 2016), whether the learning content met learners’ needs (Gay, 2016), and whether the learning content was stimulating for learners (Gómez-Rey et al., 2016). Learning interaction The significance of interaction for online learning is not ignored in the review studies. Among the 44 articles, 14 articles with 20 factors measured four aspects of learning interaction (14/20): teacher-learner interaction (3/3) (Kang & Im, 2013), learner-learner Interaction (7/9) (Lee, 2002), learner-content interaction (1/1) (Yang & Tsai, 2008) and the social connectedness of learner-environment interaction (5/7) (Slagter Van Tryon & Bishop, 2012). The factors in learner-learner interaction primarily measured online discussion (3/3), peer support (2/3), and mixed factors (3/3). Social connectedness refers to students’ perceptions of themselves as being socially connected to their peers or others in online learning environments (Slagter Van Tryon & Bishop, 2012). This reveals the multiple forms of interaction that are needed for online learning. Discussion and conclusion In sum, 241 factors constituted by 1,171 items were reviewed. Our analyses revealed that from a holistic perspective, close relationships can be found among the existing online learning measurements. Nevertheless, some research gaps have also surfaced. Firstly, according to Li’s (2001) theory of seven instructional elements, the content measured by these questionnaires can be divided into five categories: (1) learning environment, (2) learner, (3) teacher, (4) learning content, and (5) learning interaction. These five categorical elements were the foci of past research but they could also be considered as key dimensions when educators design online learning or teaching. However, in addition to these five elements, the study classified two other elements, namely instructional objective and approach, into the teacher dimension. Transitioning from face-to-face learning to online learning can be complicated. Online teaching entails enabling students to learn at any time and in any place, using diverse software to process massive amounts of messy information without a teacher by their side. Therefore, it is necessary to propose well-considered instructional approaches and to re-plan the original instructional objectives to match the learning models with the e-equipment, such as project-based learning, collaborative learning, or inquiry learning. COVID-19 has now pushed a large number of teachers to teach online. Zhou et al. (2017) found that what teachers may lack is web-pedagogical-content knowledge (WPCK), followed by web-pedagogical knowledge (WPK) compared with their web knowledge and web- content knowledge. Moreover, the two factors of teachers’ WPK and WPCK could significantly predict each of the factors of their online homework guidance. Thus, the novelty of instructional approaches and objectives should be paid special attention to, but may be ignored by teachers. We found that learner associated factors are the most studied among the five categories (35/111), and the current survey seems to provide relatively comprehensive measures. Researchers have measured how learners study – through the self-system, metacognitive system and cognitive system – and what learning outcomes they achieve. In terms of the learner dimension, the self-system, including motivation (Kao et al., Australasian Journal of Educational Technology, 2021, 38(1). 192 2011; Ngan & Law, 2015) and self-efficacy (Hung, 2016; Kao et al., 2011), received the widest attention, whereas the cognitive system, and particularly cognitive strategies, was relatively understudied. Learners’ motivation (8/17) and metacognitive strategies (10/16) were widely employed among researchers. Moreover, researchers focused more on intrinsic motivation (4/9) than extrinsic motivation (2/4) or intention (4/4). The monitoring ability also received more attention than regulation or planning (González- Gómez et al., 2012; Hung, 2016) in accord with the finding that learner control was an essential factor in online learning readiness (Hung, 2016). In sum, the review of online learners’ measurements showed that the online learning process was similar to traditional learning, or that researchers surveyed online learners and offline learners in the same way. We speculate two reasons why the measurements of learners lacked novelty. First, as educators or researchers, we may not know the students who are digital natives. Second, the existing questionnaires have been focusing on formal learning contexts with student samples, ignoring informal learning with all internet users. This is not in line with the human learning needs of lifelong learning these days. There may thus be a need for a substantial qualitative study to identify the unique factors that shape students’ online learning. Although the learners’ measurements are fairly comprehensive, there are many shortcomings in the measuring instruments’ fineness and stability in addition to the lack of novelty. For instance, the measurement of online learning satisfaction was often mixed, as shown in Table 4. Furthermore, its items were varied, incomplete, and somewhat chaotic, reflecting a lack of consensus compared to the classic leisure satisfaction scale (Beard & Ragheb, 1980), whose six factors and 24 items have been implemented stably over 40 years of research. Yunusa and Umar (2021) contributed a possible reason for the situation after reviewing 53 articles to explore the predictors of e-learning satisfaction. They pointed out that over the past few years, studies have taken subjective measures to understand users’ satisfaction with e-learning. Therefore, a new online learning satisfaction questionnaire needs to be developed based on theories. There may be two paths to follow. One is to make a detailed and all-sided analysis of previous satisfaction measurement instruments like Table 4 and clarify the existing structure. The other is to refer to the psychological or philosophical theory about satisfaction. The online learning environment (21/58) was the second area of focus. In this dimension, researchers paid more attention to hardware and software (effort expectancy, performance expectancy, facilitating condition) than the social environment (social influence and economy). The change in the learning environment from traditional to online is more apparent than the change in other elements, so it has been an issue of concern and has thus been measured fully. The number of articles in relation to these factors was less than those about learners because its connotations are simpler than those of learners. Two major theoretical frameworks were used to measure the online learning environment, the technology acceptance model (TAM) (e.g., Yuen & Ma, 2008) and UTAUT (e.g., Teo, 2010) containing TAM (Oye et al., 2014). Alghazi et al.’s (2020) reviewing studies on mobile learning acceptance showed a similar finding that the UTAUT was most frequently used and followed by TAM. The third area of focus was the teacher dimension (15/47). Although teachers and learners play equally essential roles in the teaching process, the study did not find any classic pedagogical theory supporting the dimension. We speculated that this lack was due to an insufficient number of papers. Both the number of papers (15) and the number of factors (52) measuring teachers were approximately half those that measured learners (35/111). It is well known that sampling is much more difficult among teachers than among students. Even so, the study categorised the factors regarding teachers into the instructional readiness, instructional designer, and instructional supporter categories, which accords with our general knowledge of teachers. Perhaps future research could draw more extensively from teacher education frameworks to further investigate measurement of instructional behaviours. Because of its differences from the traditional instructional environment, online instructional design cannot simply follow existing tradition but must also adapt to the online environment. Therefore, it is vital to understand teachers’ perspectives, and the differences between students’ and teachers’ concerns (Gómez- Rey et al., 2016). Because of the differences (Morris et al., 2019), this study suggests that future surveys can compare these two groups’ feelings and attitudes to discover the gap between what students want and what teachers provide. Besides, a lack of face-to-face interaction in online learning leads to an emphasis on instructional support. However, the measurement of such support was not precise enough because one factor included various meanings. These results indicate that it might be preferable to develop the factors into Australasian Journal of Educational Technology, 2021, 38(1). 193 multiple dimensions. Through the exploration of 14 papers measuring learning interaction, we found that it involved multiple interactions such as learner-learner interaction, teacher-learner interaction, learner-content interaction, and learner-environment interaction, although the amount of research on each type of interaction was still limited. The results show that the interactive forms of online learning are rich. Research showed that the dimension might influence students’ dropout rate (Hone & El Said, 2016) and e-learning satisfaction (Yunusa & Umar, 2021), indicating that they deserved more attention. Learning content (5/5) was the least measured category. This may be because learning content is subject- related and is not the focus of online learning. Nevertheless, the Internet provides a great wealth of learning resources, and the attributes of learning content are rich, such as its harmfulness (Zhou et al., 2017) and fragmentation. Researchers need to take note of this. Finally, the proposed dimensional framework based on the 241 factors and 1,171 items can be used to develop new online learning questionnaires. It provides a broad and relatively mature online learning questionnaire database for monitoring online learning. It is also possible to conceive other meaningful factors and dimensions outside of this study’s dimensional framework. The dimensional framework can comprehensively guide the online learning instructional design, and be used as a self-checklist to help teachers clarify which factors are neglected in their instructional design. Limitation and future works Online learning has been and will continue to be adopted by many education institutions. Consequently, surveys and measurements are important for online learning research. It is thus crucial to review the measurement research regularly and from various angles. In this article, we have aimed to present a systematic analysis of 44 SSCI papers on online learning measurement, and have identified the relevant factors and dimensions. Nonetheless, this research also has certain limitations. First, to meet our research requirements, 241 factors were refined and classified according to several theories, but this approach may have prevented us from observing online learning styles, such as self- regulating learning (Kocdar et al., 2018), online collaborative learning (Korkmaz, 2012), or online project- based learning (Lin, 2018), in our research due to the fragmentation of these factors. Second, because the naming of factors may depend on the researcher’s subjective understanding, it is necessary to examine the items of a factor to ascertain the actual content measured. Therefore, in the existing online learning survey research, only 44 papers containing relevant items were selected for this study, which may be a limitation. Third, carrying out a survey was not the only way to measure online learning; interviews, tests, behaviour analysis, and other methods were also implemented. Nevertheless, studies that used other research methods were not included in this study, and therefore, some dimensions that were less commonly measured in the studies selected may be measured more frequently through other methods. For example, tests have been used to measure knowledge systems. Lastly, although we used nearly 30 keywords for paper retrieval, the retrieval still took online learning as its core and did not contain sub-concepts such as online homework, online discussion, or online exercises. Therefore, some studies that may be relevant were not retrieved. In addition to the abovementioned omissions, some highly correlated concepts such as information literacy, cloud learning, blended learning, or informal learning were not included in this study, which is another limitation. This paper was written during the global COVID-19 outbreak. Online teaching and learning was flourishing in many countries around the world in this period. It is believed that more online learning research results will emerge in the near future. It is necessary for future review research to focus on tracking this new research to lead the direction of online learning measurement and, consequently, lead online learning to mature development. Australasian Journal of Educational Technology, 2021, 38(1). 194 Acknowledgements We would like to thank Mengying Han, Jiajun Zou, and Xingjun Guo for their work in the early stage of coding. This paper is supported by National Natural Science Foundation of China (NSFC) under grant No. 61977009. References Abbad, M. M., Morris, D., & De Nahlik, C. (2009). Looking under the bonnet: Factors affecting student adoption of e-learning systems in Jordan. The International Review of Research in Open and Distributed Learning, 10(2), 1-25. https://doi.org/10.19173/irrodl.v10i2.596 Adams, D., Simpson, K., Davies, L., Campbell, C., & Macdonald, L. (2019). Online learning for university students on the autism spectrum: A systematic review and questionnaire study. Australasian Journal of Educational Technology, 35(6), 111-131. https://doi.org/10.14742/ajet.5483 Alghazi, S. S., Wong, S. Y., Kamsin, A., Yadegaridehkordi, E., & Shuib, L. (2020). Towards sustainable mobile learning: A brief review of the factors influencing acceptance of the use of mobile phones as learning tools. Sustainability, 12(24), 10527. https://doi.org/10.3390/su122410527 Artino, A. R., & McCoach, D. B. (2008). Development and initial validation of the online learning value and self-efficacy scale. Journal of Educational Computing Research, 38(3), 279-303. https://doi.org/10.2190/EC.38.3.c Beard, J. G., & Ragheb, M. G. (1980). Measuring leisure satisfaction. Journal of Leisure Research, 12(1), 20-33. https://doi.org/10.1080/00222216.1980.11969416 Bruner, J. S., & Press, B. (1966). Toward a theory of instruction. The Belknap Press of Harvard University Press. Chang, H. Y., Wang, C. Y., Lee, M. H., Wu, H. K., Liang, J. C., Lee, S., Chiou, G. L., Lo, H. C., Lin, J. W., Hsu, C. Y., Wu, Y. T., Chen, S. F., Hwang, F. K., & Tsai, C. C. (2015). A review of features of technology-supported learning environments based on participants' perceptions. Computers in Human Behavior, 53, 223-237. https://doi.org/10.1016/j.chb.2015.06.042 Chen, L., Zhang, R., & Liu, C. (2014). Listening strategy use and influential factors in web-based computer assisted language learning. Journal of Computer Assisted Learning, 30(3), 207-219. https://doi.org/10.1111/jcal.12041 Cheng, G., & Chau, J. (2016). Exploring the relationships between learning styles, online participation, learning achievement and course satisfaction: An empirical study of a blended learning course. British Journal of Educational Technology, 47(2), 257-278. https://doi.org/10.1111/bjet.12243 Ellis, R. A., Ginns, P., & Piggott, L. (2009). E-learning in higher education: Some key aspects and their relationship to approaches to study. Higher Education Research & Development, 28(3), 303-318. https://doi.org/10.1080/07294360902839909 Fu, F., Su, R., & Yu, S. (2009). Egameflow: A scale to measure learners’ enjoyment of e-learning games. Computers & Education, 52(1), 101-112. https://doi.org/10.1016/j.compedu.2008.07.004 Gagné, M. R. (1985). The conditions of learning and theory of instruction (4th ed.). Holt, Rinehart and Winston. Gay, G. H. E. (2016). An assessment of online instructor e-learning readiness before, during, and after course delivery. Journal of Computing in Higher Education, 28(2), 199-220. https://doi.org/10.1007/s12528-016-9115-z Gómez-Rey, P., Barbera, E., & Fernández-Navarro, F. (2016). Measuring teachers and learners' perceptions of the quality of their online learning experience. Distance Education, 37(2), 146-163. https://doi.org/10.1080/01587919.2016.1184396 González-Gómez, F., Guardiola, J., Martín Rodríguez, Ó., & Montero Alonso, M. Á. (2012). Gender differences in e-learning satisfaction. Computers & Education, 58(1), 283-290. https://doi.org/10.1016/j.compedu.2011.08.017 Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A survey study. Computers & Education, 98, 157-168. https://doi.org/10.1016/j.compedu.2016.03.016 Hung, M. (2016). Teacher readiness for online learning: Scale development and teacher perceptions. Computers & Education, 94, 120-133. https://doi.org/10.1016/j.compedu.2015.11.012 Hung, M., & Chou, C. (2015). Students’ perceptions of instructors’ roles in blended and online learning environments: A comparative study. Computers & Education, 81, 315-325. https://doi.org/10.1016/j.compedu.2014.10.022 Hung, M., Chou, C., Chen, C., & Own, Z. (2010). Learner readiness for online learning: Scale https://doi.org/10.19173/irrodl.v10i2.596 https://doi.org/10.14742/ajet.5483 https://doi.org/10.3390/su122410527 https://doi.org/10.2190/EC.38.3.c https://doi.org/10.1080/00222216.1980.11969416 https://doi.org/10.1016/j.chb.2015.06.042 https://doi.org/10.1111/jcal.12041 https://doi.org/10.1111/bjet.12243 https://doi.org/10.1080/07294360902839909 https://doi.org/10.1016/j.compedu.2008.07.004 https://doi.org/10.1007/s12528-016-9115-z https://doi.org/10.1080/01587919.2016.1184396 https://doi.org/10.1016/j.compedu.2011.08.017 https://doi.org/10.1016/j.compedu.2016.03.016 https://doi.org/10.1016/j.compedu.2015.11.012 https://doi.org/10.1016/j.compedu.2014.10.022 Australasian Journal of Educational Technology, 2021, 38(1). 195 development and student perceptions. Computers & Education, 55(3), 1080-1090. https://doi.org/10.1016/j.compedu.2010.05.004 Hwang, G., & Fu, Q. (2019). Trends in the research design and application of mobile language learning: A review of 2007-2016 publications in selected SSCI journals. Interactive Learning Environments, 27(4), 567-581. https://doi.org/10.1080/10494820.2018.1486861 Islam, A. K. M. N. (2013). Investigating e-learning system usage outcomes in the university context. Computers & Education, 69, 387-399. https://doi.org/10.1016/j.compedu.2013.07.037 Jansen, R. S., van Leeuwen, A., Janssen, J., Kester, L., & Kalz, M. (2017). Validation of the self- regulated online learning questionnaire. Journal of Computing in Higher Education, 29(1), 6-27. https://doi.org/10.1007/s12528-016-9125-x Kang, M., & Im, T. (2013). Factors of learner-instructor interaction which predict perceived learning outcomes in online learning environment. Journal of Computer Assisted Learning, 29(3), 292-301. https://doi.org/10.1111/jcal.12005 Kao, C., Wu, Y., & Tsai, C. (2011). Elementary school teachers’ motivation toward web-based professional development, and the relationship with internet self-efficacy and belief about web-based learning. Teaching and Teacher Education, 27(2), 406-415. https://doi.org/10.1016/j.tate.2010.09.010 Kocdar, S., Karadeniz, A., Bozkurt, A., & Buyuk, K. (2018). Measuring self-regulation in self-paced open and distance learning environments. International Review of Research in Open and Distributed Learning, 19(1), 25-43. https://doi.org/10.19173/irrodl.v19i1.3255 Korkmaz, Ö. (2012). A validity and reliability study of the online cooperative learning attitude scale (OCLAS). Computers & Education, 59(4), 1162-1169. https://doi.org/10.1016/j.compedu.2012.05.021 Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78(2), 311- 328. https://doi.org/10.1037/0021-9010.78.2.311 Lee, I. (2002). Gender differences in self-regulated on-line learning strategies within Korea's university context. Educational Technology Research and Development, 50(1), 101-111. https://doi.org/10.1007/BF02504967 Lee, S. J., Srinivasan, S., Trail, T., Lewis, D., & Lopez, S. (2011). Examining the relationship among student perception of support, course satisfaction, and learning outcomes in online learning. The Internet and Higher Education, 14(3), 158-163. https://doi.org/10.1016/j.iheduc.2011.04.001 Lee, S. W. (2013). Investigating students’ learning approaches, perceptions of online discussions, and students’ online and academic performance. Computers & Education, 68, 345-352. https://doi.org/10.1016/j.compedu.2013.05.019 Li, B. D. (2001). Theories of instruction. People’s Education Press of China. Liaw, S. (2008). Investigating students’ perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the blackboard system. Computers & Education, 51(2), 864-873. https://doi.org/10.1016/j.compedu.2007.09.005 Lin, C. (2018). The development of an instrument to measure the project competences of college students in online project-based learning. Journal of Science Education and Technology, 27(1), 57-69. https://doi.org/10.1007/s10956-017-9708-y Lu, H., & Chiou, M. (2010). The impact of individual differences on e-learning system satisfaction: A contingency approach. British Journal of Educational Technology, 41(2), 307-323. https://doi.org/10.1111/j.1467-8535.2009.00937.x Martin, F., Klein, J. D., & Sullivan, H. (2007). The impact of instructional elements in computer-based instruction. British Journal of Educational Technology, 38(4), 623-636. https://doi.org/10.1111/j.1467-8535.2006.00670.x Marzano, R. J., & Kendall, J. S. (2007). The new taxonomy of educational objectives (2nd ed.). Corwin Press. McGrew, S., Breakstone, J., Ortega, T., Smith, M., & Wineburg, S. (2018). Can students evaluate online sources? Learning from assessments of civic online reasoning. Theory & Research in Social Education, 46(2), 165-193. https://doi.org/10.1080/00933104.2017.1416320 Morris, N. P., Swinnerton, B., & Coop, T. (2019). Lecture recordings to support learning: A contested space between students and teachers. Computers & Education, 140(103604). https://doi.org/10.1016/j.compedu.2019.103604 Muljana, P. S., & Luo, T. (2019). Factors contributing to student retention in online learning and recommended strategies for improvement: A systematic literature review. Journal of Information https://doi.org/10.1016/j.compedu.2010.05.004 https://doi.org/10.1080/10494820.2018.1486861 https://doi.org/10.1016/j.compedu.2013.07.037 https://doi.org/10.1007/s12528-016-9125-x https://doi.org/10.1111/jcal.12005 https://doi.org/10.1016/j.tate.2010.09.010 https://doi.org/10.19173/irrodl.v19i1.3255 https://doi.org/10.1016/j.compedu.2012.05.021 https://doi.org/10.1037/0021-9010.78.2.311 https://doi.org/10.1007/BF02504967 https://doi.org/10.1016/j.iheduc.2011.04.001 https://doi.org/10.1016/j.compedu.2013.05.019 https://doi.org/10.1016/j.compedu.2007.09.005 https://doi.org/10.1007/s10956-017-9708-y https://doi.org/10.1111/j.1467-8535.2009.00937.x https://doi.org/10.1111/j.1467-8535.2006.00670.x https://doi.org/10.1080/00933104.2017.1416320 https://doi.org/10.1016/j.compedu.2019.103604 Australasian Journal of Educational Technology, 2021, 38(1). 196 Technology Education: Research, 18, 19-57. https://doi.org/10.28945/4182 Ngan, S., & Law, K. M. Y. (2015). Exploratory network analysis of learning motivation factors in e- learning facilitated computer programming courses. The Asia-Pacific Education Researcher, 24(4), 705-717. https://doi.org/10.1007/s40299-014-0223-0 Oxford, R. L., Lavine, R. Z., & Crookall, D. (1989). Language learning strategies, the communicative approach and their classroom implications. Foreign Language Annals, 22(1), 29-39. https://doi.org/10.1111/j.1944-9720.1989.tb03139.x Oye, N. D., A. Iahad, N., & Ab. Rahim, N. (2014). The history of UTAUT model and its impact on ICT acceptance and usage by academicians. Education and Information Technologies, 19(1), 251-270. https://doi.org/10.1007/s10639-012-9189-9 Palalas, A., & Wark, N. (2020). The relationship between mobile learning and self-regulated learning: A systematic review. Australasian Journal of Educational Technology, 36(4), 151-172. https://doi.org/10.14742/ajet.5650 Proserpio, L., & Magni, M. (2012). Teaching without the teacher? Building a learning environment through computer simulations. International Journal of Information Management, 32(2), 99-105. https://doi.org/10.1016/j.ijinfomgt.2011.09.002 Regmi, K., & Jones, L. (2020). A systematic review of the factors – enablers and barriers – affecting e- learning in health sciences education. BMC Medical Education, 20(911). https://doi.org/10.1186/s12909-020-02007-6 Sarrab, M., Al Shibli, I., & Badursha, N. (2016). An empirical study of factors driving the adoption of mobile learning in Omani higher education. The International Review of Research in Open and Distributed Learning, 17(4), 331-349. https://doi.org/10.19173/irrodl.v17i4.2614 Selim, H. M. (2007). Critical success factors for e-learning acceptance: Confirmatory factor models. Computers & Education, 49(2), 396-413. https://doi.org/10.1016/j.compedu.2005.09.004 Sharma, S. K., Sarrab, M., & Al-Shihi, H. (2017). Development and validation of mobile learning acceptance measure. Interactive Learning Environments, 25(7), 847-858. https://doi.org/10.1080/10494820.2016.1224250 Slagter Van Tryon, P. J., & Bishop, M. J. (2012). Evaluating social connectedness online: The design and development of the social perceptions in learning contexts instrument. Distance Education, 33(3), 347-364. https://doi.org/10.1080/01587919.2012.723168 Tekinarslan, E. (2009). Turkish university students’ perceptions of the world wide web as a learning tool: An investigation based on gender, socio-economic background, and web experience. The International Review of Research in Open and Distributed Learning, 10(2), 1-19. https://doi.org/10.19173/irrodl.v10i2.598 Teo, T. (2010). Development and validation of the e-learning acceptance measure (ELAM). The Internet and Higher Education, 13(3), 148-152. https://doi.org/10.1016/j.iheduc.2010.02.001 Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425-478. https://doi.org/10.2307/30036540 Yang, F., & Tsai, C. (2008). Investigating university student preferences and beliefs about learning in the web-based context. Computers & Education, 50(4), 1284-1303. https://doi.org/10.1016/j.compedu.2006.12.009 Yuen, A. H. K., & Ma, W. W. K. (2008). Exploring teacher acceptance of e-learning technology. Asia- Pacific Journal of Teacher Education, 36(3), 229-243. https://doi.org/10.1080/13598660802232779 Yunusa, A. A., & Umar, I. N. (2021). A scoping review of critical predictive factors (CPFs) of satisfaction and perceived learning outcomes in e-learning environments. Education and Information Technologies, 26(1), 1223-1270. https://doi.org/10.1007/s10639-020-10286-1 Zheng, C., Liang, J., Yang, Y., & Tsai, C. (2016). The relationship between Chinese university students’ conceptions of language learning and their online self-regulation. System, 57, 66-78. https://doi.org/10.1016/j.system.2016.01.005 Zhou, J. (2017). Exploring the factors affecting learners’ continuance intention of MOOCs for online collaborative learning: An extended ECM perspective. Australasian Journal of Educational Technology, 33(5), 123-135. https://doi.org/10.14742/ajet.2914 Zhou, Y., Chai, C. S., Liang, J., Jin, M., & Tsai, C. (2017). The relationship between teachers’ online homework guidance and technological pedagogical content knowledge about educational use of Web. Asia-Pacific Education Researcher, 26(5), 239-247. https://doi.org/10.1007/s40299-017-0344-3 Zou, D, Luo, S., Xie, H., & Hwang, G. (2020). A systematic review of research on flipped language classrooms: Theoretical foundations, learning activities, tools, research topics and findings, Computer https://doi.org/10.28945/4182 https://doi.org/10.1007/s40299-014-0223-0 https://doi.org/10.1111/j.1944-9720.1989.tb03139.x https://doi.org/10.1007/s10639-012-9189-9 https://doi.org/10.14742/ajet.5650 https://doi.org/10.1016/j.ijinfomgt.2011.09.002 https://doi.org/10.1186/s12909-020-02007-6 https://doi.org/10.19173/irrodl.v17i4.2614 https://doi.org/10.1016/j.compedu.2005.09.004 https://doi.org/10.1080/10494820.2016.1224250 https://doi.org/10.1080/01587919.2012.723168 https://doi.org/10.19173/irrodl.v10i2.598 https://doi.org/10.1016/j.iheduc.2010.02.001 https://doi.org/10.2307/30036540 https://doi.org/10.1016/j.compedu.2006.12.009 https://doi.org/10.1080/13598660802232779 https://doi.org/10.1007/s10639-020-10286-1 https://doi.org/10.1016/j.system.2016.01.005 https://doi.org/10.14742/ajet.2914 https://doi.org/10.1007/s40299-017-0344-3 Australasian Journal of Educational Technology, 2021, 38(1). 197 Assisted Language Learning. https://doi.org/10.1080/09588221.2020.1839502 Corresponding author: Yushun Li, 06174@bnu.edu.cn Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC- ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY- NC-ND 4.0. Please cite as: Zhou, Y., An, X., Li, X., Li, L., Gong, X., Chai, C.S., Liang, J-C., Li, Y., & Tsai, C-C. (2021). A literature review of questionnaires for the assessment of online learning with a specific focus on the factors and items employed. Australasian Journal of Educational Technology, 38(1), 182- 204. https://doi.org/10.14742/ajet.6719 https://doi.org/10.1080/09588221.2020.1839502 https://creativecommons.org/licenses/by-nc-nd/4.0/ https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.6719 Australasian Journal of Educational Technology, 2021, 38(1). 198 Appendix 1 Reviewed articles Abbad, M. M., Morris, D., & De Nahlik, C. (2009). Looking under the bonnet: Factors affecting student adoption of e-learning systems in Jordan. The International Review of Research in Open and Distributed Learning, 10(2), 1-25. https://doi.org/10.19173/irrodl.v10i2.596 Artino, A. R., & McCoach, D. B. (2008). Development and initial validation of the online learning value and self-efficacy scale. Journal of Educational Computing Research, 38(3), 279-303. https://doi.org/10.2190/EC.38.3.c Berigel, M., Kokoc, M., & Karalen, H. (2012). Exploring university students’ level of online learning acceptance. Eurasian Journal of Educational Research, 49/A, 275-288. https://ejer.com.tr/exploring- university-students-level-of-online-learning-acceptance/ Bhagat, K. K., Wu, L. Y., & Chang, C. (2016). Development and validation of the perception of students towards online learning (POSTOL). Educational Technology & Society, 19(1), 350-359. https://www.jstor.org/stable/jeductechsoci.19.1.350 Bolliger, D. U., & Wasilik, O. (2009). Factors influencing faculty satisfaction with online teaching and learning in higher education. Distance Education, 30(1), 103-116. https://doi.org/10.1080/01587910902845949 Bradford, G. R. (2011). A relationship study of student satisfaction with learning online and cognitive load: Initial results. The Internet and Higher Education, 14(4), 217-226. https://doi.org/10.1016/j.iheduc.2011.05.001 Chen, L., Zhang, R., & Liu, C. (2014). Listening strategy use and influential factors in web-based computer assisted language learning. Journal of Computer Assisted Learning, 30(3), 207-219. https://doi.org/10.1111/jcal.12041 Cheng, G., & Chau, J. (2016). Exploring the relationships between learning styles, online participation, learning achievement and course satisfaction: An empirical study of a blended learning course. British Journal of Educational Technology, 47(2), 257-278. https://doi.org/10.1111/bjet.12243 Choy, J. L. F., & Quek, C. L. (2016). Modelling relationships between students’ academic achievement and community of inquiry in an online learning environment for a blended course. Australasian Journal of Educational Technology, 32(4), 106-124. https://doi.org/10.14742/ajet.2500 Ellis, R. A., Ginns, P., & Piggott, L. (2009). E-learning in higher education: Some key aspects and their relationship to approaches to study. Higher Education Research & Development, 28(3), 303-318. https://doi.org/10.1080/07294360902839909 Fu, F., Su, R., & Yu, S. (2009). Egameflow: A scale to measure learners’ enjoyment of e-learning games. Computers & Education, 52(1), 101-112. https://doi.org/10.1016/j.compedu.2008.07.004 Gay, G. H. E. (2016). An assessment of online instructor e-learning readiness before, during, and after course delivery. Journal of Computing in Higher Education, 28(2), 199-220. https://doi.org/10.1007/s12528-016-9115-z Gómez-Rey, P., Barbera, E., & Fernández-Navarro, F. (2016). Measuring teachers and learners' perceptions of the quality of their online learning experience. Distance Education, 37(2), 146-163. https://doi.org/10.1080/01587919.2016.1184396 González-Gómez, F., Guardiola, J., Martín Rodríguez, Ó., & Montero Alonso, M. Á. (2012). Gender differences in e-learning satisfaction. Computers & Education, 58(1), 283-290. https://doi.org/10.1016/j.compedu.2011.08.017 Hung, M. (2016). Teacher readiness for online learning: Scale development and teacher perceptions. Computers & Education, 94, 120-133. https://doi.org/10.1016/j.compedu.2015.11.012 Hung, M., & Chou, C. (2015). Students’ perceptions of instructors’ roles in blended and online learning environments: A comparative study. Computers & Education, 81, 315-325. https://doi.org/10.1016/j.compedu.2014.10.022 Hung, M., Chou, C., Chen, C., & Own, Z. (2010). Learner readiness for online learning: Scale development and student perceptions. Computers & Education, 55(3), 1080-1090. https://doi.org/10.1016/j.compedu.2010.05.004 Islam, A. K. M. N. (2013). Investigating e-learning system usage outcomes in the university context. Computers & Education, 69, 387-399. https://doi.org/10.1016/j.compedu.2013.07.037 Jansen, R. S., van Leeuwen, A., Janssen, J., Kester, L., & Kalz, M. (2017). Validation of the self- regulated online learning questionnaire. Journal of Computing in Higher Education, 29(1), 6-27. https://doi.org/10.19173/irrodl.v10i2.596 https://doi.org/10.2190/EC.38.3.c https://doi.org/10.1080/01587910902845949 https://doi.org/10.1016/j.iheduc.2011.05.001 https://doi.org/10.1111/jcal.12041 https://doi.org/10.1111/bjet.12243 https://doi.org/10.14742/ajet.2500 https://doi.org/10.1080/07294360902839909 https://doi.org/10.1016/j.compedu.2008.07.004 https://doi.org/10.1007/s12528-016-9115-z https://doi.org/10.1080/01587919.2016.1184396 https://doi.org/10.1016/j.compedu.2011.08.017 https://doi.org/10.1016/j.compedu.2015.11.012 https://doi.org/10.1016/j.compedu.2014.10.022 https://doi.org/10.1016/j.compedu.2010.05.004 https://doi.org/10.1016/j.compedu.2013.07.037 Australasian Journal of Educational Technology, 2021, 38(1). 199 https://doi.org/10.1007/s12528-016-9125-x Kang, M., & Im, T. (2013). Factors of learner-instructor interaction which predict perceived learning outcomes in online learning environment. Journal of Computer Assisted Learning, 29(3), 292-301. https://doi.org/10.1111/jcal.12005 Kao, C., Wu, Y., & Tsai, C. (2011). Elementary school teachers’ motivation toward web-based professional development, and the relationship with internet self-efficacy and belief about web-based learning. Teaching and Teacher Education, 27(2), 406-415. https://doi.org/10.1016/j.tate.2010.09.010 Keramati, A., Afshari-Mofrad, M., & Kamrani, A. (2011). The role of readiness factors in E-learning outcomes: An empirical study. Computers & Education, 57(3), 1919-1929. https://doi.org/10.1016/j.compedu.2011.04.005 Kocdar, S., Karadeniz, A., Bozkurt, A., & Buyuk, K. (2018). Measuring self-regulation in self-paced open and distance learning environments. International Review of Research in Open and Distributed Learning, 19(1), 25-43. https://doi.org/10.19173/irrodl.v19i1.3255 Koo, A. (2008). Factors affecting teachers’ perceived readiness for online collaborative learning: A case study in Malaysia. Educational Technology & Society, 11(1), 266-278. https://www.jstor.org/stable/jeductechsoci.11.1.266 Korkmaz, Ö. (2012). A validity and reliability study of the online cooperative learning attitude scale (OCLAS). Computers & Education, 59(4), 1162-1169. https://doi.org/10.1016/j.compedu.2012.05.021 Lee, I. (2002). Gender differences in self-regulated on-line learning strategies within Korea's university context. Educational Technology Research and Development, 50(1), 101-111. https://doi.org/10.1007/BF02504967 Lee, M., & Tsai, C. (2005). Exploring high school students’ and teachers’ preferences toward the constructivist internet-based learning environments in Taiwan. Educational Studies, 31(2), 149-167. https://doi.org/10.1080/03055690500095522 Lee, S. J., Srinivasan, S., Trail, T., Lewis, D., & Lopez, S. (2011). Examining the relationship among student perception of support, course satisfaction, and learning outcomes in online learning. The Internet and Higher Education, 14(3), 158-163. https://doi.org/10.1016/j.iheduc.2011.04.001 Lee, S. W. (2013). Investigating students’ learning approaches, perceptions of online discussions, and students’ online and academic performance. Computers & Education, 68, 345-352. https://doi.org/10.1016/j.compedu.2013.05.019 Levy, Y. (2008). An empirical development of critical value factors (CVF) of online learning activities: An application of activity theory and cognitive value theory. Computers & Education, 51(4), 1664- 1675. https://doi.org/10.1016/j.compedu.2008.04.003 Liaw, S. (2008). Investigating students’ perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the blackboard system. Computers & Education, 51(2), 864-873. https://doi.org/10.1016/j.compedu.2007.09.005 Lin, C. (2018). The development of an instrument to measure the project competences of college students in online project-based learning. Journal of Science Education and Technology, 27(1), 57-69. https://doi.org/10.1007/s10956-017-9708-y Lu, H., & Chiou, M. (2010). The impact of individual differences on e-learning system satisfaction: A contingency approach. British Journal of Educational Technology, 41(2), 307-323. https://doi.org/10.1111/j.1467-8535.2009.00937.x Ngan, S., & Law, K. M. Y. (2015). Exploratory network analysis of learning motivation factors in e- learning facilitated computer programming courses. The Asia-Pacific Education Researcher, 24(4), 705-717. https://doi.org/10.1007/s40299-014-0223-0 Sarrab, M., Al Shibli, I., & Badursha, N. (2016). An empirical study of factors driving the adoption of mobile learning in Omani higher education. The International Review of Research in Open and Distributed Learning, 17(4), 331-349. https://doi.org/10.19173/irrodl.v17i4.2614 Selim, H. M. (2007). Critical success factors for e-learning acceptance: Confirmatory factor models. Computers & Education, 49(2), 396-413. https://doi.org/10.1016/j.compedu.2005.09.004 Sharma, S. K., Sarrab, M., & Al-Shihi, H. (2017). Development and validation of mobile learning acceptance measure. Interactive Learning Environments, 25(7), 847-858. https://doi.org/10.1080/10494820.2016.1224250 Slagter Van Tryon, P. J., & Bishop, M. J. (2012). Evaluating social connectedness online: The design and development of the social perceptions in learning contexts instrument. Distance Education, 33(3), 347-364. https://doi.org/10.1080/01587919.2012.723168 https://doi.org/10.1007/s12528-016-9125-x https://doi.org/10.1111/jcal.12005 https://doi.org/10.1016/j.tate.2010.09.010 https://doi.org/10.1016/j.compedu.2011.04.005 https://doi.org/10.19173/irrodl.v19i1.3255 https://doi.org/10.1016/j.compedu.2012.05.021 https://doi.org/10.1007/BF02504967 https://doi.org/10.1080/03055690500095522 https://doi.org/10.1016/j.iheduc.2011.04.001 https://doi.org/10.1016/j.compedu.2013.05.019 https://doi.org/10.1016/j.compedu.2008.04.003 https://doi.org/10.1016/j.compedu.2007.09.005 https://doi.org/10.1007/s10956-017-9708-y https://doi.org/10.1111/j.1467-8535.2009.00937.x https://doi.org/10.1007/s40299-014-0223-0 https://doi.org/10.19173/irrodl.v17i4.2614 https://doi.org/10.1016/j.compedu.2005.09.004 https://doi.org/10.1080/10494820.2016.1224250 https://doi.org/10.1080/01587919.2012.723168 Australasian Journal of Educational Technology, 2021, 38(1). 200 Tekinarslan, E. (2009). Turkish university students’ perceptions of the world wide web as a learning tool: An investigation based on gender, socio-economic background, and web experience. The International Review of Research in Open and Distributed Learning, 10(2), 1-19. https://doi.org/10.19173/irrodl.v10i2.598 Teo, T. (2010). Development and validation of the e-learning acceptance measure (ELAM). The Internet and Higher Education, 13(3), 148-152. https://doi.org/10.1016/j.iheduc.2010.02.001 Yang, F., & Tsai, C. (2008). Investigating university student preferences and beliefs about learning in the web-based context. Computers & Education, 50(4), 1284-1303. https://doi.org/10.1016/j.compedu.2006.12.009 Yuen, A. H. K., & Ma, W. W. K. (2008). Exploring teacher acceptance of e-learning technology. Asia- Pacific Journal of Teacher Education, 36(3), 229-243. https://doi.org/10.1080/13598660802232779 Zheng, C., Liang, J., Yang, Y., & Tsai, C. (2016). The relationship between Chinese university students’ conceptions of language learning and their online self-regulation. System, 57, 66-78. https://doi.org/10.1016/j.system.2016.01.005 Zhou, J. (2017). Exploring the factors affecting learners’ continuance intention of MOOCs for online collaborative learning: An extended ECM perspective. Australasian Journal of Educational Technology, 33(5), 123-135. https://doi.org/10.14742/ajet.2914 https://doi.org/10.19173/irrodl.v10i2.598 https://doi.org/10.1016/j.iheduc.2010.02.001 https://doi.org/10.1016/j.compedu.2006.12.009 https://doi.org/10.1080/13598660802232779 https://doi.org/10.1016/j.system.2016.01.005 https://doi.org/10.14742/ajet.2914 Australasian Journal of Educational Technology, 2021, 38(1). 201 Appendix 2 Reliability and validity of the reviewed questionnaires Year First author Questionnaire N 1 2 3 4 5 6 7 8 EFA CFA 2018 Lin, C. The knowledge integration, project skills, and self-efficacy scales (KIPSSE) 194 √ √ √ √ √ √ 2018 Kocdar, S. The self-regulation in self-paced open and distance learning environments 1279 √ √ √ √ √ √ 2017 Zhou, J. The factors influencing learners' continuance intention of MOOCS for online collaborative learning 435 √ √ √ √ √ √ √ 2017 Jansen, R. S. The self-regulated online learning questionnaire 154+153 √ √ √ 2017 Sharma, S. K. Mobile learning acceptance measure 806 √ √ √ √ √ √ 2016 Sarrab, M. Adoption of mobile learning 806 √ √ √ √ √ 2016 Zheng, C. Online self-regulated English learning (OSEL) 401 √ √ √ 2016 Hung, M. Teacher readiness for online learning measure (TROLM) 128+248 √ √ √ √ √ 2016 Cheng, G. Scale of learning styles, online participation, learning achievement and course satisfaction 78 √ √ √ √ 2016 Choy, J. L. F. Scale for academic achievement and community of inquiry in an online learning environment 227 √ √ √ 2016 Gay, G. H. E. An electronic learning systems (ELSS) success section and an e- readiness section 208 √ √ 2016 Gómez- Rey, P. The online learning quality index based on teachers and learners’ perceptions (OLQ-TLP) 153+709 √ √ Australasian Journal of Educational Technology, 2021, 38(1). 202 Year First author Questionnaire N 1 2 3 4 5 6 7 8 EFA CFA 2016 Bhagat, K. K. Perception of students towards online learning (POSTOL) 208 √ √ √ √ √ √ √ 2015 Ngan, S. Learning motivation factors in e-learning 365 √ √ √ 2015 Hung, M. Online instructor role and behavior scale (OIRBS) 750 √ 2014 Chen, L. The students’ strategy use, learning motivation, styles, and anxiety in the web-based computer assisted language learning (CALL) system 82 √ √ 2013 Kang, M. Learner-instructor interaction and learning outcomes in online learning environment 654 √ √ 2013 Islam, A. K. M. N. Perceived ease of use, usefulness, academic performance, community building assistance, and learning assistance, and E-learning system use 249 √ √ √ 2013 Lee, S. W. The perceptions of asynchronous online discussion questionnaire (PAOD) 111+204 √ √ √ 2012 Korkmaz, Ö. The online cooperative learning attitude scale (OCLAS) 599+242 √ √ √ √ √ √ √ 2012 Slagter Van Tryon, P. J. The social perceptions in learning contexts instrument (SPLCI) 50 √ √ √ √ 2012 González- Gómez, F. Questions asked to the pupils in e-learning courses in university of Granada 1185 2012 Berigel, M. Online learning acceptance scale 1416 √ √ √ √ 2011 Keramati, A. The role of readiness factors in E-learning outcomes scale 96 √ √ √ 2011 Bradford, G. R. Student satisfaction with learning online and cognitive load 1401 √ √ Australasian Journal of Educational Technology, 2021, 38(1). 203 Year First author Questionnaire N 1 2 3 4 5 6 7 8 EFA CFA 2011 Kao, C. The motivation toward web-based professional development survey (MWPD), the internet self-efficacy survey (ISS), the beliefs about web- based learning survey (BWL) 484 √ √ 2011 Lee, S. J. Survey on students’ perception of support and course satisfaction in an online course 110 √ 2010 Hung, M. Student’s online learning readiness scale (OLRS) 1051 √ √ √ √ √ 2010 Lu, H. E-learning satisfaction 522 √ √ √ √ 2010 Teo, T. E-learning acceptance measure 197+189 √ √ √ √ √ √ 2009 Fu, F. Learner’s enjoyment of e-learning games (eGameFlow) 166 √ √ √ √ √ √ √ 2009 Ellis, R. A. The student experience of e-learning 200 √ √ 2009 Abbad, M. M. Factors affecting student adoption of e-learning systems 470 √ √ √ 2009 Bolliger, D. U. The online faculty satisfaction survey (OFSS) 102 √ √ 2009 Tekinarslan , E. The attitude scale toward the Web 722 √ √ 2008 Koo, A. Teachers’ perceived readiness for online collaborative learning 86 √ √ √ 2008 Levy, Y. The critical value factors (CVFS) of the online learning activities 209 √ √ 2008 Artino, A. R. Online learning value and self-efficacy scale (OLVSES) 204+646+481 √ √ √ √ 2008 Yang, F. The web-based learning environment preference (WLEP) questionnaire The belief about web-based learning (BWL) questionnaire 475 √ √ 2008 Yuen, A. H. K. Teacher acceptance of e-learning technology 152 √ √ √ √ Australasian Journal of Educational Technology, 2021, 38(1). 204 Year First author Questionnaire N 1 2 3 4 5 6 7 8 EFA CFA 2008 Liaw, S. Attitudes toward e-learning 424 √ 2007 Selim, H. M. E-learning critical success factors (CSF) 70+538 √ 2005 Lee, M. The constructivist internet-based learning environment survey improvement (CILESI) 630+78 √ √ √ 2002 Lee, I. Self-regulated online learning strategies 156 √ √ Note. Columns: 1. internal consistent reliability, 2. test-retest reliability, 3. composite reliability, 4. content validity, 5. construct validity, 6. criterion-related validity, 7. convergent validity, 8. discriminant validity