Australasian Journal of Educational Technology, 2019, 35(4). 109 Development and validation of an instrument to measure students’ perceptions of technology-enabled active learning Ronnie Homi Shroff, The Hong Kong Polytechnic University, Hong Kong Fridolin Sze Thou Ting and Wai Hung Lam Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong This article reports on the design, development, and validation of a new instrument, the Technology-Enabled Active Learning Inventory (TEAL), to measure students’ perceptions of active learning in a technology-enabled learning context. By laying the theoretical foundation, a conceptual framework for technology-enabled active learning was developed. The conceptual framework formed the basis of the instrument development process including the design, development and validation of TEAL to measure students’ perceptions of active learning in a technology-enabled learning context. The self-reporting questionnaire consisted of four scales: interactive engagement, problem-solving skills, interest and feedback. All scales were assessed on a 7-point Likert scale. The survey items were designed to measure the four aspects of technology-enabled active learning and were verified by two panels using a formalised card sorting procedure as well as confirmatory factor analysis of a small-scale (n = 61) pilot survey. The TEAL questionnaire demonstrated internal consistency. Reliability as measured by Cronbach’s coefficient alpha ranged from 0.83 to 0.88 indicating good reliability and internal consistency of the items. The resultant instrument is a valid and reliable instrument that can be used in future research to gather and represent data on students’ perceptions of active learning in a technology-enabled learning context. Introduction Active learning has been a topic of intense research in education literature and demonstrated to be a key element in student learning, mainly related to the adoption and integration of technology in teaching and learning contexts (Dori & Belcher, 2005; Hassan, Puteh, & Buhari, 2015; Keengwe, 2014). Research has shown that student-centred strategies such as active learning are more effective than traditional lecture- based teaching models (Chiu & Cheng, 2017; Kinoshita, Knight, & Gibbes, 2017; Park & Choi, 2014). Learning takes place when students actively acquire new information and experiences and form their own interpretations (Chi & Wylie, 2014). Active learning gives learners the opportunity to participate in and take control of their own learning processes (Marton, 2018). Incorporating active learning approaches into a classroom setting results in a powerful model for teaching and learning because active learning supports the instructional process by enabling students to participate in engaging activities that reinforce their learning in meaningful ways. Moreover, active learning approaches are learner centred, as they engage students in learning, thus supporting a learning setting of immersion, exploration and reflection (Noteborn, Dailey-Hebert, Carbonell, & Gijselaers, 2014). Faculty who employ active learning approaches are able to give students an opportunity to plan, examine, justify and reflect upon their ideas, thus allowing students to learn to think for themselves, while also being able to critically evaluate the world around them (Ní Raghallaigh & Cunniffe, 2013). Hence, active-learning approaches engage students in learning and stimulate higher thinking processes (Kim, Sharma, Land, & Furlong, 2013). Research has demonstrated that students engaged in active-learner–centred activities demonstrate higher levels of motivation towards their courses and are, therefore, more actively engaged in their learning (Pirker, Riffnaller-Schiefer, & Gütl, 2014; Su & Cheng, 2015). The basic premise of active learning involves focusing on reinforcing higher-order thinking skills and instructional techniques, requiring learners to actively participate in the ownership of their learning (Kim et al., 2013; Marton, 2018). However, the term active learning lacks a concise definition, even though it is used frequently in educational literature and educational research. Moreover, a major obstacle is the lack of universally accepted definitions and measurements as different researchers from different fields, such as education, social psychology, healthcare and engineering disciplines, provide different definitions of the term. Hence to date, there is no singular, concise definition of active learning within a research context or Australasian Journal of Educational Technology, 2019, 35(4). 110 educational landscape. However, in an attempt to avoid any ambiguities, it is possible to provide a set of relevant and generally accepted definitions, as summarised in Table 1 below. Table 1 Definitions of active learning and primary proponents of each definition Definitions of active learning Proponents Field “learning in which the learner uses opportunities to decide about aspects of the learning process.” “the extent to which the learner is challenged to use his or her mental abilities while learning.” (van Hout-Wolters, Simons, & Volet, 2000, p. 1) (van Hout-Wolters, et al., 2000, p. 1) Education “activities that involve the students in the learning process.” (Nagda, Gurin, & Lopez, 2003, p. 8) Social psychology “any instructional method that engages students in the learning process.” (Prince, 2004, p. 1) Engineering education “a philosophy of education based on the premise that students best internalize information when they are directly involved in their own learning.” (Greek, 1995, p. 2) Criminal justice education “engagement in meaningful tasks where students have ownership of the content.” (McCown, Driscoll, & Roop, 1996, p. 236) Educational psychology “an approach or methodology for learning that draws on, integrates and creatively synthesises numerous learning methods.” (Dewing, 2010, p. 274) Nursing and healthcare “instructional activities involving students in doing things and thinking about what they are doing; to be actively involved, students must engage in such higher order thinking tasks as analysis, synthesis, and evaluation.” (Bonwell & Eison, 1991, p. 2) Higher education “an educational process where high levels of learning interactions and mental involvement are initiated by the learner.” (Ren et al., 2015, p. 6) Engineering According to Hung, Tan, and Koh (2006), active learning is the process of learning whereby learners are accountable for their own as well as one another’s learning and by which the learners are “actively developing thinking/learning strategies and constantly formulating new ideas and refining them through their conversational exchanges with others” (p. 30). A key essential element of active learning is to actively engage students in deeper learning by fostering their ability to create new knowledge and apply the acquired knowledge and skills by demonstrating well-developed judgement and responsibility as learners (Ní Raghallaigh & Cunniffe, 2013). Moreover, research demonstrates that the use of an active learning methodology not only increases student engagement, but also improves student retention of material and subsequently develops students’ critical thinking and problem-solving skills (Kvam, 2000; Mumtaz & Latif, 2017). Considerations in using a developed instrument Although active learning has been widely studied and validated in educational research as a compelling reason to enhance student learning, we were unable to locate scales or inventories that were specifically related to active learning in technology-supported learning contexts. Currently, there exists no comprehensive instrument that measures the degree by which individual students view active learning in a technology-enabled learning context. These gaps or absences provided the catalyst for the development of Australasian Journal of Educational Technology, 2019, 35(4). 111 a reliable and valid instrument that could be used to gather and represent data on students’ perceptions of active learning in technology-enabled learning contexts. Hence, the primary objective of this article is to cover this gap by reporting on the development and validation of the Technology-Enabled Active Learning Inventory (TEAL) designed to measure students’ active learning in a technology-enabled learning context and to test the validity and reliability of the newly developed instrument. The value of developing and validating an instrument is its potential for improved student performance – hence, there are various practical reasons for developing an instrument. Firstly, technology-enabled active learning strategies support intellectual development and higher-order competencies such as critical thinking and problem- solving skills in technology rich contexts. One of the key goals of active learning is to enable students to use higher levels of cognitive functioning through cognitively deeper and richer learning experiences. Secondly, feedback from students as to the effectiveness of active learning in a technology-enabled learning context should allow for improvements in course design. Thirdly, development of an instrument may also inform future research regarding implications for theory and practice in active learning in technology- enabled learning contexts. Theoretical framework Active learning Active learning theory contends that learners are active participants in an active environment, building their own knowledge by interacting with other learners and engaging in self-regulatory activities (Kunselman & Johnson, 2004). Active learning approaches such as inquiry learning, problem-solving and discussion method and think-pair-share activities offer an opportunity for deeper understanding, thereby allowing students to learn the process of approaching a problem, applying equations and learning from their mistakes through reflection on their learning from different perspectives. Moreover, the prevalence of technology use by students makes a compelling case for faculty to use technologies that enable students to actively construct new knowledge through interactive engagement activities designed to promote conceptual understanding, thinking and reasoning skills (Bhagat & Huang, 2018; Nicol, Owens, Le Coze, MacIntyre, & Eastwood, 2017). Hence, active learning takes place through meaningful activities where a student is able to make connections to previous knowledge and apply this new knowledge to those activities. In an active learning setting, each student has the opportunity to participate in and contribute to an assigned task. Fitch (2004) presents an extensive literature review of active learning research studies in technology based contexts, demonstrating that “there is convincing evidence that interactivity is a critical part of any form of technology-based learning” (p. 72). The potential role of technology in supporting active learning strategies has been given fresh impetus by the emergence of mobile technologies that enable collaboration, problem- solving, cognitive engagement and inquiry-based discovery. Technology-enabled active learning A fundamental issue is that developments in technology, pedagogy and instruction are not fully integrated, so as to transform the learning landscape into one that is learner-centred and active. How are teachers expected to assess students’ responses to faculty’s use of active learning within a technology-enabled context? Many educators and administrators consider technology to be a means of automatic enhancement for teaching, learning and assessment. Moreover, there exists a major disparity in the understanding of the role and impact of technology used in today’s educational arena. Technology can provide the tools and resources with which to achieve the goals and objectives of promoting students’ active learning strategies (Nicol et al., 2017). For example, the use of a simple mobile application (i.e., app) to write mathematical expressions, draw a diagram or post questions in class, can not only enhance communication and dialogue, but also support student collaboration through textual dialogue, discussion and debate, thereby giving students the flexibility to post problems and receive inputs from their peers and instructors in class. It is no longer expected of the instructor to solve every problem or answer every question. Instead, students are held accountable for working with each other as well as with the instructor to solve problems, discuss unclear concepts and move on to more complex concepts. This is a valuable lesson for students to learn – to apply problem-solving and critical thinking skills to authentic challenges and situations (Herrington, Reeves, & Oliver, 2006; Shroff, Keyes, & Linger, 2015) Herrington, Oliver, and Reeves (2003) characterise authentic learning as having real-world relevance, whereby learners become immersed in real-world problem-solving activities in a collaborative based environment. By providing learners the opportunity to practise complex skills and ask questions, instructors afford them the opportunity to assess their students’ understanding and remediate important points in real time. Hence, Australasian Journal of Educational Technology, 2019, 35(4). 112 technology-enhanced active learning advances the notion that the learner takes an active role in the learning process through much deeper levels of learning by interacting actively with technology. (Lim & Tschopp- Harris, 2018). Moreover, educational technologies, for example, have the potential to support active learning by offering mobile learning tools that enable students to develop their abilities to think critically and problem-solve through manipulation of concepts that can be helpful for generating new ideas and synthesising these ideas into new understandings. The integration of technology into the classroom, particularly digital technologies such as mobile devices, tablets and social media platforms, is becoming increasingly common as a means of facilitating active learning approaches inside the classroom (Looi et al., 2010; Martin & Ertzberger, 2013). Where active learning has been implemented in undergraduate courses, numerous studies have demonstrated greater student-learning gains, as compared to courses where teachers employed the traditional means of giving lectures to the class (Freeman et al., 2014; Kim et al., 2013; Machemer & Crawford, 2007). Research has also demonstrated that active learning methods, when implemented correctly in or outside of the classroom setting, ensure positive student behaviour, facilitate learning and enhance student achievement (Broadbent & Poon, 2015; Han, Capraro, & Capraro, 2015). Moreover, research demonstrates a consistent relationship between motivation and achievement when students are engaged in setting their own behavioural expectations (Sharples, 2013). Furthermore, active learning instructional strategies such as problem- solving, questioning and providing prompt feedback have demonstrated enhanced engagement, greater retention of information and improved academic achievement (Freeman et al., 2014; Kvam, 2000). Prior research confirms that the proper application of active learning methodologies also results in greater retention and understanding, higher development of thinking and application skills and enhanced improvement of learner ability to collaborate with others (Kvam, 2000; Prince, 2004). Active learning has also been shown to have a positive effect on critical thinking and problem-solving skills (Grabinger, Dunlap, & Duffield, 1997; Kim et al., 2013). Moreover, research has conclusively demonstrated that active learning develops problem-solving proficiency and supports the desired learning outcomes (Sivan, Leung, Woon, & Kember, 2000; Zwaal & Otting, 2012). Active learning pedagogies designed to stimulate learner creativity and move learning from a receptive to an interactive mode markedly promote analysis and reflection, which are essential parts of learning, particularly in terms of applicability of knowledge (Abrami, 2001; Matsushita, 2018). Hence, active learning strategies go beyond recall by deeply engaging students through the use of authentic learning strategies to promote critical thinking and foster the development of higher-level learning skills. Based on a review of research on active learning, the following four domains of active learning were identified: social, cognitive, affective and evaluative strategies. The social domain of active learning is characterised by interactive engagement and interaction (Dori & Belcher, 2005). In the social domain, an individual’s knowledge is developed through social interaction (Adams, 2006). Research has shown that engaging interactively may lead to greater depth of knowledge and deeper conceptual understanding (Laurillard, 2002). The cognitive domain of active learning is characterised by purposeful activities such as problem-solving, which encourage students to construct knowledge, enabling them to make connections to previous knowledge, synthesise new information and apply new concepts and ideas (Chi & Wylie, 2014; Dori & Belcher, 2005). Active learning approaches require learners to go beyond memory in recalling or restating previously learned information and move towards more active learner–centred forms of learning such as those at the higher end of the spectrum of Bloom’s taxonomy, requiring learners to engage in deeper levels of thinking by applying, analysing, synthesising and evaluating information (Anderson & Krathwohl, 2001). Hence, active learning requires learners to engage in deeper processing strategies and higher cognitive engagement. The behavioural domain of active learning is characterised by interest in the activity itself, for example, out of curiosity, a sense of challenge and a desire for choices (Mozelius, Fagerström, & Söderquist, 2017). In the behavioural domain, interest is a key factor determining choice and for completing challenging tasks within an active learning context (Rotgans & Schmidt, 2011a,b). Finally, the evaluative domain of active learning is characterised by prompt feedback of assessment for learning (Martyn, 2007; Van den Bergh, Ros, & Beijaard, 2013). Towards a conceptual framework for technology-enabled active learning By laying the theoretical foundation, a conceptual framework for technology-enabled active learning was developed, comprising the four main domains: social, cognitive, behavioural and evaluative, previously Australasian Journal of Educational Technology, 2019, 35(4). 113 discussed and illustrated in Figure 1 below. The conceptual framework presented below forms the theoretical and methodological basis of the instrument development process presented in this article. Figure 1. Active learning conceptual framework in a technology-enabled context Based on the conceptual framework presented above, we established that the core elements of a technology- enabled active learning context are learner interactive engagement in the learning process, problem-solving skills that require greater cognitive complexity, activities that evoke interest and that require an exercise of judgement in the face of uncertainty, as well as activities that encourage feedback (see Table 2). Table 2 Active learning constructs, dimensions and technology-supported activities Domain Active learning constructs Dimensions of active learning Technology-supported activities Social Interactive engagement (ITR) Engagement/interaction Interacting with the features of the technology in a responsive manner Human–computer interaction Actively engaging with the user- interface in a way that promotes dialogue Interacting with peers through an engaging user interface Facilitating the exchange of information by engaging with content presented in diverse formats Cognitive Problem-solving skills (PRS) Critical thinking Generating ideas by contributing information from multiple viewpoints Analytic reasoning Analysing information, formulating independent judgements Articulating reasoned arguments through review SOCIAL Interactive engagement (ITR) EVALUATIVE Feedback (FEE) COGNITIVE Problem-solving skills (PRS) BEHAVIOURAL Interest (INT) Technology-Enabled Active Learning (TEAL) Australasian Journal of Educational Technology, 2019, 35(4). 114 Behavioural Interest (INT) Challenge Engaging in thought-provoking dialogue with points of view that challenge perspectives Curiosity Exploring various options when navigating the user interface Exerting effort in the face of difficulty by persisting at challenging tasks Evaluative Feedback (FEE) Evaluative feedback Receive timely feedback to improve performance Receiving inputs to keep track of performance Receiving feedback on progression We now move on to examine the following four constructs that fall under the four domains of active learning: interactive engagement, problem-solving skills, interest and feedback. Each construct is discussed below and forms the basis of the conceptual framework and measurement instrument used in this study. Interactive engagement Within the social domain of active learning is the construct of interactive engagement – this construct consists of the following two sub-scales central to active learning: social interaction and human–computer interaction. Within the social domain is the interaction construct, and it is the variable considered central to active learning in a technology-enabled context. Typically, both the engagement and interaction concepts are closely related. From a technology perspective, engagement is defined as learners’ active involvement and participation in purposeful mobile learning activities to achieve learning goals (Falcão, Mendes de Andrade e Peres, de Morais, & da Silva Oliveira, 2018). Thurmond (2003) defined interaction as “the learner’s engagement with the course content, other learners, the instructor, and the technological medium used in the course” (p. 4). Moreover, interaction is a defining variable of active learning, and numerous findings support the effectiveness of interaction in technology-based education (Huang & Liaw, 2018; Tawfik et al., 2018). Based on Vygotsky’s (1978) social learning theory, interaction performs a critical function in the process of cognitive development of a learner, since knowledge is constructed through an interaction process with others, influenced by the environment. Collaboration is an interactive process in which individuals work together by communicating and coordinating activities towards a shared goal (Alioon & Delialioğlu, 2017). In operationalising these two variables, our view of engagement revolves around students’ active interaction (i.e., learner–content interaction, learner–interface interaction) and collaboration (i.e., learner–peer interaction) and the technology that they are using. Through interaction, individuals develop dialogues within the structure of activities; as a result, active learning occurs. The distinct characteristic of interaction is its importance not only on engagement at the individual level, but also on group collaboration to achieve a common goal (Wang, Cheng, Chen, Mercer, & Kirschner, 2017). This interaction makes up a major component of the learner’s expectations to succeed at a given task. The adaptation and use of emerging and appropriate mobile learning technologies can support a broad range of learning activities to create meaningful learning experiences with respect to what students learn and how they demonstrate mastery (Chang, Liu, & Huang, 2017). From a technology- enhanced learning context, the learner interacts with the mobile tools through learner–content interaction, learner–interface interaction and learner–peer interaction. Problem-solving skills Within the cognitive domain of active learning is the construct of problem-solving – this construct consists of the following two sub-scales central to active learning: critical thinking and analytic reasoning. The instructional approaches underlying problem-solving methods have been established through a constructivist learning framework that suggests embedding learning in relevant and authentic activities, to construct shared meaning and to support multiple perspectives (Machumu & Zhu, 2017). A problem- solving–based active learning approach enables students to view problems with a deeper perspective, thereby undergoing deeper learning and developing their critical thinking and analytic reasoning processes. Critical thinking is the process of actively interpreting, analysing and evaluating all perceived information, Australasian Journal of Educational Technology, 2019, 35(4). 115 in order to make thoughtful decisions (Tsui, 2002). Analytic thinking, on the other hand, is defined as “developing the capacity to think in a thoughtful, discerning way, to solve problems, to analyse data and recall and use information” (Amer, 2005, p. 1). Analytic thinking is therefore a cognitive process characterised by logical reasoning, requiring the learner to identify or create a problem to solve and draw appropriate inferences and conclusions (Espey, 2018). For example, a technology-enhanced learning tool that is designed to support critical thinking could integrate questioning techniques, requiring learners to engage in analysis, synthesis and a process of evaluation. Interest Within the behavioural domain of active learning is the construct of interest – this construct consists of the following two sub-scales central to active learning: challenge and curiosity. As pertaining to learning behaviour, if learners display interest in performing a task or skill or are drawn to the challenges in a learning context, they will be more predisposed to exploring opportunities to engage in authentic and meaningful ways (Schraw & Lehman, 2001). Interest in a learning task or activity is the consequence of students’ recognising the captivating characteristics associated with a particular learning activity (Mitchell, 1993). Mitchell proposed that the construct of interest is conceptualised in relation to making a distinction between catching and holding interest. As indicated by Mitchell (p. 6), the change from “catching” to “holding” an individual’s interest is contingent upon the appropriate conditions for learning that make learning more meaningful and long-lasting for that individual, based upon his or her goals and motivational beliefs about attaining those goals (Nyman, 2017). This process involves facilitating cognitive dissonance and, subsequently, challenging the learner’s present cognitive schema (Blaschke, 2018). For example, one of the challenges encountered by participants engaged in discussions using social media is being able to use the digital interface to assimilate the amount of information being generated through textual dialogue. Consequently, the information is filtered according to its importance and relevance, thereby allowing for a vigorous exchange of views and for the discussion to stay tightly focused. Since online content is rarely organised in a linear fashion, part of the individual user’s challenge is to filter the information that is generated into some discerning structure (e.g., reorganising comments made, summarising or analysing the main contributions). Furthermore, technology-enabled active learning stimulates curiosity and a desire to resolve any incongruity. There have been numerous research studies conducted on curiosity, which is strictly an intrinsic drive characterised by exploration, investigation and learning (Berlyne, 1966; Oudeyer, Gottlieb, & Lopes, 2016; Shroff & Vogel, 2009). If learning is involved, it usually takes the form of exploration to satisfy curiosity. For curiosity to be effective, the role of the learning environment is to provide the learner with opportunities to probe knowledge and explore and discover relationships between concepts and ideas. A technology-enabled active learning context may increase an individual’s sense of curiosity, because the effort of engaging in the activity, for example, may place the individual in an active role of exploration, investigation and discovery, thereby enabling him or her to use the digital interface in meaningful ways (Verdejo et al., 2008). Feedback Within the evaluative domain of active learning is the feedback construct and evaluative feedback sub-scale considered central to active learning in a technology-enabled context. Feedback is an integral aspect of active learning and refers to any information that makes learners evaluate their own performance. Feedback is essential as it not only drives the learner towards the expected outcome(s) but correspondingly allows the learner to learn from his or her mistakes and to set goals for future practice. In playing a game-based app, for example, feedback is typically always instantaneous, specifically targeted towards the user to adapt his or her approach for more appealing results. Jung, Schneider, and Valacich (2010) established that the provision of evaluative feedback through points and clear goals such as levels and leader boards in an idea- generation activity afforded significant performance gains. Hence, this player feedback can be in the form of achievements, avatars, collections, levels, badges or quests and such positive feedback creates a sense of progression. The principal goal of this evaluative feedback provided is to continue to retain the users’ attention and give performance-oriented feedback at the end of each activity to increase the users’ motivation and engagement. Hence, evaluative feedback fulfils a purpose beyond notifying users regarding different variations to the game state (Hämäläinen, Niilo-Rämä, Lainema, & Oksanen, 2018). Moreover, feedback mechanisms can be strengthened by harnessing elements of game design, through the provision Australasian Journal of Educational Technology, 2019, 35(4). 116 of visual cues or a progress bar, thus helping learners to view their progress as they work through a number of tasks or activities (Aldemir, Celik, & Kaplan, 2018). Research methodology Instrument development process To provide a high degree of confidence in the constructs and item content as well as construct validity and reliability, the Moore and Benbasat (1991) instrument development process was carried out to create and test the survey instrument, since this instrument development process provides a high degree of confidence in the constructs and item content as well as construct validity and reliability. Based on Moore and Benbasat (1991), the following 3-stage development procedure helped clarify and refine the items and constructs of the survey instrument: (1) Item creation – creating a pool of items to match each construct definition. The objective of this stage was to ensure content validity. (2) Card sorting – using a total of four judges in multiple rounds to sort items into construct categories (scales) and consequently examining judges’ inter-rater reliabilities and their consistency in labelling these scales. (3) Instrument testing – administering the survey instrument to a small-scale pilot sample with the objective of checking scale reliability. The purpose of the pilot study was to test the instrument and to ensure that the respondents correctly understood the comprehensiveness of the survey instrument items. The pilot study finalised the development of the survey instrument by testing its validity and reliability (i.e., analysis of survey data). Item creation The goal of the item creation step was to ensure content validity of the measurement items in order to make sure that the instrument covered all the items to reflect the definition of the constructs that are proposed as part of the conceptual framework (see Figure 1) (Bohrnstedt, 1970). The items for the instrument were generated from the framework and literature described earlier. First, we generated an initial item pool for the various constructs. Then, items considered too narrow in focus and applicable only to a particular situation were removed. After the item pools were created, they were re-evaluated to eliminate those which appeared redundant or ambiguous (i.e., items which might load on more than one factor). Card sorting In order to ensure construct validity, by knowing the extent to which the constructs may be ambiguous, a card sorting procedure was performed following Moore and Benbasat’s (1991) development process. The objective of performing the two sorting rounds was to ensure construct validity, the first round being exploratory while the second was confirmatory. To successfully reach these goals, four judges were selected to arrange the respective items into construct categories by ranking how well the items fit into their respective construct definitions. In the first round, the judges were not informed about the labels or names of the underlying constructs, but were instead asked to provide their own labels and definitions for the constructs. In the second round, the judges created a matrix with construct definitions at the top of the columns and items listed as the rows and were instructed to sort the cards into the four predefined categories. Hence, confidence in the construct validity of the scales increased if the judges’ definitions matched the scale’s intent. To assess the reliability of the sorting conducted by the judges, we used two measurements. First, we measured the level of agreement in categorising all 20 items and four categories of items across all four judges at one time, using Cohen’s kappa (Maxwell, 1970). In the first round, the kappa scores averaged 0.80. The value for Kappa coefficient of 0.90 was higher than the value obtained in the first round, thereby indicating an excellent fit, based on the guidelines of Landis and Koch (1977) for interpreting the kappa coefficient. Australasian Journal of Educational Technology, 2019, 35(4). 117 A second measurement of validity and reliability was an item placement ratio, which measured how many items were placed by the panel of judges for each round within the target construct. This meant that we were able to measure the overall frequency with which the judges placed items within the intended theoretical constructs. Hence, four theoretical constructs comprising of 5 items were developed for each construct. With a panel of four judges, a theoretical total of 20 placements could be made for the four constructs. A matrix of item placements for the first round was created as shown in Table 3 and Table 4 below (including a N/A (not applicable) column, where judges could place items that they felt fit into none of the categories). Table 3 Matrix of item placement – judge’s classification of first round Interactive engagement (ITR) Problem- solving skills (PRS) Interest (INT) Feedback (FEE) N/A Total % Hits Interactive engagement (ITR) 15 2 1 2 0 20 75 Problem- solving skills (PRS) 2 16 1 1 0 20 80 Interest (INT) 1 2 15 2 0 20 75 Feedback (FEE) 2 2 2 14 0 20 70 Item placements: 80 Hits: 60 Overall hit ratio: 75% By examining the diagonal matrix (Table 3) indicating a theoretical maximum of 80 placements (4 constructs at 20 placements), a total of 60 hits was attained, demonstrating an overall placement hit ratio of 75%. Furthermore, examining each row provided an indication of how the items created to tap the particular constructs really being classified. For instance, the “Problem-solving skills” row shows that 16-item placements were within the target construct; however, in the “Feedback” row, only 70% (14/20) were within target. Hence, attention was given to those items that were off-diagonal and any items that were vague, poorly worded or tapped a non-intended construct were identified. Based on the placements made by the judges, the items were re-examined and any inappropriately worded or ambiguous items (i.e., fitting in more than one category) were subsequently reworded or rephrased. The revised items were then subjected to a second round with a new set of four judges. Thus, a second round of item placements was considered necessary to help us to further clarify and refine the items and constructs of the survey instrument (see Table 4). Table 4 Matrix of item placement – judge’s classification of second round Interactive engagement (ITR) Problem- solving skills (PRS) Interest (INT) Feedback (FEE) N/A Total % Hits Interactive engagement (ITR) 20 0 0 0 0 20 100 Problem- solving skills (PRS) 1 19 0 0 0 20 95 Interest (INT) 1 0 18 1 0 20 90 Feedback (FEE) 1 0 1 18 0 20 90 Item placements: 80 Hits: 75 Overall hit ratio: 94% Examination of the resulting item placement in the second round (Table 4) showed a higher agreement among the judges compared to the first round, indicating a significant improvement in item placement. Hence, the reworded items were accurately matched by all four judges in the second round. This led to an overall hit rate of 94%, demonstrating that all constructs obtained a high item placement ratio, thereby indicating a high degree of construct validity (Moore & Benbasat, 1991). Australasian Journal of Educational Technology, 2019, 35(4). 118 Instrument testing The research setting and activity A total of 139 (N = 139) undergraduate students enrolled in a first year one semester calculus course offered at the Hong Kong Polytechnic University constituted a sufficient pool of subjects and were considered an appropriate fit within the intent and objective of this study. The selection of this course was based on the following criteria. Firstly, this course provided a rich opportunity for applying Kahoot!, a game-based digital learning platform into the classroom. Secondly, game-based learning activities in the form of quizzes, discussions and surveys, employing Kahoot! were carefully structured into the design and organisation of the course. Course structure AMA1110 Basic Mathematics I – Calculus and Probability and Statistics is an undergraduate course offered by the Department of Applied Mathematics at the Hong Kong Polytechnic University. The course provides students with a clear understanding of the basic concepts and applications of elementary differential calculus with emphasis on the use of mathematical techniques in tackling practical problems in science and engineering. Upon completion of the course, students are able to apply analytical reasoning to solve problems in science and engineering and demonstrate abilities of logical and analytical thinking. The Kahoot! game-based platform is embedded into the course to engage students through problem-solving and critical thinking of mathematical concepts. Technology The Kahoot! game-based digital learning platform was selected to supplement this study. Using Kahoot! provided the instructor with an effective way to create and generate quizzes, discussions and surveys, to engage students in accomplishing tasks in a game play format. To begin with, the instructor created four to six Kahoot! questions per lecture based on mathematical concepts and problems that were reviewed in the lecture. After each topic or section in a lecture was completed, the instructor would ask a Kahoot! question based on the topic or section just covered. When playing Kahoot!, the instructor would first launch a Kahoot! game session, which in turn generated a unique game pin for each session. The students were required to go to the Kahoot! site (https://kahoot.it/) and enter the game pin to log in to the game session on their mobile device (tablets, smartphones, laptops). Once logged in, the objective of the students (individual or team based) is to attempt to answer a multiple-choice question correctly, and in the shortest amount of time to score the highest number of points. Firstly, the instructor posted a question, which was displayed on a screen together with several optional answers shown in various colours and corresponding graphical symbols. Secondly, students attempted to answer the question by selecting the correct colour and corresponding symbol associated with the correct answer. In between each question, a distribution was displayed by means of a scoreboard presented on the screen, showing how the students performed by revealing the team’s names or individual player’s nicknames and ranked scores of the top five players. Measurement scales The finalised instrument comprised two sections. Section I was developed to identify the demographic traits of the respondents. It contained demographic items such as academic year, gender, interaction and students’ self-assessment of using Kahoot! The questions in section II were constructed from an extensive review of literature and a conceptual framework on technology-enabled active learning. Our research model comprised of 20 items (see Table 6) that measured “interactive engagement” (5 items), “problem-solving skills” (5 items) “interest” (5 items) and “feedback” (5 items). The response scale for all items was a 7- point, positively packed Likert scale (Lam & Klockars, 1982) coded as 7: strongly agree; 6: moderately agree; 5: slightly agree; 4: neither agree nor disagree; 3: slightly disagree; 2: moderately disagree; 1: strongly disagree. Data collection A hard-copy version of TEAL wherein the order of items was randomised was distributed to 139 students to complete, with the help of the instructor facilitating the course. The collection of these questionnaires yielded 61 usable data responses, providing a response rate of 43%. A power test was also performed to determine the appropriate sample size necessary to produce a test of the appropriate power. The results demonstrated that a sample size of 61 is adequate to detect, with power equal to 0.80. With a sample size of 61, the study had a power of 0.77 to yield a statistically significant result, close within the 0.80 range, a https://kahoot.it/ Australasian Journal of Educational Technology, 2019, 35(4). 119 commonly accepted threshold in these analyses (Cohen, 1977). The data collected from the 61 responses was analysed to present evidence for the validity and reliability of the survey instrument. Results and analyses The analysis process followed the intent of the study. To begin with, the validity of model use in the context of the study was analysed. Having established validity and robust construct relationships, researchers’ data results were subsequently analysed. The results and analysis presented here serve to address the instrument development and instrument validation process, ensuring construct validity and item reliability—including content, convergent and discriminant validity. This section describes the findings of the confirmatory factor analysis and the reliability analysis of the four constructs. Descriptive statistics The descriptive statistics of the four constructs are shown in Table 5. All means are above the midpoint of 4.00. The standard deviations range from 0.96 to 1.32 indicating a narrow spread around the mean. Table 5 Summary of means and standard deviations Constructs Question Mean Std N Interactive engagement (ITR) 1 5.02 1.32 61 5 5.00 1.12 60 9 5.07 .97 60 13 5.18 1.218 61 17 5.10 1.274 61 Problem-solving skills (PRS) 2 5.15 1.03 61 6 4.92 .96 60 10 5.13 1.04 60 14 4.97 .966 61 18 4.93 1.133 60 Interest (INT) 3 5.17 1.01 60 7 4.88 1.06 58 11 5.00 1.16 60 15 5.03 1.048 61 19 5.12 1.091 60 Feedback (FEE) 4 5.08 1.20 61 8 5.05 1.28 61 12 5.05 1.023 61 16 5.03 1.154 61 20 5.18 1.073 61 Construct validity To test the construct validity of the items in the survey instrument, both exploratory factor analysis and confirmatory factor analysis (CFA) were conducted. The reliabilities of factors (for the items loading on each factor) were assessed using Cronbach’s (1951) alpha. Exploratory factor analysis using a principal axis factor method was conducted to determine the factor structure. All items demonstrated high loadings which ranged from .627 to .823. Table 6 shows the items, constructs and factor loadings of TEAL for the sample of 61 students, using the individual student as the unit of analysis. The results of the CFA determined that the scales were not only reliable, but also valid for the factors under study. Australasian Journal of Educational Technology, 2019, 35(4). 120 Table 6 Constructs, items and loading statistics Question Constructs Items Factor loading Interactive engagement (ITR) Using Kahoot! 1 ITR1 allowed me respond expediently to my actions, resulting in a fully responsive interaction .804 5 ITR2 enabled me to skilfully interact with the features in a responsive manner .762 9 ITR3 allowed me to actively engage with the user-interface in a way that promotes dialogue .748 13 ITR4 helped me to interact more effectively with peers through an engaging interface .751 17 ITR5 facilitated the exchange of information by engaging with content presented in diverse formats .749 Problem-solving skills (PRS) Using Kahoot! 2 PRS1 allowed me to methodically generate ideas by contributing information from multiple viewpoints .769 6 PRS2 enabled me to solve a problem systematically by taking into account different points of view .712 10 PRS3 encouraged me to think critically about the broader concepts related to the problem .822 14 PRS4 let me to analyse my own views and their wider contexts in order to draw firm conclusions .669 18 PRS5 allowed me to define the problem systematically by viewing it from different angles in an effort to find possible solutions .696 Interest (INT) Using Kahoot! 3 INT1 Allowed me to engage in thought-provoking dialogue with points of view that challenged my perspectives .699 7 INT2 encouraged me to explore a variety of different issues that I may not have otherwise considered .775 11 INT3 piqued my curiosity by exploring various options when navigating the user interface .627 15 INT4 held my attention by challenging me to look into issues that I may not have otherwise thought of .663 19 INT5 encouraged me to exert effort in the face of difficulty by persisting at tasks I found challenging .823 Feedback (FEE) Using Kahoot! 4 FEE1 allowed me to receive timely feedback that helped me improve my performance .740 8 FEE2 enabled me to receive inputs, so that I was able to keep track of my own performance .792 12 FEE3 allowed me to receive prompt feedback, so that I was aware of my own progression towards knowledge acquisition .632 16 FEE4 allowed me to receive prompt feedback, so that I was aware of my own progression towards mastery of my skills .795 20 FEE5 enabled me to receive responses that allow further understanding .746 The constructs were analysed using Cronbach’s (1951, 1970) alpha. All of the measures utilised in this study displayed excellent internal consistency, ranging from 0.83 to 0.88 (see Table 7), thereby exceeding the reliability estimates (α = 0.70) recommended by Nunnally (1967). Australasian Journal of Educational Technology, 2019, 35(4). 121 Table 7 Cronbach’s alpha reliability coefficient Construct Items Alpha Interactive engagement (ITR) 5 .88 Problem-solving skills (PRS) 5 .83 Interest (INT) 5 .85 Feedback (FEE) 5 .86 Convergent and discriminant validity Average variance extracted (AVE) of the respective constructs was over the threshold value of 0.50 or higher (J. F. Hair, Anderson, Babin, & Black, 2010). For this model, the AVEs ranged from .625 to .875; therefore, all constructs exhibited a high degree of convergent validity. Following Fornell and Larcker (1981), discriminant validity was demonstrated by verifying that the square root of the average variance extracted (diagonal elements in Table 8) is higher than the correlation between constructs (off-diagonal). Discriminant validity, as inferred from the results shown in Table 8, was not supported because the AVE by each construct was considerably less than the shared variance between them (Fornell & Larcker, 1981). Hence, a one-factor model could be implied, in which the 20 items could be assumed to be indicators of a single latent factor. Table 8 Assessment of convergent and discriminant validity Construct ITR PRS INT FEE Interactive engagement (ITR) .582 Problem-solving skills (PRS) .814 .541 Interest (INT) .875 .841 .591 Feedback (FEE) .675 .701 .625 .552 Note. Diagonal values (bold figures) are the square roots of the AVE. Off-diagonal values are the correlations between constructs. Table 9 displays a summary of the overall model fit measures. This model was determined to be valid, as indicated by the adequacy indices such as chi-square statistic χ2 (N = 61) = 258, p < 0.01. The chi-square statistic is an intuitive index for measurement of goodness-of-fit between data and model. As recommended by J. Hair, Anderson, Tatham, and Black (1998), several other fit indices are examined. According to Gefen, Straub, and Boudreau (2000) and Hair et al. (1998), goodness-of-fit index (GFI), comparative fit index (CFI) and normed fit index (NFI) are best if above 0.90 and demonstrate marginal acceptance if above 0.80, adjusted goodness-of-fit index (AGFI) above 0.80 and root mean square residual (RMR) below 0.05. Furthermore, these fit indices indicated that the proposed measurement model revealed a modest fit with the data collected. This study suggests that the model fit was reasonably adequate to assess the results for the structural model. Thus, we could move forward by examining the path coefficients of the structural model. Table 9 Goodness of fit measures Fit measures Values χ2 258.917 RMR .087 RMSEA .093 GFI .863 CFI .892 AGFI .810 NFI .749 Note. CFI, cut-off > .90 Australasian Journal of Educational Technology, 2019, 35(4). 122 Discussion, limitations and future directions In this study, by laying a theoretical foundation, a conceptual and methodological framework for technology-enabled active learning was developed together with a self-reported instrument, TEAL, to measure students’ perceptions of active learning in a technology-enabled learning context. The instrument was developed and verified using a formalised procedure (Moore & Benbasat, 1991). To test the construct validity of the items in the instrument, CFA was performed and reliability of constructs assessed using Cronbach's (1951) alpha. Construct validity focused on how well the variables chosen captured the essence of that construct. Our analyses demonstrated that the 20-item TEAL scale had good reliability and validity. Our findings revealed that all four constructs of the TEAL scale demonstrated very good internal consistency with Cronbach’s alpha ranging from 0.83 to 0.88. The goodness-of-fit indices for the model were: GFI = 0.863, AGFI = 0.810, CFI = 0.892, which meant that the GFI, AGFI and the CFI of the proposed measurement model were well in the range of the suggested value of 0.90 (Bentler & Bonnet, 1980). These fit indices indicate that the proposed measurement model is satisfactory and suggest a good fit to the data. Hence, the results suggest the model provides a parsimonious fit to the data. Overall, the findings indicate that the TEAL instrument could effectively be used to assess students’ perceptions of active learning in a technology-enabled learning context in terms of their interactive engagement (ITR), problem-solving skills (PRS), interest (INT) and feedback (FEE). There are several limitations of the present study findings that must be acknowledged to help drive future research. Firstly, future research could consider larger sample sizes. Secondly, the instrument used was self-administered and, therefore, students’ perceptions are a self-reported measure and may lack objectivity to a certain extent. Finally, this study does not take a sample size of members from every age group, socio- economic status or different ethnic groups and, therefore, the results cannot be generalised for the entire population. These limitations demonstrate that more behaviour-analytic research in educational settings may be warranted. Both the development of a conceptual framework for technology-enabled active learning and the construction of the TEAL inventory as a valid and reliable measurement tool provide important implications for further study in guiding new approaches to teaching and learning with technology. In order to gain a more robust understanding of technology-enabled active learning contexts, directions for future research studies could, for example, include an investigation of the causal relationships between the constructs (interactive engagement, problem-solving skills, interest and feedback) on performance or perceived learning outcomes. Moreover, future research studies could be conducted to understand the effects of the psychological construct of control on active learning using the theory of personal causation (deCharms, 1968) and the construct of perceived locus of causality (Rotter, 1966). For example, in a technology-enabled active learning context, learners are able to navigate, discover and exercise a sense of control. Hence students’ locus of control is thought to be an important variable that warrants investigation and could also extend the scope of future studies. Implications and conclusion The TEAL inventory and conceptual framework was developed based on the literature review. Each of the four scales exhibited comparatively strong factor structure, internal consistency and reliability. The instrument development research described in this article offers several contributions. The most notable contribution is the creation of an overall instrument to gather and represent data on students’ perceptions of active learning in a technology-enabled learning context. The instrument creation process included reviewing existing literature on active learning developed by other researchers, creating items and then undertaking an extensive scale development process. This was done by developing and verifying an instrument for measuring each of the four scales of the proposed model using a formalised procedure. To test the construct validity of the items in the instrument, CFA was performed to evaluate the validity of the four factors for use with students. The result is a parsimonious, 20-item instrument, comprising four scales, all with acceptable levels. Finally, another potential contribution of this study is a stronger theoretical basis that could be further used by the growing community of researchers and educators as a means of assessing students’ perceptions of active learning in technology-enabled learning contexts. Australasian Journal of Educational Technology, 2019, 35(4). 123 This study is of notable importance in that design, refinement and validation of the TEAL inventory provides us with a valid and reliable instrument for future research in assessing students’ perceptions of active learning in a technology-enabled learning context on a much larger scale. Since active learning is an important educational strategy, a reliable and valid instrument to measure students’ perceptions of active learning in a technology-enabled learning context is essential. Acknowledgements This work is supported by the project entitled “Developing Active Learning Pedagogies and Mobile Applications in University STEM Education” and is funded by the University Grants Committee of the Hong Kong Special Administrative Region with additional support from the Hong Kong Polytechnic University. References Abrami, P. C. (2001). Understanding and promoting complex learning using technology. Educational Research and Evaluation, 7(2-3), 113–136. https://doi.org/10.1076/edre.7.2.113.3864 Adams, P. (2006). Exploring social constructivism: Theories and practicalities. Education, 34(3), 243– 257. https://doi.org/10.1080/03004270600898893 Aldemir, T., Celik, B., & Kaplan, G. (2018). A qualitative investigation of student perceptions of game elements in a gamified course. Computers in Human Behavior, 78, 235–254. https://doi.org/10.1016/j.chb.2017.10.001 Alioon, Y., & Delialioğlu, Ö. (2017). The effect of authentic m‐learning activities on student engagement and motivation. British Journal of Educational Technology. https://doi.org/10.1111/bjet.12559 Amer, A. (2005). Analytical thinking. Cairo: Cairo University, Center for Advancement of Postgraduate Studies and Research in Engineering Sciences. Retrieved from http://www.pathways.cu.edu.eg/subpages/training_courses/C10-Analytical-EN.pdf Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s taxonomy of educational objectives. New York, NY: Longman. Bentler, P. M., & Bonnet, D. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), 588–606. https://doi.org/10.1037/0033-2909.88.3.588 Berlyne, D. E. (1966). Curiosity and exploration. Science, 153(3731), 25–33. Retrieved from https://www.jstor.org/stable/1719694 Bhagat, K. K., & Huang, R. (2018). Improving learners’ experiences through authentic learning in a technology-rich classroom. In T. W. Chang, R. Huang, & Kinshuk (Eds.) Authentic learning through advances in technologies (pp. 3–15). Springer: Singapore. https://doi.org/10.1007/978-981-10-5930- 8_1 Blaschke, L. M. (2018). Self-determined learning (heutagogy) and digital media creating integrated educational environments for developing lifelong learning skills. In D. Kergel, B. Heidkamp, P. Telléus, T. Rachwal, & S. Nowakowski (Eds.) The digital turn in higher education (pp. 129–140). Wiesbaden: Springer VS. https://doi.org/10.1007/978-3-658-19925-8_10 Bohrnstedt, G. W. (1970). Reliability and validity assessment in attitude measurement. In G. F. Summers (Ed.), Attitude measurement (pp. 80-99). Chicago, IL: Rand-McNally. Bonwell, C., & Eison, J. (1991). Active learning: Creating excitement in the classroom (ASHE-ERIC Higher Education Report No. 1). Washington, DC: The George Washington University. Broadbent, J., & Poon, W. (2015). Self-regulated learning strategies & academic achievement in online higher education learning environments: A systematic review. The Internet and Higher Education, 27, 1–13. https://doi.org/10.1016/j.iheduc.2015.04.007 Chang, W. H., Liu, Y. C., & Huang, T. H. (2017). Perceptions of learning effectiveness in M‐learning: Scale development and student awareness. Journal of Computer Assisted Learning, 3(35, 461–472 https://doi.org/10.1111/jcal.12192 Chi, M. T., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219–243. https://doi.org/10.1080/00461520.2014.965823 Chiu, P. H. P., & Cheng, S. H. (2017). Effects of active learning classrooms on student learning: a two- year empirical investigation on student perceptions and academic performance. Higher Education Research & Development, 36(2), 269–279. https://doi.org/10.1080/07294360.2016.1196475 https://doi.org/10.1076/edre.7.2.113.3864 https://doi.org/10.1080/03004270600898893 https://doi.org/10.1016/j.chb.2017.10.001 https://doi.org/10.1111/bjet.12559 https://doi.org/10.1037/0033-2909.88.3.588 https://www.jstor.org/stable/1719694 mailto:https://doi.org/10.1007/978-981-10-5930-8_1 mailto:https://doi.org/10.1007/978-981-10-5930-8_1 https://doi.org/10.1007/978-3-658-19925-8_10 https://doi.org/10.1016/j.iheduc.2015.04.007 https://doi.org/10.1111/jcal.12192 https://doi.org/10.1080/00461520.2014.965823 https://doi.org/10.1080/07294360.2016.1196475 Australasian Journal of Educational Technology, 2019, 35(4). 124 Cohen, J. (1977). Statistical power analysis for the behavioral sciences (Rev. ed.). Hillsdale, NJ: Lawrence Erlbaum. Cronbach, L. (1951). Coefficient alpha and the internal consistency of tests. Psychometrika, 16(3), 297– 334. https://doi.org/10.1007/BF02310555 Cronbach, L. (1970). Essentials of psychological testing (3rd ed.). New York, NY: Harper & Row. deCharms, R. (1968). Personal causation. New York, NY: Academic Press. Dewing, J. (2010). Moments of movement: Active learning and practice development. Nurse Education in Practice, 10(1), 22–26. https://doi.org/10.1016/j.nepr.2009.02.010 Dori, Y. J., & Belcher, J. (2005). How does technology-enabled active learning affect undergraduate students’ understanding of electromagnetism concepts? The Journal of the Learning Sciences, 14(2), 243–279. https://doi.org/10.1207/s15327809jls1402_3 Espey, M. (2018). Enhancing critical thinking using team-based learning. Higher Education Research & Development, 37(1), 15–29. https://doi.org/10.1080/07294360.2017.1344196 Falcão, T. P., Mendes de Andrade e Peres, F. M., de Morais, D. C. S., & da Silva Oliveira, G. (2018). Participatory methodologies to promote student engagement in the development of educational digital games. Computers & Education, 116, 161–175. https://doi.org/10.1016/j.compedu.2017.09.006 Fitch, J. L. (2004). Student feedback in the college classroom: A technology solution. Educational Technology Research and Development, 52(1), 71–77. https://doi.org/10.1007/BF02504773 Fornell, C., & Larcker, D. F. (1981). Structural equation models with unobservable variables and measurement error: Algebra and statistics. Journal of Marketing Research, 18(3), 382–388. Retrieved from https://www.jstor.org/stable/10.2307/3150980 Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415. https://doi.org/10.1073/pnas.1319030111 Gefen, D., Straub, D., & Boudreau, M.-C. (2000). Structural equation modeling and regression: Guidelines for research practice. Communications of the Association for Information Systems, 4(1), Article 7. Retrieved from https://aisel.aisnet.org/cgi/viewcontent.cgi?article=2531&context=cais Grabinger, S., Dunlap, J. C., & Duffield, J. A. (1997). Rich environments for active learning in action: Problem-based learning. Research in Learning Technology, 5(2), 5–17. https://doi.org/10.1080/0968776970050202 Greek, C. E. (1995). Using active learning strategies in teaching criminology: A personal account. Journal of Criminal Justice Education, 6(1), 153–164. https://doi.org/10.1080/10511259500083381 Hair, J., Anderson, R., Tatham, R., & Black, W. (1998). Multivariate data analysis. Upper Saddle River, NJ: Prentice-Hall. Hair, J. F., Anderson, R. E., Babin, B. J., & Black, W. C. (2010). Multivariate data analysis: A global perspective. Upper Saddle River, NJ: Pearson. Hämäläinen, R. H., Niilo-Rämä, M., Lainema, T., & Oksanen, K. (2018). How to raise different game collaboration activities: The association between game mechanics, players’ roles and collaboration processes. Simulation & Gaming, 49(1), 50–71. https://doi.org/10.1177/1046878117752470 Han, S., Capraro, R., & Capraro, M. M. (2015). How science, technology, engineering, and mathematics (STEM) project-based learning (PBL) affects high, middle, and low achievers differently: The impact of student factors on achievement. International Journal of Science and Mathematics Education, 13(5), 1089–1113. https://doi.org/10.1007/s10763-014-9526-0 Hassan, N. F., Puteh, S., & Buhari, R. (2015). Student understanding through the application of technology enabled Active Learning in practical training. Procedia-Social and Behavioral Sciences, 204, 318–325. https://doi.org/10.1016/j.sbspro.2015.08.158 Herrington, J., Oliver, R., & Reeves, T. C. (2003). Patterns of engagement in authentic online learning environments. Australasian Journal of Educational Technology, 19(1), 59–71. https://doi.org/10.14742/ajet.1701 Herrington, J., Reeves, T. C., & Oliver, R. (2006). Authentic tasks online: A synergy among learner, task, and technology. Distance Education, 27(2), 233–247. https://doi.org/10.1080/01587910600789639 Huang, H.-M., & Liaw, S.-S. (2018). An analysis of learners’ intentions toward virtual reality learning based on constructivist and technology acceptance approaches. The International Review of Research in Open and Distributed Learning, 19(1), 91–115. https://doi.org/10.19173/irrodl.v19i1.2503 Hung, D., Tan, S. C., & Koh, T. S. (2006). Engaged learning: Making learning an authentic experience. In D Hung, & M. S. Khine (Eds.), Engaged learning with emerging technologies (pp. 29–48). Dordrecht: Springer. https://doi.org/10.1007/1-4020-3669-8_2 https://doi.org/10.1007/BF02310555 https://doi.org/10.1016/j.nepr.2009.02.010 https://doi.org/10.1207/s15327809jls1402_3 https://doi.org/10.1080/07294360.2017.1344196 https://doi.org/10.1016/j.compedu.2017.09.006 https://doi.org/10.1007/BF02504773 https://www.jstor.org/stable/10.2307/3150980 https://doi.org/10.1073/pnas.1319030111 https://aisel.aisnet.org/cgi/viewcontent.cgi?article=2531&context=cais https://doi.org/10.1080/0968776970050202 https://doi.org/10.1080/10511259500083381 https://doi.org/10.1177%2F1046878117752470 https://doi.org/10.1007/s10763-014-9526-0 https://doi.org/10.1016/j.sbspro.2015.08.158 https://doi.org/10.14742/ajet.1701 https://doi.org/10.1080/01587910600789639 https://doi.org/10.19173/irrodl.v19i1.2503 https://doi.org/10.1007/1-4020-3669-8_2 Australasian Journal of Educational Technology, 2019, 35(4). 125 Jung, J., Schneider, C., & Valacich, J. (2010). Enhancing the motivational affordance of information systems: The effects of real-time performance feedback and goal setting in group collaboration environments. Management Science, 56(4), 724–742. https://doi.org/10.1287/mnsc.1090.1129 Keengwe, J. (2014). Promoting active learning through the integration of mobile and ubiquitous technologies. Hershey, PA: IGI Global. Kim, K., Sharma, P., Land, S. M., & Furlong, K. P. (2013). Effects of active learning on enhancing student critical thinking in an undergraduate general science course. Innovative Higher Education, 38(3), 223–235. https://doi.org/10.1007/s10755-012-9236-x Kinoshita, T. J., Knight, D. B., & Gibbes, B. (2017). The positive influence of active learning in a lecture hall: an analysis of normalised gain scores in introductory environmental engineering. Innovations in Education and Teaching International, 54(3), 275–284. https://doi.org/10.1080/14703297.2015.1114957 Kunselman, J. C., & Johnson, K. A. (2004). Using the case method to facilitate learning. College Teaching, 52(3), 87–92. https://doi.org/10.3200/CTCH.52.3.87-92 Kvam, P. H. (2000). The effect of active learning methods on student retention in engineering statistics. The American Statistician, 54(2), 136–140. https://doi.org/10.1080/00031305.2000.10474526 Lam, T., & Klockars, A. J. (1982). Anchor point effects on the equivalence of questionnaire items. Journal of Educational Measurement, 19(4), 317–322. https://doi.org/10.1111/j.1745- 3984.1982.tb00137.x Landis, J. R., & Koch, C. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. https://www.jstor.org/stable/2529310 Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the effective use of learning technologies. London: Routledge. Lim, D. H., & Tschopp-Harris, K. (2018). Inverted constructivism to leverage mobile-technology-based active learning. In S. Keengwe (Ed.), Handbook of research on mobile technology, constructivism, and meaningful learning (pp. 240–258). Hershey, PA: IGI Global. Looi, C. K., Seow, P., Zhang, B., So, H. J., Chen, W., & Wong, L. H. (2010). Leveraging mobile technology for sustainable seamless learning: A research agenda. British Journal of Educational Technology, 41(2), 154–169. https://doi.org/10.1111/j.1467-8535.2008.00912.x Machemer, P. L., & Crawford, P. (2007). Student perceptions of active learning in a large cross- disciplinary classroom. Active Learning in Higher Education, 8(1), 9–30. https://doi.org/10.1177/1469787407074008 Machumu, H., & Zhu, C. (2017). The relationship between student conceptions of constructivist learning and their engagement in constructivist based blended learning environments. International Journal of Learning Technology, 12(3), 253–272. https://doi.org/10.1504/IJLT.2017.088408 Martin, F., & Ertzberger, J. (2013). Here and now mobile learning: An experimental study on the use of mobile technology. Computers & Education, 68, 76–85. https://doi.org/10.1016/j.compedu.2013.04.021 Marton, F. (2018). Towards a pedagogical theory of learning. In K. Matsushita (Ed.), Deep active learning (pp. 59–77). Singapore: Springer. https://doi.org/10.1007/978-981-10-5660-4_4 Martyn, M. (2007). Clickers in the classroom: An active learning approach. Educause Quarterly, 30(2), 71–74. Retrieved from https://er.educause.edu/articles/2007/4/clickers-in-the-classroom-an-active- learning-approach Matsushita, K. (2018). An invitation to deep active learning. In K. Matsushita (Ed.), Deep active learning (pp. 15–33). Singapore: Springer. https://doi.org/10.1007/978-981-10-5660-4_2 Maxwell, A. E. (1970). Basic statistics in behavioural research. Harmondsworth, UK: Penguin. McCown, R., Driscoll, M., & Roop, P. (1996). Educational psychology: A learning-centered approach to classroom management (2nd ed.). Boston, MA: Allyn and Bacon. Mitchell, M. (1993). Situational interest: Its multifaceted structure in the secondary school mathematics classroom. Journal of Educational Psychology, 85, 424–436. https://doi.org/10.1037/0022- 0663.85.3.424 Moore, G., & Benbasat, I. (1991). Development of an instrument to measure the perceptions of adopting an information technology innovation. Information Systems Research, 2(3), 192–222. https://doi.org/10.1287/isre.2.3.192 Mozelius, P., Fagerström, A., & Söderquist, M. (2017). Motivating factors and tangential learning for knowledge acquisition in educational games. Electronic Journal of e-Learning, 15(4), 343–354. Retrieved from http://www.ejel.org/issue/download.html?idArticle=606 https://doi.org/10.1287/mnsc.1090.1129 https://doi.org/10.1007/s10755-012-9236-x https://doi.org/10.1080/14703297.2015.1114957 https://doi.org/10.3200/CTCH.52.3.87-92 https://doi.org/10.1080/00031305.2000.10474526 https://doi.org/10.1111/j.1745-3984.1982.tb00137.x https://doi.org/10.1111/j.1745-3984.1982.tb00137.x https://doi.org/10.1111/j.1467-8535.2008.00912.x https://doi.org/10.1177%2F1469787407074008 https://doi.org/10.1504/IJLT.2017.088408 https://doi.org/10.1016/j.compedu.2013.04.021 https://doi.org/10.1007/978-981-10-5660-4_4 https://doi.org/10.1007/978-981-10-5660-4_2 https://doi.org/10.1037/0022-0663.85.3.424 https://doi.org/10.1037/0022-0663.85.3.424 https://doi.org/10.1287/isre.2.3.192 http://www.ejel.org/issue/download.html?idArticle=606 Australasian Journal of Educational Technology, 2019, 35(4). 126 Mumtaz, S., & Latif, R. (2017). Learning through debate during problem-based learning: An active learning strategy. Advances in Physiology Education, 41(3), 390–394. https://doi.org/10.1152/advan.00157.2016 Nagda, B. A., Gurin, P., & Lopez, G. E. (2003). Transformative pedagogy for democracy and social justice. Race, Ethnicity and Education, 6(2), 165-191. https://doi.org/10.1080/13613320308199 Ní Raghallaigh, M., & Cunniffe, R. (2013). Creating a safe climate for active learning and student engagement: an example from an introductory social work module. Teaching in Higher Education, 18(1), 93–105. https://doi.org/10.1080/13562517.2012.694103 Nicol, A. A., Owens, S. M., Le Coze, S. S., MacIntyre, A., & Eastwood, C. (2017). Comparison of high- technology active learning and low-technology active learning classrooms. Active Learning in Higher Education, 1–13. https://doi.org/10.1177/1469787417731176 Noteborn, G., Dailey-Hebert, A., Carbonell, K. B., & Gijselaers, W. (2014). Essential knowledge for academic performance: Educating in the virtual world to promote active learning. Teaching and Teacher Education, 37, 217–234. https://doi.org/10.1016/j.tate.2013.10.008 Nunnally, J. (1967). Psychometric theory. New York, NY: McGraw-Hill. Nyman, R. (2017). Interest and engagement: Perspectives on mathematics in the classroom (Doctoral dissertation). Retrieved from https://www.gu.se/english/research/publication?publicationId=252798 Oudeyer, P.-Y., Gottlieb, J., & Lopes, M. (2016). Intrinsic motivation, curiosity, and learning: Theory and applications in educational technologies. Brain Research, 229, 257–284. https://doi.org/10.1016/bs.pbr.2016.05.005 Park, E. L., & Choi, B. K. (2014). Transformation of classroom spaces: Traditional versus active learning classroom in colleges. Higher Education, 68(5), 749–771. https://doi.org/10.1007/s10734-014-9742-0 Pirker, J., Riffnaller-Schiefer, M., & Gütl, C. (2014). Motivational active learning: Engaging university students in computer science education. In Proceedings of the 19th Annual Conference on Innovation and Technology in Computer Science Education (pp. 297–302). New York, NY: ACM. https://doi.org/10.1145/2591708.2591750 Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223–231. https://doi.org/10.1002/j.2168-9830.2004.tb00809.x Ren, S., McKenzie, F. D., Chaturvedi, S. K., Prabhakaran, R., Yoon, J., Katsioloudis, P. J., & Garcia, H. (2015). Design and comparison of immersive interactive learning and instructional techniques for 3D virtual laboratories. Presence: Teleoperators and Virtual Environments, 24(2), 93–112. https://doi.org/10.1162/PRES_a_00221 Rotgans, J. I., & Schmidt, H. G. (2011a). The role of teachers in facilitating situational interest in an active-learning classroom. Teaching and Teacher Education, 27(1), 37–42. https://doi.org/10.1016/j.tate.2010.06.025 Rotgans, J. I., & Schmidt, H. G. (2011b). Situational interest and academic achievement in the active- learning classroom. Learning and Instruction, 21(1), 58–67. https://doi.org/10.1016/j.learninstruc.2009.11.001 Rotter, J. (1966). Generalised expectancies for internal versus external control of reinforcement. Psychological Monographs: General and Applied, 80(1), 1–28. https://doi.org/10.1037/h0092976 Schraw, G., & Lehman, S. (2001). Situational interest: A review of the literature and directions for future research. Educational Psychology Review, 13(1), 23–52. https://doi.org/10.1023/A:1009004801455 Sharples, M. (2013). Mobile learning: Research, practice and challenges. Distance Education in China, 3(5), 5–11. Retrieved from http://oro.open.ac.uk/id/eprint/37510 Shroff, R. H., Keyes, C., & Linger, W. (2015). A proposed taxonomy of theoretical and pedagogical perspectives of mobile applications to support ubiquitous learning. Ubiquitous Learning: An International Journal, 8(4), 23–44. https://doi.org/10.18848/1835-9795/CGP/v08i04/58074 Shroff, R. H., & Vogel, D. R. (2009). Assessing the factors deemed to support individual student intrinsic motivation in technology supported online and face-to-face discussions. Journal of Information Technology Education: Research, 8, 59–85. https://doi.org/10.28945/160 Sivan, A., Leung, R. W., Woon, C.-C., & Kember, D. (2000). An implementation of active learning and its effect on the quality of student learning. Innovations in Education and Teaching International, 37(4), 381–389. https://doi.org/10.1080/135580000750052991 Su, C. H., & Cheng, C. H. (2015). A mobile gamification learning system for improving the learning motivation and achievements. Journal of Computer Assisted Learning, 31(3), 268–286. https://doi.org/10.1111/jcal.12088 https://doi.org/10.1152/advan.00157.2016 https://doi.org/10.1080/13613320308199 https://doi.org/10.1080/13562517.2012.694103 https://doi.org/10.1177%2F1469787417731176 https://doi.org/10.1016/j.tate.2013.10.008 https://www.gu.se/english/research/publication?publicationId=252798 https://doi.org/10.1016/bs.pbr.2016.05.005 https://doi.org/10.1007/s10734-014-9742-0 https://doi.org/10.1145/2591708.2591750 https://doi.org/10.1162/PRES_a_00221 https://doi.org/10.1016/j.tate.2010.06.025 https://doi.org/10.1016/j.learninstruc.2009.11.001 https://doi.org/10.1037/h0092976 https://doi.org/10.1023/A:1009004801455 http://oro.open.ac.uk/id/eprint/37510 https://doi.org/10.18848/1835-9795/CGP/v08i04/58074 https://doi.org/10.28945/160 https://doi.org/10.1080/135580000750052991 https://doi.org/10.1111/jcal.12088 Australasian Journal of Educational Technology, 2019, 35(4). 127 Tawfik, A. A., Giabbanelli, P. J., Hogan, M., Msilu, F., Gill, A., & York, C. S. (2018). Effects of success v failure cases on learner-learner interaction. Computers & Education, 118, 120–132. https://doi.org/10.1016/j.compedu.2017.11.013 Thurmond, V. A. (2003). Defining interaction and strategies to enhance interactions in Web-based courses. Nurse Educator, 28(5), 237–241. Retrieved from https://journals.lww.com/nurseeducatoronline/Fulltext/2003/09000/Defining_Interaction_and_Strategi es_to_Enhance.13.aspx Tsui, L. (2002). Fostering critical thinking through effective pedagogy: Evidence from four institutional case studies. The Journal of Higher Education, 73(6), 740–763. https://doi.org/10.1080/00221546.2002.11777179 Van den Bergh, L., Ros, A., & Beijaard, D. (2013). Teacher feedback during active learning: Current practices in primary schools. British Journal of Educational Psychology, 83(2), 341–362. https://doi.org/10.1111/j.2044-8279.2012.02073.x van Hout-Wolters, B., Simons, R.-J., & Volet, S. (2000). Active learning: Self-directed learning and independent work. In R. J. Simons, J. van der Linden, & T. Duffy (Eds.), New learning (pp. 21–36). Dordrecht: Springer. https://doi.org/10.1007/0-306-47614-2_2 Verdejo, M., Celorrio, C., Lorenzo, E., Millán, M., Prades, S., & Vélez, J. (2008). Constructing mobile technology-enabled environments for an integrated learning approach. In H. Ryu, & D. Parsons (Eds.), Innovative mobile learning: Techniques and technologies (pp. 145-171). Hershey, PA: IGI Global. Vygotsky, L. (1978). Interaction between learning and development. Readings on the Development of Children, 23(3), 34–41. Retrieved from http://www.psy.cmu.edu/~siegler/vygotsky78.pdf Wang, M., Cheng, B., Chen, J., Mercer, N., & Kirschner, P. A. (2017). The use of web-based collaborative concept mapping to support group learning and interaction in an online environment. The Internet and Higher Education, 34, 28–40. https://doi.org/10.1016/j.iheduc.2017.04.003 Zwaal, W., & Otting, H. (2012). The impact of concept mapping on the process of problem-based learning. Interdisciplinary Journal of Problem-based Learning, 6(1), 104–128. https://doi.org/10.7771/1541-5015.1314 Corresponding author: Ronnie H. Shroff, ronnie.shroff@polyu.edu.hk Australasian Journal of Educational Technology © 2019. Please cite as: Shroff, R. H., Ting, F. S. T., & Lam, W. H. (2019). Development and validation of an instrument to measure students’ perceptions of technology-enabled active learning. Australasian Journal of Educational Technology, 35(4), 109-127. https://doi.org/10.14742/ajet.4472 https://doi.org/10.1016/j.compedu.2017.11.013 https://journals.lww.com/nurseeducatoronline/Fulltext/2003/09000/Defining_Interaction_and_Strategies_to_Enhance.13.aspx https://journals.lww.com/nurseeducatoronline/Fulltext/2003/09000/Defining_Interaction_and_Strategies_to_Enhance.13.aspx https://doi.org/10.1080/00221546.2002.11777179 https://doi.org/10.1111/j.2044-8279.2012.02073.x https://doi.org/10.1007/0-306-47614-2_2 https://doi.org/10.1016/j.iheduc.2017.04.003 https://doi.org/10.7771/1541-5015.1314 mailto:ronnie.shroff@polyu.edu.hk https://doi.org/10.14742/ajet.4472 Considerations in using a developed instrument Theoretical framework Active learning Technology-enabled active learning Towards a conceptual framework for technology-enabled active learning Problem-solving skills Interest Research methodology Instrument development process Item creation Card sorting Instrument testing The research setting and activity Course structure Measurement scales Discussion, limitations and future directions Implications and conclusion Acknowledgements References Ren, S., McKenzie, F. D., Chaturvedi, S. K., Prabhakaran, R., Yoon, J., Katsioloudis, P. J., & Garcia, H. (2015). Design and comparison of immersive interactive learning and instructional techniques for 3D virtual laboratories. Presence: Teleoperators... Rotgans, J. I., & Schmidt, H. G. (2011a). The role of teachers in facilitating situational interest in an active-learning classroom. Teaching and Teacher Education, 27(1), 37–42. https://doi.org/10.1016/j.tate.2010.06.025