Microsoft Word - BRAIN_vol_9_issue_special1_2018_v1l.doc 56 Success in Mobile and Ubiquitous Learning: Indicators of Effectiveness Billy T.M. Wong The Open University of Hong Kong Hong Kong SAR, China billywtm@gmail.com Abstract Mobile and ubiquitous learning has been a key focus in e-learning and practising broadly worldwide. Among the research publications in this area, a majority of them focused on evaluating the effectiveness of relevant practices and reported positive outcomes. To interpret such results, the contexts in which mobile devices were used for learning and the indicators of effectiveness adopted for evaluation are of prime importance. However, the use of those indicators in relation to the contexts of practice has not been adequately studied. This paper presents a systematic review on the use of various indicators of effectiveness for the practices of mobile and ubiquitous learning. The review covers a total of 50 cases from relevant literature for the period 2007–2016. Regarding the contexts of practice, the results show that 92% of the cases involved the use of mobile devices for accessing online or offline information; 40% involved social interaction among peers or between students and teachers; and 74% involved the apps or learning materials developed for specific courses. The indicators of effectiveness revolved around 10 categories, namely learning achievements, perceived usefulness, motivation, ease of use, satisfaction, learning attitude, cognitive load, system usage, self-efficacy, and social engagement. Results also show that studies of mobile and ubiquitous learning practices mostly focused on specific courses with less than 100 participants. Based on the findings, a number of limitations in interpreting the success of mobile learning practices are discussed. Keywords: mobile learning, ubiquitous learning, e-learning, education, indicators of effectiveness 1. Introduction As one of the key developments in technology-enhanced learning, mobile and ubiquitous learning has been playing an increasingly crucial role in education in the past decade. By removing geographical barriers and time constraints, it allows educational institutions to deliver flexible education and implement innovative pedagogies. The advances in mobile technology and improvements in network infrastructure have further motivated educational institutions worldwide to practise mobile and ubiquitous learning (UNESCO; 2013). To illustrate the overall trend in its popularity, the number of items on Google Scholar related to mobile and ubiquitous learning have increased from 2,570 in 2007 to 6,550 in 2011, and 9,540 in 20161 — a nearly 3 times increase in a decade. A majority of the studies on mobile and ubiquitous learning practices have focused on the extent of their effectiveness (Wu et al.; 2012), usually reported using a variety of indicators. Examples of common indicators include learners’ perceptions of usefulness (Alrasheedi and Capretz; 2014); learners’ attitudes (Wittich et al.; 2016); the frequency of use of a mobile device for learning (Shao and Seif; 2014); and learning performance (Teri et al.; 2013). They reveal various aspects in which mobile learning was considered to be effective in practice. Despite being widely used in the literature, these indicators of effectiveness have not been systematically studied. In previous reviews of mobile and ubiquitous learning, the indicators have sometimes been categorised together with success factors — the conditions required for ensuring the effectiveness of mobile learning. For example, Alrasheedi, Capretz and Raza (2015) identified 13 success factors which had a strong impact on the implementation of mobile and ubiquitous learning, 1 Searched on 12 November 2017. B. T. M. Wong - Success in Mobile and Ubiquitous Learning: Indicators of Effectiveness 57 including user perception and increased productivity. Yadegaridehkordi, Iahad and Baloch (2013) also found that perceived usefulness and attitude towards mobile and ubiquitous learning were two of the success factors. Although success factors and indicators of effectiveness are tightly related — that is, the causes of success and the effects of actions — they are different concepts and should be handled separately (e.g. Hilgarth; 2011). This paper presents a review of indicators of effectiveness in mobile and ubiquitous learning practices, showing how mobile technologies have been applied in a broad range of contexts, and the areas of effectiveness revealed in the practices. The findings of this study suggest indicators that can be used to measure the effectiveness of mobile and ubiquitous learning practices, and appropriate ways of interpreting them. 2. Relevant studies The effectiveness of mobile and ubiquitous learning can be evaluated from multiple perspectives. Examples include a socio-cultural perspective, such as the proportion of people in a society showing a positive attitude towards mobile and ubiquitous learning; an institutional perspective, such as the development of infrastructure for mobile technology in an institution; and an individual perspective, such as students’ learning experience. Accordingly, various indicators have been used for evaluating the success of mobile and ubiquitous learning practices. In a review of mobile and ubiquitous learning, Crompton, Burke, Gregory and Gräbe (2016) studied 49 practices of mobile and ubiquitous learning in science, showing that they fell into one or more of three categories — designing mobile learning systems, evaluating the effects, and investigating the affective domains of mobile and ubiquitous learning. The results highlighted the significance of evaluation in mobile and ubiquitous learning studies. The evaluation of learning effectiveness is also a major focus in meta-analyses of mobile and ubiquitous learning. For instance, Sung, Chang and Liu (2016) analysed 110 journal articles from 1993 to 2013 about the effects of mobile and ubiquitous learning, using an effect size measure. Their findings showed that learning with mobiles was in general more effective than conventional modes using only pen-and-paper or desktop computers. The effects would be influenced by variables of educational contexts, such as the learners’ ages, the teaching methods and the subject disciplines. Likewise, Zydney and Warner (2016) investigated articles on mobile apps for science learning, covering the mobile app design, underlying theoretical foundations, and students’ learning outcomes. They found that the most common outcome reported was basic scientific knowledge or conceptual understanding. Also, Wang, Liu and Hwang (2016) studied how mobile technologies have changed ubiquitous language learning in museums, focusing on system usefulness, activity usefulness and activity playfulness. The foci of evaluation reveal the areas of mobile and ubiquitous learning practices that can be assessed to show their effectiveness. The indicators of the effectiveness of mobile and ubiquitous learning have, however, only been studied indirectly, usually together with success factors for mobile and ubiquitous learning. Alrasheedi et al. (2015) examined the critical factors for the success of mobile learning in higher education from the university students’ perspective, covering 30 studies from 17 countries. As noted earlier, the 13 success factors identified included factors such as user perception and increase in productivity which have also been used as indicators of the effectiveness of mobile learning. Cardoso and Abreu (2015) analysed mobile learning publications on the types and environments of mobile learning practices, and the determining factors which promoted the acceptance of mobile learning by students and teachers. Among the factors identified, one was the changes in the processes of information access, which is an expected outcome after practising mobile learning as well as an indicator of its effectiveness. Previous studies therefore have not given a clear overall picture of the indicators used in evaluating the effectiveness of mobile and ubiquitous learning practices. This paper serves to contribute to this area through conducting a comprehensive review of them. BRAIN – Broad Research in Artificial Intelligence and Neuroscience, Volume 9, Special Issue on Educational Psychology (April, 2018), ISSN 2067-8957 58 3. Method This study aims to review the indicators of effectiveness in studies evaluating mobile and ubiquitous learning practices, as well as the contextual factors of the practices which may affect the evaluation results. The relevant literature was collected from Google Scholar using the keywords “mobile learning” or “ubiquitous learning”, together with “indicators of effectiveness” or “success factors” in the period 2007–2016. The literature was manually checked to include only studies reporting empirical practices of mobile and ubiquitous learning. A total of 50 cases were then collected, covering information such as countries, level of education, use of mobile devices for learning, and measures of the effectiveness of the practices. The information gathered was categorised into three areas: (1) contextual information on the mobile and ubiquitous learning practices, e.g. geographical region, educational level, number of participants, and level of intervention; (2) the use of mobile devices, e.g. the purpose of use, level of interactivity, and functions of the mobile devices; and (3) the indicators of effectiveness for the practices. 4. Results 4.1. Contexts of the mobile and ubiquitous learning practices The 50 cases of mobile and ubiquitous learning practices covered 14 countries/regions, including China, Japan, Taiwan, Korea, Sri Lanka, Turkey, Spain, Greece, the Netherlands, Britain, Australia, New Zealand, South Africa and the USA. Figure 1 shows the geographical distribution of the cases. Among the 50 cases, 70% were conducted in Asia, 16% in Europe, 4% in Oceania, 4% in Africa, and 2% in North America. Therefore, the results of this study represent more of the situation in Asia. Figure 1. Geographical distribution of the case studies Figure 2 shows the data collection methods applied in the studies. Surveys were used in most of the selected cases (94%), followed by interviews (38%) and experiments (34%). Apart from these approaches, observation (12%), field materials (8%), video recordings (6%) and discussion threads (2%) were also used in some of the studies. Thirty of the cases involved the use of more than one method, mostly combining a survey and another one or more method. These suggest that most of the studies involved quantitative data at least in part. Figure 2. Data collection methods of the studies (Note: A study may involve more than one method.) B. T. M. Wong - Success in Mobile and Ubiquitous Learning: Indicators of Effectiveness 59 Figure 3 presents the education levels of the mobile and ubiquitous learning practices. Most studies took place at the tertiary level (66.7%). Also, 25.5% of the studies were conducted at the primary school level and 7.8% at the secondary school level. Figure 3. Education level of the studies Figure 4 shows the number of participants in the studies, with the majority being on a small scale, having less than 100 participants (72%). Twenty percent of the studies involved over 100 but less than 1,000 participants, and only two cases included more than 1,000 participants (4%). Figure 4. Number of participants in the studies Figure 5 illustrates the levels of intervention of the studies. Most studies concentrated on the course level (76.9%) and some on either the programme level (13.5%) or the institutional level (9.6%). Two studies were conducted at more than one level. These results supplement the above number of participants where most of the studies were conducted on a small scale. Figure 5. Level of intervention of the studies 4.2. Uses of mobile devices for learning The applications of mobile devices for learning were multifaceted. They depended on the purposes of the mobile and ubiquitous learning practices, such as acquiring new knowledge, revising or practising what had been taught, seeking help from instructors, and communicating with peers. Figure 6 shows the purposes of mobile and ubiquitous learning in the studies. A majority of the cases were multi-purpose (54%). Those with a single specific purpose were focused on “practice or revision” (34%) and “knowledge acquisition” (10%). There was a case where the application was to help new students to become familiar with the campus and teach them about the use of the facilities. BRAIN – Broad Research in Artificial Intelligence and Neuroscience, Volume 9, Special Issue on Educational Psychology (April, 2018), ISSN 2067-8957 60 Figure 6. Purposes of using mobile devices for learning Figure 7 presents the level of interactivity in using mobile devices. Sixty percent of the cases involved only one-way access for information either online or offline, while 8% involved social interaction with information exchange among learners. Also 32% of the cases included both levels of interactivity in learning. Figure 7. Level of interactivity of the studies Figure 8 captures the functions of mobile devices used in the practices. The results show that tailor-made applications for specific practices were most common (74%), followed by the use of a speaker (32%) and a camera (26%), where learners had to listen to audio materials using speakers or access online information by scanning QR-codes through cameras. In the various practices, other functions were also used, such as messaging (16%) for interacting with diverse parties and GPS (14%) for outdoor learning activities. For the practices using older models of mobile devices without cameras, tools such as an RFID reader (8%) were used for accessing information via communication tags. Figure 8. Functions of the mobile devices used in the practices (Note: Each case could involve the use of more than one function.) B. T. M. Wong - Success in Mobile and Ubiquitous Learning: Indicators of Effectiveness 61 4.3. Indicators of effectiveness Figure 9 presents the indicators used for evaluating the effectiveness of mobile and ubiquitous learning practices. Learning achievements (64%) and perceived usefulness (56%) were the two most frequently used, followed by motivation (26%), ease of use (26%) and satisfaction (24%). Some indictors were used relatively less in the studies, such as learning attitude (14%), cognitive load (12%), system usage (8%), self-efficacy (6%) and social engagement (2%). Those indicators were usually adopted in the studies using qualitative methods for data collection. The results suggest a possible relationship between the data collection methods and the indicators. The choices of indicators represent, in principle, how the effectiveness of mobile and ubiquitous learning practices can be most appropriately evaluated and presented using particular study methods. Figure 9. Proportion of indicators used in the case studies 5. Discussion This study provides an overall picture of how the practices of mobile and ubiquitous learning have been evaluated for effectiveness. It summarises the indicators used in the evaluations, which are the most common type of mobile and ubiquitous learning study (Wu et al.; 2012). The results also supplement past efforts to figure out the factors promoting the adoption and facilitating effective implementation of mobile and ubiquitous learning (e.g. Alrasheedi et al.; 2015; Cochrane; 2014; Krotov; 2015; Yadegaridehkordi et al.; 2013). The contexts of mobile and ubiquitous learning practices in this study are in general consistent with those in the relevant review literature. For example, as in Wu et al. (2012), most practices were in higher education, followed by primary education. The results are also consistent with the finding of Sung et al. (2016) that websites or mobile apps developed for specific teaching and learning goals have been widely used. Those websites or apps account for most mobile and ubiquitous learning practices in this study, and so its results are representative of the practices. In the review of data collection methods, the dominant use of quantitative methods, especially surveys, corresponds with the findings in other technology-assisted learning contexts (Wu et al.; 2012; Zawacki-Richter et al.; 2009). This study’s results, however, differ from Wu et al. (2012) which found experiments as the second most common method for evaluating mobile and ubiquitous learning, whereas this study showed that interviews were more widely used than experiments. This suggests that interviews as a qualitative approach to complement survey findings have also been adopted in a certain number of studies. There are no observable differences in the indictors of effectiveness for mobile and ubiquitous learning practices involving access to information or social interaction. It is worth noting that less than half of the practices involved social interaction. According to Slavin (2012) and Sung BRAIN – Broad Research in Artificial Intelligence and Neuroscience, Volume 9, Special Issue on Educational Psychology (April, 2018), ISSN 2067-8957 62 et al. (2016), it is inconclusive whether social interaction with the aid of mobile devices enhances learning achievement. In addition to the wide use of perceived usefulness and learning achievements as indicators of effectiveness, the cases of mobile and ubiquitous learning practices also reveal a tendency for more indicators to have been adopted in recent studies. The average numbers of indicators in the studies have been increased from 2 in 2007, to 2.33 in 2011 and 2.71 in 2016. While more studies on a larger scale are needed to examine the issue, the present figures suggest that a broader range of factors have been considered regarding the effectiveness of mobile and ubiquitous learning. 6. Conclusion This paper has presented the diverse ways for reflecting the effectiveness of mobile and ubiquitous learning practices. While mobile and ubiquitous learning has been shown to bring better learning performance than conventional modes of learning (Sung et al.; 2016), this study’s findings provide various indicators to measure its effectiveness in a broad range of contexts. The results of this study suggest the need to conduct evaluation of mobile and ubiquitous learning practices on a larger scale. As the generalisability of the evaluation results depends on the extent of representativeness, it is uncertain whether the same degree of effectiveness will be obtained when extending the practices to a wider group of learners, as most evaluations focused only on single courses. Evaluation at the programme or institutional level, covering different subject disciplines, would help to provide an answer on this issue. As no single indicator can serve all the purposes of evaluation, using multiple indicators could be one possible way to obtain a comprehensive understanding of the effectiveness of a mobile and ubiquitous learning practice. This study has shown a preliminary trend for more indicators to be used in recent evaluations. In this regard, the work which lies ahead includes how various indicators can be adopted to complement each other and interpretation of the multiple dimensions of effectiveness. References Alrasheedi, M., & Capretz, L. F. (2014). Learner perceptions of a successful mobile learning platform: A systematic empirical study. Proceedings of the World Congress on Engineering and Computer Science, San Francisco, USA, pages 306–310. Alrasheedi, M., Capretz, L. F., & Raza, A. (2015). A systematic review of the critical factors for success of mobile learning in higher education (university students’ perspective). Journal of Educational Computing Research, 52(2), 257–276. Cardoso, T., & Abreu, R. (2015). Mobile learning and education: Synthesis of open access research. Handbook of mobile teaching and learning (pp. 133–148), Berlin Heidelberg: Springer. Cochrane, T. D. (2014). Critical success factors for transforming pedagogy with mobile Web 2.0. British Journal of Educational Technology, 45(1), 65–82. Crompton, H., Burke, D., Gregory, K. H., & Gräbe, C. (2016). The use of mobile learning in science: A systematic review. Journal of Science Education and Technology, 25(2), 149–160. Hilgarth, B. (2011). e-Learning success in action! From case study research to the creation of the cybernetic e-learning management model. International Journal of Computer Information Systems and Industrial Management Applications, 3, 415–426. Krotov, V. (2015). Critical success factors in m-learning: A socio-technical perspective. Communications of the Association for Information Systems, 36(6), 105–126. Shao, D., & Seif, H. (2014). Mobile learning a new wave of learning: A survey among University of Dodoma students. International Journal of Computer Applications, 98(16), 1–4. Slavin, R. E. (2012). Instruction based on cooperative learning. In R. E. Mayer, & P. A. Alexander (Eds.), Handbook of research on learning and instruction, New York, NY: Routledge, pages 344–360. B. T. M. Wong - Success in Mobile and Ubiquitous Learning: Indicators of Effectiveness 63 Sung, Y. T., Chang, K. E., & Liu, T. C. (2016). The effects of integrating mobile devices with teaching and learning on students’ learning performance: A meta-analysis and research synthesis. Computers & Education, 94, 252–275. Teri, S., Acai, A., Griffith, D., Mahmoud, Q., Ma, D. W. L., & Newton, G. (2013). Student use and pedagogical impact of a mobile learning application. Biochemistry and Molecular Biology Education, 42(2), 121–135. UNESCO. (2013). Policy guidelines for mobile learning. The United Nations Educational, Scientific and Cultural Organization. Retrieved from http://unesdoc.unesco.org/images/0021/002196/219641e.pdf Wang, H. Y., Liu, G. Z., & Hwang, G. J. (2016). Integrating socio-cu;tural contexts and location- based systems for ubiquitous language language learning in museums: A state of the art review of 2009–2014. British Journal of Educational Technology. Retrieved from http://onlinelibrary.wiley.com/doi/10.1111/bjet.12424/epdf Wittich, C.M., Wang, A.T., Fiala, J.A., Mauck, K.F., Mandrekar, J.N., Ratelle, J.T., & Beckman, T.J. (2016). Measuring participants’ attitudes toward mobile device conference applications in continuing medical education: Validation of an instrument. Journal of Continuing Education in the Health Professions, 36(1), 69–73. Wu, W.-H., Wu, Y.-C. J., Chen, C.-Y., Kao, H.-Y., Lin, C.-H., & Huang, S.-H. (2012). Review of trends from mobile learning studies: A meta-analysis. Computers & Education, 59(2), 817– 827. Yadegaridehkordi, E., Iahad, N. A., & Baloch, H. Z. (2013). Success factors influencing the adoption of M-learning. International Journal of Continuing Engineering Education and Life- Long Learning, 23(2), 167–178. Zawacki-Richter, O., Bäcker, E. M., & Vogt, S. (2009). Review of distance education research (2000 to 2008): Analysis of research areas, methods, and authorship patterns. International Review of Research in Open and Distance Learning, 10(6), 21–45. Zydney, J. M., & Warner, Z. (2016). Mobile apps for science learning: Review of research. Computers & Education, 94, 1–17.