Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes A Review of Gamification Impact on Student Behavioral and Learning Outcomes https://doi.org/10.3991/ijim.v15i21.24381 Muhammad Nurtanto1(), Nur Kholifah2, Erif Ahdhianto3, Achmad Samsudin4, Fajar Danur Isnantyo5 1 Universitas Sultan Ageng Tirtayasa, Banten, Indonesia 2 Yogyakarta State University, Yogyakarta, Indonesia 3 Universitas Negeri Malang, Malang, Indonesia 4 Universitas Pendidikan Indonesia, Bandung, Indonesia 5 Sebelas Maret University, Surakarta, Indonesia mnurtanto23@untirta.ac.id Abstract—Gamification has become a new trend in learning in the 21st century, by utilizing technology with game elements to change behavior and support learning outcomes. However, few researchers have focused on the com- prehensive impact of gamification in helping researchers to better understand developments over the past few years. The ScienceDirect, Taylor and Francis, Springer, Wiley, and SAGE publisher databases were surveyed and a total of forty articles from 2016–2021, were selected for review. Data analysis using NVivo 12 Plush software, with Hierarchy Chart and Mind Map methods. The main findings indicate the positive impact of gamification on student behavior and learning outcomes, including affective, cognitive, behavioral, and perfor- mance or others. The researcher recommends the continuity of gamification on learning outcomes and behavior, that interface design and teacher cognitive abil- ities are strategies or successful learning. Keywords—gamification, motivation, engagement, orientation knowledge, affective outcomes 1 Introduction Recently, digital games have been used for a wide range of purposes in schools and colleges. The increase in the development of learning models with the use of digital games is driven by the widespread change in people’s behavior in the use of digital technology in their daily lives, [1]. In this way, the use of digital technology has become common in human life, including the use of digital games for students. The increased interaction with the use of digital games offers possibilities for developing new learn- ing models with a digital gaming approach. In the context of education, many believe that the use of digital games can provide new ways to support learning and will ulti- mately impact the improvement of learning outcomes [1–3]. 22 http://www.i-jim.org https://doi.org/10.3991/ijim.v15i21.24381 mailto:mnurtanto23@untirta.ac.id Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes The learning model using game elements and game use techniques that are designed in an off-game context or a learning model with a game approach is referred to as gamification [4, 5]. Currently, gamification or game-based learning has become a new trend in learning by using technology with game elements to promote behavior and encourage desired learning results. A review of the gamification literature showed that gamification was mainly used in education [6–9]. This learning model was built on the basis of constructivist learning, which predicts the need for experiential learning through social interactions with the environment and peers [10, 11]. Interestingly, this learning approach encourages fun and meaningful learning so students can participate in productive learning activities [13]. Furthermore, gamification is also considered to be able to encourage human motivation and performance for certain activities [14]. The benefits of this gamification approach provide an incentive for students to participate in classroom learning activities. A study [15], developed a gamified platform that is integrated with an online e-learning portal with the finding that gamification is able to attract users’ attention to engage in the education system and increase interactivity and engagement. In fact, student motivation and participation in learning activities are important elements for improving learning outcomes, [15, 16]. In many cases, students feel bored and are not interested in learning materials because teaching methods are not considered commu- nicative, [12]. This lack of interest will become an impediment to direct student partic- ipation in learning and there is concern that this will negatively impact student learning outcomes. Motivation plays a significant role in affecting student learning outcomes. A high motivation to engage in learning with the gamification method will be more effective in influencing students’ subject matter expertise and attitudes toward lifelong learning [18]. In short, the concept of gamification in learning is one way of changing student behaviour, through a digital play approach. Because, essentially, game-based learning describes the use of fun content as an online learning technique to achieve educational objectives [16, 17]. Gamification has an important position in the world of education both in schools and universities, but there is still little research that discusses how the impact of implement- ing gamification on student learning outcomes has been comprehensively discussed. Information about the impact of applying gamification will provide an overview for teachers or teachers to plan and develop gamification learning methods. The purpose of this literature review research is to analyze the significant impact of gamification in improving learning outcomes and behavior of elements of students’ attitudes, cog- nitive, behavior, and performance. The research question in this context is what the deeper game elements of gamification inside are improving learning outcomes and changing behavior. Thus, the popularity of gamification and its variety of success in the context of education strengthens potential, beliefs, and preferences. This study reviews and collects existing knowledge about the topic and presents it in a structured manner. 2 Methods To reveal the effect of the gamification method on student learning outcomes, we conducted a literature review (SLR) using a systematic and specific method to identify, iJIM ‒ Vol. 15, No. 21, 2021 23 Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes select, and collect all materials relevant to the research question, [21]. Specific steps include (1) database search; (2) inclusion and exclusion criteria; (3) study selection; (4) data analysis and extraction; (5) summarizing and interpreting findings; and (6) compiling reports, [22]. Selecting documents through a review of titles, abstracts and complete documents containing the specified search terms and adopting search keyword links using Boolean (OR) and (AND) operators. The document search for this study used keywords “(games OR gamification) AND (education OR learning output)” in both online catalogues. For ease of finding articles in this study, we limit articles to the education sector. The types of papers reviewed include review papers and original searches. In addition, we are restricting the period for publishing articles from 2016 to 2020. The search criteria for articles in the two online catalogues consisting of ScienceDirect and Taylor and Francis Online are shown in Table 1. Table 1. Distribution of retrieved papers among sources Publishers Search Link Total Amount No of Selected ScienceDirect https://www.sciencedirect.com 16.915 18a Taylor and Francis https://www.tandfonline.com 1.371 13a Springer Nature https://link.springer.com 79 4a Wiley https://onlinelibrary.wiley.com 24 3a SAGE Publications https://methods.sagepub.com 19 2a Note: aTheoretical papers. The inclusion criteria were applied to screen study variants, select, and include articles relevant to the research objectives and exclude studies that did not meet the requirements. Inclusion and exclusion criteria, are presented in Table 2 below. Table 2. Inclusion and exclusion criteria Inclusion Criteria Exclusion Criteria Publications in English Non-English publications Gamification in education Outside the area of education Peer reviewed articles published from 2016 onwards Conference papers and dissertations Empirical quantitative and mixed-methods studies Discussions, qualitative and theoretical studies Instruments explicitly measuring learning outcomes Competence, anxiety, and self-confidence No restrictions on how teacher identity is conceptualized Open-ended questions A total of 40 articles was further analyzed to provide findings regarding the research question. The validity and credibility of the study were applied by means of extraction, as a triangulation procedure [23]. The thorough review is based on review and analy- sis of learning outcomes. Child elements or criteria are searched for using systematic keywords. Then convergence and verification of findings are carried out, data analysis is carried out at the final stage after the completion of the triangulation procedure. The data analysis stage uses the Nvivo 12 Plus application with the aim of making it easier 24 http://www.i-jim.org https://www.sciencedirect.com https://www.tandfonline.com https://link.springer.com/ https://onlinelibrary.wiley.com/ https://methods.sagepub.com Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes to find derived elements or criteria based on the research objectives, [24]. The steps taken are (1) auto code analysis is used to automatically find common themes; (2) the selection of themes from the results of the auto code based on the research objectives, namely the impact of gamification on learning outcomes; (3) code consolidation where the similarity of meaning is grouped according to the impact of gamification on learn- ing outcomes; and (4) submission of data to map elements of the impact of gamification on learning outcomes. 3 Result 3.1 Characteristic research Gamification is a game design element with a non-game context, [25]. The 40 research articles from four leading publishers in 2016–2021. Distribution of studies (see Figure 1), by year as follows: 2016 (17.5%, n=7), 2017 (12.5%, n=5), 2018 (12.5%, n=5), 2019 (15.0%, n= 6), 2020 (27.5%, n=11), and 2021 (15.0%, n=6). The distribu- tion of studies by the publisher is as follows: Elsevier BV (45.0%, n=18), Taylor and Francis (32.5%, n=13), Springer Nature (10.0%, n=4), Wiley (7.5%, n=3), and SAGE (5.0%, n=2). Overall, the selected articles are gamification in the context of education. 4 1 1 1 3 1 1 2 3 4 1 1 8 1 1 1 3 2 1 T ay lo r a nd F ra nc is E ls ev ie r B V Sp ri ng er N at ur e SA G E P ub lic at io ns E ls ev ie r B V Sp ri ng er N at ur e W ile y E ls ev ie r B V T ay lo r a nd F ra nc is T ay lo r a nd F ra nc is E ls ev ie r B V Sp ri ng er N at ur e E ls ev ie r B V T ay lo r a nd F ra nc is Sp ri ng er N at ur e SA G E P ub lic at io ns E ls ev ie r B V W ile y T ay lo r a nd F ra nc is 2016 2017 2018 2019 2020 2021 Fig. 1. Distribution of studies in years (2016–2021) 3.2 Elements of gamification on behavior and learning outcomes The researcher identifies the findings into four elements that are sourced from empir- ical studies [26], and additional elements (others) that contribute and are personal. The elements in gamification include affective learning outcomes (33.33%, n=20), iJIM ‒ Vol. 15, No. 21, 2021 25 Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes cognitive learning outcomes (16.57%, n=10), behavioral learning outcomes (21.67%, n=13), student performance (20.00%, n= 12), and other elements (8.33%, n=5). Refer- ence articles in each element are correlated with other relevant elements. The elements that are most often discussed in the success of gamification are studies on affective learning outcomes, and behavior. Table 3. Element of gamification outcomes Element of Gamification Nu (%) Papers Affective Outcomes 20 (33.33) (Aguiar-Castillo, Clavijo-Rodriguez, et al., 2020; Bakhanova et al., 2020; Cózar-Gutiérrez & Sáez-López, 2016; Díaz-Ramírez, 2020; Gil-Doménech & Berbegal-Mirabent, 2019; Huang & Hew, 2018; Hwang et al., 2016; Isabelle, 2020; Jayalath & Esichaikul, 2020; Khan et al., 2017; Khowaja & Salim, 2019; Laubersheimer et al., 2016; Legaki et al., 2020; Li & Chu, 2021; Morschheuser et al., 2017; Sailer & Sailer, 2021; Shipherd & Burt, 2018; Toda, do Carmo, et al., 2019; Toda, Klock, et al., 2019; Whitton & Langan, 2019; Xu et al., 2017), [24–44] Cognitive Outcomes 10 (16.57) (Bakhanova et al., 2020; Barrett et al., 2016; Caño de las Heras et al., 2021; Lameras et al., 2017; Legaki et al., 2020, 2021; Martí-Parreño et al., 2021; Pitura & Terlecka-Pacut, 2018; Sailer & Sailer, 2021; Shipherd & Burt, 2018), [25, 36, 39, 40, 45–50] Behaviour Outcomes 13 (21.67) (Aguiar-Castillo, Clavijo-Rodriguez, et al., 2020; Bakhanova et al., 2020; Barrett et al., 2016; Caño de las Heras et al., 2021; Carenys & Moya, 2016; Díaz-Ramírez, 2020; Laubersheimer et al., 2016; Martí- Parreño et al., 2021; Morschheuser et al., 2017; Pitura & Terlecka- Pacut, 2018; Sailer & Sailer, 2021; Toda, do Carmo, et al., 2019; Zou et al., 2019), [24, 25, 27, 35, 38, 39, 41, 45, 46, 49–52] Student Performance 12 (20.00) (Bai et al., 2020; Castronovo et al., 2018; Gil-Doménech & Berbegal- Mirabent, 2019; Hofacker et al., 2016; Hwang et al., 2016; Isabelle, 2020; Legaki et al., 2021; Liu, 2016; Martí-Parreño et al., 2021; Punia et al., 2020; Sailer & Sailer, 2021; Sanchez et al., 2020), [28, 30, 31, 39, 48, 49, 53–58] Others *Image *Self-efficacy *Sense of belonging 5 (8.33) (Aguiar-Castillo, Clavijo-Rodriguez, et al., 2020; Aguiar-Castillo, Hernández-López, et al., 2020; Díaz-Ramírez, 2020; Isabelle, 2020; Shi et al., 2017), [24, 27, 31, 59, 60] 3.3 Hierarchy chart and project map on gamification output The elements in the gamification output are illustrated (Figure 2), where each ele- ment consists of several studies. 26 http://www.i-jim.org Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes Fig. 2. Conceptual framework of elements in hierarchy chart-based gamification outcomes Fig. 3. Project map of gamification output element iJIM ‒ Vol. 15, No. 21, 2021 27 Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes The area of the area describes that the study is discussed most often, while the small area is the discussion that is discussed the least. Overall description of gamification output (Figure 3). Project Map explains that Gamification output consists of 4 prior- ity elements and 1 complementary element. Elements on affective outcomes describe studies on student engagement, student intrinsic motivation, student extrinsic motiva- tion, student interest, student enjoyment, student satisfaction, and student innovation. Elements on behavior outcomes consist of critical thinking, social skills, teamwork skills, digital literacy, disciplines, communication skills, and deep learning. Elements on cognitive outcomes consist of orientation knowledge and forecasting outcomes. Ele- ments in student performance consist of student technology, student capacities, student achievement, and emotional. Additional elements consist of self-efficacy, image, and sense of belonging. In a study [64], explaining emotions, motivation, self-efficacy as variables, but this decision is still biased. Thus, this study is discussed in a wider cluster. 3.4 Hierarchy chart and project map on gamification output Fig. 4. The correlation between elements using Pearson’ coefficient 28 http://www.i-jim.org Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes In Table 4 describes the correlation between elements of the gamification output using a measured parameter, namely Person correlation (0–1). The higher the value (PC) or close to the value 1, the correlation between the two is very high. In Figure 4. The highest correlation is the element of affective outcome-behavior outcome of 0.812 (close to 1), while the lowest relationship is cognitive outcomes-others of 0.460. The three highest variables discussed are student intrinsic motivation, orientation knowl- edge, and student engagement. Furthermore, the relationship between variables is dis- cussed for the person coefficient value exceeding 0.50 as follows. Table 4. Correlation between elements and variables in gamification outcomes Element Vs Variables Pearson Coefficient Affective Outcomes Student Intrinsic Motivation 0,9016 Affective Outcomes Student Engagement 0,8827 Affective Outcomes Student Satisfaction 0,5784 Affective Outcomes Student Innovation 0,5746 Behaviour Outcomes Social Skills 0,7315 Behaviour Outcomes Digital Literacy 0,6869 Behaviour Outcomes Teamwork Skills 0,6439 Behaviour Outcomes Communication Skills 0,5173 Cognitive Outcomes Oriented Knowledge 0,9421 Cognitive Outcomes Student Performance 0,6911 Cognitive Outcomes Student Achievement 0,6447 Others Sense of belonging 0,5793 Others Self-Efficacy 0,5157 Others Student Performance 0,5021 4 Discussion The use of gamification learning programs in education has a significant impact on increasing the expected learning outcomes. An important impact of the applica- tion of gamification on the learning process is an increase in the affective element. Most gamification studies emphasize the effect of gamification on the affective element [24–44]. There are 20 out of 40 articles that emphasize the impact of gamification on the affective domain. The definition of the affective domain refers to the way a person handles something emotionally, such as internal and external motivation, enthusiasm, enjoyment, satisfaction, interest, and innovation. The results of the study explain that the variables of enthusiasm and internal motivation as key variables in the elements of affective learning outcomes, in the last five years. The enthusiasm of students in game- based learning is caused by motivation from within. Finally, enthusiasm and motivation are mutually reinforcing. Meanwhile, motivation arises from fun experiences during game-based learning. An important reason that gamification motivates, [25, 52, 62] learners (1) attempt more difficult tasks [32], and (2) develop the information literacy skills necessary for success, [38]. Thus, the learning objectives can be achieved. iJIM ‒ Vol. 15, No. 21, 2021 29 Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes Previous studies have found much on the impact of applying gamification in the context of education, including cognitive outcomes. Learning outcomes in cognitive terms refer to the structure of knowledge that develops using games as a means of learning [62]. Many studies have shown the benefits of using digital games in various research contexts, such as the study conducted [66], revealed that gamification has a positive impact on student retention. Another study, multi-dimensional game-based learning helps software learners increase student knowledge [67]. Because, in reality, the concept of gamification provides an approach of knowledge transfer methodology in the context of learning [68]. It is therefore entirely natural that the use of gamification has an impact on students’ understanding. One of the main goals of implementing various game-based learning methods and strategies is to change student behavior. In this literature study, the findings reveal one of the impacts of applying gamification, namely increasing learning outcomes in the form of behavioral outcomes. Outcomes of behavior refer to the psychomotor field in Bloom’s taxonomy. This domain describes the ability to manipulate physical behavior that focuses on changing and/or developing behaviors and/or skills. The results of this literature review study indicate that the impact of the application of gamification has an influence on behavior change, including teamwork, communication skills, social skills and digital literacy, [8, 37, 38]. Teachers believe that gamification promotes teamwork and oral communication skills, [9]. In other contexts, the application of gamification provides benefits for the development of skills such as teamwork, digital literacy, English, and content knowl- edge (history) [69, 71]. Meanwhile, Gamification has an impact on improving students’ social skills [9]. The gamification feature allows students to learn collaboratively with other students. Case studies of learning problems (quiz type) can be arranged to be solved in groups or in collaboration. The maximum use of gamification by teachers and students will increase student achievement in learning. Previous studies have discussed how gamification affects performance element. For example, research findings [72], show that challenge-based games with a gamification approach increase student performance by 34.75% on the topic of statistical forecasting. Meanwhile, [67] the use of gamification can improve project quality and output across all software processes and their study concluded that multi-dimensional game-based learning increases the probability of project completion in a timely manner. The findings of other studies are relevant to learning outcomes, namely image, self-efficacy, and sense of belonging. The use of learning using gamifi- cation can improve the perceived image and self-efficacy also increases, up to the level of sense of belonging [30, 34, 62, 73]. These three studies are other elements that are abstract (intangible) or other impacts after using gamification. 5 Conclusion This literature review concludes that the use of gamification plays a significant role in improving student learning outcomes. It has been proven that gamification has an impact on student engagement, intrinsic motivation, extrinsic motivation, interest, enjoyment, satisfaction, and innovation in learning activities. In addition, the success of 30 http://www.i-jim.org Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes gamification in behavioral outcome elements has an impact on increasing critical think- ing, social skills, teamwork skills, digital literacy, disciplines, communication skills, and deep learning. The impact of success on elements of cognitive learning outcomes includes orientation knowledge and forecasting outcomes. Meanwhile, the impact of gamification on student performance includes technology, capacities, achievement, and emotional. The impact of gamification was also found on students’ personal such as self-efficacy, image, and sense of belonging. These findings have implications for the development of strategies in learning using gamification, so that all elements of learn- ing outcomes can be achieved in their entirety. 6 References [1] Arifin, Z., Nurtanto, M., Warju, W., Rabiman, R., & Kholifah, N. (2020). The tawock con- ceptual model at content knowledge for professional teaching in vocational education. Inter- national Journal of Evaluation and Research in Education, 9(3), 697–703. Scopus. https:// doi.org/10.11591/ijere.v9i3.20561 [2] Legaki, N.-Z., Xi, N., Hamari, J., Karpouzis, K., & Assimakopoulos, V. (2020). The effect of challenge-based gamification on learning: An experiment in the context of statistics education. International Journal of Human-Computer Studies, 144, 102496. https://doi. org/10.1016/j.ijhcs.2020.102496 [3] Díaz-Ramírez, J. (2020). Gamification in engineering education – An empirical assessment on learning and game performance. Heliyon, 6(9), e04972. https://doi.org/10.1016/j.heli- yon.2020.e04972 [4] Zainuddin, Z., Chu, S. K. W., Shujahat, M., & Perera, C. J. (2020). The impact of gamifi- cation on learning and instruction: A systematic review of empirical evidence. Educational Research Review, 30, 100326. https://doi.org/10.1016/j.edurev.2020.100326 [5] Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: Defining “gamification. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments. [6] Werbach, K., & Hunter, D. (2012). For the Win: How Game Thinking Can Revolutionize Your Business. Wharton Digital Press. [7] Koivisto, J., & Hamari, J. (2019). The rise of motivational information systems: A review of gamification research. International Journal of Information Management, 45, 191–210. https://doi.org/10.1016/j.ijinfomgt.2018.10.013 [8] Sandrone, S., & Carlson, C. (2021). Gamification and game-based education in neurology and neuroscience: Applications, challenges, and opportunities. Brain Disorders, 1, 100008. https://doi.org/10.1016/j.dscb.2021.100008 [9] Martí-Parreño, J., Galbis-Córdova, A., & Currás-Pérez, R. (2021). Teachers’ beliefs about gamification and competencies development: A concept mapping approach. Innovations in Education and Teaching International, 58(1), 84–94. https://doi.org/10.1080/14703297.201 9.1683464 [10] Parra-González, M. E., López-Belmonte, J., Segura-Robles, A., & Moreno-Guerrero, A.-J. (2021). Gamification and flipped learning and their influence on aspects related to the teach- ing-learning process. Heliyon, 7(2), e06254. https://doi.org/10.1016/j.heliyon.2021.e06254 [11] York, J., & DeHaan, J. W. (2018). A constructivist approach to game-based language learn- ing: Student perceptions in a beginner-level EFL context. International Journal of Game- Based Learning, 8(1), 19–40. https://doi.org/10.4018/IJGBL.2018010102. iJIM ‒ Vol. 15, No. 21, 2021 31 https://doi.org/10.11591/ijere.v9i3.20561 https://doi.org/10.11591/ijere.v9i3.20561 https://doi.org/10.1016/j.ijhcs.2020.102496 https://doi.org/10.1016/j.ijhcs.2020.102496 https://doi.org/10.1016/j.heliyon.2020.e04972 https://doi.org/10.1016/j.heliyon.2020.e04972 https://doi.org/10.1016/j.edurev.2020.100326 https://doi.org/10.1016/j.ijinfomgt.2018.10.013 https://doi.org/10.1016/j.dscb.2021.100008 https://doi.org/10.1080/14703297.2019.1683464 https://doi.org/10.1080/14703297.2019.1683464 https://doi.org/10.1016/j.heliyon.2021.e06254 https://doi.org/10.4018/IJGBL.2018010102 Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes [12] Vidakis, N., Barianos, A. K., Trampas, A. M., Papadakis, S., Kalogiannakis, M., & Vassilakis, K. (2020). In-Game Raw Data Collection and Visualization in the Context of the “ThimelEdu” Educational Game (pp. 629–646). Springer International Publishing. https:// doi.org/10.1007/978-3-030-58459-7_30 [13] Chou, Y. K. (2016). Actionable gamification: Beyond points, badges, and leaderboards. Octalysis Media. [14] Sailer, M., Hense, J. U., Mayr, S. K., & Mandl, H. (2017). How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction. Computers in Human Behavior, 69, 371–380. https://doi.org/10.1016/j. chb.2016.12.033 [15] Bouchrika, I., Harrati, N., Wanick, V., & Wills, G. (2019). Exploring the impact of gamifica- tion on student engagement and involvement with e-learning systems. Interactive Learning Environments, 1–14. https://doi.org/10.1080/10494820.2019.1623267 [16] Kalogiannakis, M., Papadakis, S., & Zourmpakis, A.-I. (2021). Gamification in Science Education. A Systematic Review of the Literature. Education Sciences, 11(1), 22. https:// doi.org/10.3390/educsci11010022 [17] Kholifah, N., Sudira, P., Rachmadtullah, R., Nurtanto, M., & Suyitno, S. (2020). The effec- tiveness of using blended learning models against vocational education student learning motivation. International Journal of Advanced Trends in Computer Science and Engineer- ing, 9(5), 7964–7968. Scopus. https://doi.org/10.30534/ijatcse/2020/151952020 [18] Gatti, L., Ulrich, M., & Seele, P. (2019). Education for sustainable development through business simulation games: An exploratory study of sustainability gamification and its effects on students’ learning outcomes. Journal of Cleaner Production, 207, 667–678. https://doi.org/10.1016/j.jclepro.2018.09.130 [19] de-Marcos, L., Garcia-Lopez, E., & Garcia-Cabot, A. (2016). On the effectiveness of game- like and social approaches in learning: Comparing educational gaming, gamification & social networking. Computers & Education, 95(1), 99–113. [20] Papadakis, S., & Kalogiannakis, M. (2019). Evaluating the effectiveness of a game-based learning approach in modifying students’ behavioural outcomes and competence, in an introductory programming course. A case study in Greece. International Journal of Teach- ing and Case Studies, 10(3), 235–250. https://doi.org/10.1504/IJTCS.2019.102760 [21] Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010). Systematic literature reviews in software engineering—A tertiary study. Information and Software Technology, 52(8), 792–805. https://doi.org/10.1016/j. infsof.2010.03.006 [22] Kitchenham, B. (2004). Procedures for Performing Systematic Reviews. Keele University. [23] Kern, F. G. (2018). The Trials and Tribulations of Applied Triangulation: Weighing Dif- ferent Data Sources. Journal of Mixed Methods Research, 12(2), 166–181. https://doi. org/10.1177/1558689816651032 [24] O’Neill, M., Booth, S., & Lamb, J. (2018). Using NVivoTM for Literature Reviews: The Eight Step Pedagogy (N7 1). The Qualitative Report, 23(13), 21–39. https://doi. org/10.46743/2160-3715/2018.3030 [25] Deterding, S., Sicart, M., Nacke, L., O’Hara, K., & Dixon, D. (2011). Gamification. Using game-design elements in non-gaming contexts. CHI ’11 Extended Abstracts on Human Factors in Computing Systems, 2425–2428. https://doi.org/10.1145/1979742. 1979575 [26] Forndran, F., & Zacharias, C. R. [UNESP. (2019). Gamified experimental physics classes: A promising active learning methodology for higher education. European Journal of Physics. https://doi.org/10.1088/1361-6404/ab215e 32 http://www.i-jim.org https://doi.org/10.1007/978-3-030-58459-7_30 https://doi.org/10.1007/978-3-030-58459-7_30 https://doi.org/10.1016/j.chb.2016.12.033 https://doi.org/10.1016/j.chb.2016.12.033 https://doi.org/10.1080/10494820.2019.1623267 https://doi.org/10.3390/educsci11010022 https://doi.org/10.3390/educsci11010022 https://doi.org/10.30534/ijatcse/2020/151952020 https://doi.org/10.1016/j.jclepro.2018.09.130 https://doi.org/10.1504/IJTCS.2019.102760 https://doi.org/10.1016/j.infsof.2010.03.006 https://doi.org/10.1016/j.infsof.2010.03.006 https://doi.org/10.1177/1558689816651032 https://doi.org/10.1177/1558689816651032 https://doi.org/10.46743/2160-3715/2018.3030 https://doi.org/10.46743/2160-3715/2018.3030 https://doi.org/10.1145/1979742.1979575 https://doi.org/10.1145/1979742.1979575 https://doi.org/10.1088/1361-6404/ab215e Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes [27] Aguiar-Castillo, L., Clavijo-Rodriguez, A., Hernández-López, L., De Saa-Pérez, P., & Pérez- Jiménez, R. (2020). Gamification and deep learning approaches in higher education. Jour- nal of Hospitality, Leisure, Sport & Tourism Education, 100290. https://doi.org/10.1016/j. jhlste.2020.100290 [28] Bakhanova, E., Garcia, J. A., Raffe, W. L., & Voinov, A. (2020). Targeting social learn- ing and engagement: What serious games and gamification can offer to participatory mod- eling. Environmental Modelling and Software, 134, 104846. https://doi.org/10.1016/j. envsoft.2020.104846 [29] Cózar-Gutiérrez, R., & Sáez-López, J. M. (2016). Game-based learning and gamification in initial teacher training in the social sciences: An experiment with MinecraftEdu. Inter- national Journal of Educational Technology in Higher Education, 13(1), 2. https://doi. org/10.1186/s41239-016-0003-4 [30] Díaz-Ramírez, J. (2020a). Gamification in engineering education – An empirical assessment on learning and game performance. Heliyon, 6(9), e04972. https://doi.org/10.1016/j.heli- yon.2020.e04972 [31] Gil-Doménech, D., & Berbegal-Mirabent, J. (2019). Stimulating students’ engagement in mathematics courses in non-STEM academic programmes: A game-based learning. Innova- tions in Education and Teaching International, 56(1), 57–65. https://doi.org/10.1080/14703 297.2017.1330159 [32] Huang, B., & Hew, K. F. (2018). Implementing a theory-driven gamification model in higher education flipped courses: Effects on out-of-class activity completion and quality of artifacts. Computers & Education, 125, 254–272. https://doi.org/10.1016/j.compedu.2018.06.018 [33] Hwang, W.-Y., Shih, T. K., Ma, Z.-H., Shadiev, R., & Chen, S.-Y. (2016). Evaluating lis- tening and speaking skills in a mobile game-based learning environment with situational contexts. Computer Assisted Language Learning, 29(4), 639–657. https://doi.org/10.1080/0 9588221.2015.1016438 [34] Isabelle, D. A. (2020). Gamification of Entrepreneurship Education. Decision Sciences Journal of Innovative Education, 18(2), 203–223. https://doi.org/10.1111/dsji.12203 [35] Jayalath, J., & Esichaikul, V. (2020). Gamification to Enhance Motivation and Engagement in Blended eLearning for Technical and Vocational Education and Training. Technology, Knowledge and Learning. https://doi.org/10.1007/s10758-020-09466-2 [36] Khan, A., Ahmad, F. H., & Malik, M. M. (2017). Use of digital game based learning and gamification in secondary school science: The effect on student engagement, learning and gender difference. Education and Information Technologies, 22(6), 2767–2804. https://doi. org/10.1007/s10639-017-9622-1 [37] Khowaja, K., & Salim, S. S. (2019). Serious Game for Children with Autism to Learn Vocab- ulary: An Experimental Evaluation. International Journal of Human–Computer Interaction, 35(1), 1–26. https://doi.org/10.1080/10447318.2017.1420006 [38] Laubersheimer, J., Ryan, D., & Champaign, J. (2016). InfoSkills2Go: Using Badges and Gamification to Teach Information Literacy Skills and Concepts to College-Bound High School Students. Journal of Library Administration, 56(8), 924–938. https://doi.org/10.108 0/01930826.2015.1123588 [39] Legaki, N.-Z., Xi, N., Hamari, J., Karpouzis, K., & Assimakopoulos, V. (2020b). The effect of challenge-based gamification on learning: An experiment in the context of statistics education. International Journal of Human-Computer Studies, 144, 102496. https://doi. org/10.1016/j.ijhcs.2020.102496 [40] Li, X., & Chu, S. K. W. (2021). Exploring the effects of gamification pedagogy on children’s reading: A mixed-method study on academic performance, reading-related mentality and behaviors, and sustainability. British al of Educational Technology, 52(1), 160–178. https:// doi.org/10.1111/bjet.13057 iJIM ‒ Vol. 15, No. 21, 2021 33 https://doi.org/10.1016/j.jhlste.2020.100290 https://doi.org/10.1016/j.jhlste.2020.100290 https://doi.org/10.1016/j.envsoft.2020.104846 https://doi.org/10.1016/j.envsoft.2020.104846 https://doi.org/10.1186/s41239-016-0003-4 https://doi.org/10.1186/s41239-016-0003-4 https://doi.org/10.1016/j.heliyon.2020.e04972 https://doi.org/10.1016/j.heliyon.2020.e04972 https://doi.org/10.1080/14703297.2017.1330159 https://doi.org/10.1080/14703297.2017.1330159 https://doi.org/10.1016/j.compedu.2018.06.018 https://doi.org/10.1080/09588221.2015.1016438 https://doi.org/10.1080/09588221.2015.1016438 https://doi.org/10.1111/dsji.12203 https://doi.org/10.1007/s10758-020-09466-2 https://doi.org/10.1007/s10639-017-9622-1 https://doi.org/10.1007/s10639-017-9622-1 https://doi.org/10.1080/10447318.2017.1420006 https://doi.org/10.1080/01930826.2015.1123588 https://doi.org/10.1080/01930826.2015.1123588 https://doi.org/10.1016/j.ijhcs.2020.102496 https://doi.org/10.1016/j.ijhcs.2020.102496 https://doi.org/10.1111/bjet.13057 https://doi.org/10.1111/bjet.13057 Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes [41] Morschheuser, B., Hamari, J., Koivisto, J., & Maedche, A. (2017). Gamified crowdsourc- ing: Conceptualization, literature review, and future agenda. International Journal of Human-Computer Studies, 106, 26–43. https://doi.org/10.1016/j.ijhcs.2017.04.005 [42] Sailer, M., & Sailer, M. (2021). Gamification of in-class activities in flipped classroom lectures. British Journal of Educational Technology, 52(1), 75–90. https://doi.org/10.1111/ bjet.12948 [43] Shipherd, A. M., & Burt, D. J. (2018). Game on! Gamifying the sport psychology college classroom. Journal of Sport Psychology in Action, 9(3), 147–158. https://doi.org/10.1080/2 1520704.2018.1434581 [44] Toda, A. M., do Carmo, R. M. C., da Silva, A. P., Bittencourt, I. I., & Isotani, S. (2019). An approach for planning and deploying gamification concepts with social networks within educational contexts. International Journal of Information Management, 46, 294–303. https://doi.org/10.1016/j.ijinfomgt.2018.10.001 [45] Toda, A. M., Klock, A. C. T., Oliveira, W., Palomino, P. T., Rodrigues, L., Shi, L., Bitten- court, I., Gasparini, I., Isotani, S., & Cristea, A. I. (2019). Analysing gamification elements in educational environments using an existing Gamification taxonomy. Smart Learning Environments, 6(1), 16. https://doi.org/10.1186/s40561-019-0106-1 [46] Whitton, N., & Langan, M. (2019). Fun and games in higher education: An analysis of UK student perspectives. Teaching in Higher Education, 24(8), 1000–1013. https://doi.org/10.1 080/13562517.2018.1541885 [47] Xu, F., Buhalis, D., & Weber, J. (2017). Serious games and the gamification of tourism. Tourism Management, 60, 244–256. https://doi.org/10.1016/j.tourman.2016.11.020 [48] Barrett, N., Swain, I., Gatzidis, C., & Mecheraoui, C. (2016). The use and effect of video game design theory in the creation of game-based systems for upper limb stroke rehabilitation. Journal of Rehabilitation and Assistive Technologies Engineering, 3, 2055668316643644. https://doi.org/10.1177/2055668316643644 [49] Caño de las Heras, S., Gargalo, C. L., Weitze, C. L., Mansouri, S. S., Gernaey, K. V., & Krühne, U. (2021). A framework for the development of Pedagogical Process Simulators (P2Si) using explanatory models and gamification. Computers & Chemical Engineering, 151, 107350. https://doi.org/10.1016/j.compchemeng.2021.107350 [50] Lameras, P., Arnab, S., Dunwell, I., Stewart, C., Clarke, S., & Petridis, P. (2017). Essen- tial features of serious games design in higher education: Linking learning attributes to game mechanics. British Journal of Educational Technology, 48(4), 972–994. https://doi. org/10.1111/bjet.12467 [51] Legaki, N.-Z., Karpouzis, K., Assimakopoulos, V., & Hamari, J. (2021). Gamification to avoid cognitive biases: An experiment of gamifying a forecasting course. Technological Forecast- ing and Social Change, 167, 120725. https://doi.org/10.1016/j.techfore.2021.120725 [52] Martí-Parreño, J., Galbis-Córdova, A., & Currás-Pérez, R. (2021b). Teachers’ beliefs about gamification and competencies development: A concept mapping approach. Innovations in Education and Teaching International, 58(1), 84–94. https://doi.org/10.1080/14703297.201 9.1683464 [53] Pitura, J., & Terlecka-Pacut, E. (2018). Action research on the application of technology assisted urban gaming in language education in a Polish upper-secondary school. Computer Assisted Language Learning, 31(7), 734–763. https://doi.org/10.1080/09588221.2018.1447490 [54] Carenys, J., & Moya, S. (2016). Digital game-based learning in accounting and business education. Accounting Education, 25(6), 598–651. https://doi.org/10.1080/09639284.2016. 1241951 [55] Zou, D., Huang, Y., & Xie, H. (2019). Digital game-based vocabulary learning: Where are we and where are we going? Computer Assisted Language Learning, 0(0), 1–27. https://doi. org/10.1080/09588221.2019.1640745 34 http://www.i-jim.org https://doi.org/10.1016/j.ijhcs.2017.04.005 https://doi.org/10.1111/bjet.12948 https://doi.org/10.1111/bjet.12948 https://doi.org/10.1080/21520704.2018.1434581 https://doi.org/10.1080/21520704.2018.1434581 https://doi.org/10.1016/j.ijinfomgt.2018.10.001 https://doi.org/10.1186/s40561-019-0106-1 https://doi.org/10.1080/13562517.2018.1541885 https://doi.org/10.1080/13562517.2018.1541885 https://doi.org/10.1016/j.tourman.2016.11.020 https://doi.org/10.1177/2055668316643644 https://doi.org/10.1016/j.compchemeng.2021.107350 https://doi.org/10.1111/bjet.12467 https://doi.org/10.1111/bjet.12467 https://doi.org/10.1016/j.techfore.2021.120725 https://doi.org/10.1080/14703297.2019.1683464 https://doi.org/10.1080/14703297.2019.1683464 https://doi.org/10.1080/09588221.2018.1447490 https://doi.org/10.1080/09639284.2016.1241951 https://doi.org/10.1080/09639284.2016.1241951 https://doi.org/10.1080/09588221.2019.1640745 https://doi.org/10.1080/09588221.2019.1640745 Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes [56] Bai, S., Hew, K. F., & Huang, B. (2020). Does gamification improve student learn- ing outcome? Evidence from a meta-analysis and synthesis of qualitative data in educa- tional contexts. Educational Research Review, 30, 100322. https://doi.org/10.1016/j. edurev.2020.100322 [57] Castronovo, F., Meter, P. N. V., & Messner, J. I. (2018). Leveraging metacognitive prompts in construction educational games for higher educational gains. International Journal of Construction Management, 0(0), 1–12. https://doi.org/10.1080/15623599.2018.1492760 [58] Hofacker, C. F., de Ruyter, K., Lurie, N. H., Manchanda, P., & Donaldson, J. (2016). Gam- ification and Mobile Marketing Effectiveness. Journal of Interactive Marketing, 34, 25–36. https://doi.org/10.1016/j.intmar.2016.03.001 [59] Liu, T.-Y. (2016). Using educational games and simulation software in a computer science course: Learning achievements and student flow experiences. Interactive Learning Environ- ments, 24(4), 724–744. https://doi.org/10.1080/10494820.2014.917109 [60] Punia, S. K., Kumar, M., Aggarwal, J., & Malik, K. (2020). Object based learning using multi-dimensional games. Journal of Discrete Mathematical Sciences and Cryptography, 23(2), 509–524. https://doi.org/10.1080/09720529.2020.1728904 [61] Sanchez, D. R., Langer, M., & Kaur, R. (2020). Gamification in the classroom: Examining the impact of gamified quizzes on student learning. Computers & Education, 144, 103666. https://doi.org/10.1016/j.compedu.2019.103666 [62] Aguiar-Castillo, L., Hernández-López, L., De Saá-Pérez, P., & Pérez-Jiménez, R. (2020). Gamification as a motivation strategy for higher education students in tourism face-to-face learning. Journal of Hospitality, Leisure, Sport & Tourism Education, 27, 100267. https:// doi.org/10.1016/j.jhlste.2020.100267 [63] Shi, V. G., Baines, T., Baldwin, J., Ridgway, K., Petridis, P., Bigdeli, A. Z., Uren, V., & Andrews, D. (2017). Using gamification to transform the adoption of servitization. Indus- trial Marketing Management, 63, 82–91. https://doi.org/10.1016/j.indmarman.2016.12.005 [64] Ekici, M. (2021). A systematic review of the use of gamification in flipped learning. Education and Information Technologies, 26(3), 3327–3346. https://doi.org/10.1007/ s10639-020-10394-y [65] Kusuma, G. P., Wigati, E. K., Utomo, Y., & Putera Suryapranata, L. K. (2018). Analysis of Gamification Models in Education Using MDA Framework. Procedia Computer Science, 135, 385–392. https://doi.org/10.1016/j.procs.2018.08.187 [66] Jamaludin, N. F., Wook, T. S. M. T., Noor, S. F. M., & Qamar, F. (2021). Gamification Design Elements to Enhance Adolescent Motivation in Diagnosing Depression. Interna- tional Journal of Interactive Mobile Technologies (IJIM), 15(10), 154–172. https://www. online-journals.org/index.php/i-jim/article/view/21137 [67] Putz, L.-M., Hofbauer, F., & Treiblmaier, H. (2020). Can gamification help to improve edu- cation? Findings from a longitudinal study. Computers in Human Behavior, 110, 106392. https://doi.org/10.1016/j.chb.2020.106392 [68] Punia, S. K., Kumar, M., Aggarwal, J., & Malik, K. (2020). Object based learning using multi-dimensional games. Journal of Discrete Mathematical Sciences and Cryptography, 23(2), 509–524. https://doi.org/10.1080/09720529.2020.1728904 [69] van der Lubbe, L. M., Gerritsen, C., Klein, M. C. A., & Hindriks, K. V. (2021). Empowering vulnerable target groups with serious games and gamification. Entertainment Computing, 38, 100402. https://doi.org/10.1016/j.entcom.2020.100402 [70] Pitura, J., & Terlecka-Pacut, E. (2018). Action research on the application of technology assisted urban gaming in language education in a Polish upper-secondary school. Computer Assisted Language Learning, 31(7), 734–763. https://doi.org/10.1080/09588221.2018.1447 490 iJIM ‒ Vol. 15, No. 21, 2021 35 https://doi.org/10.1016/j.edurev.2020.100322 https://doi.org/10.1016/j.edurev.2020.100322 https://doi.org/10.1080/15623599.2018.1492760 https://doi.org/10.1016/j.intmar.2016.03.001 https://doi.org/10.1080/10494820.2014.917109 https://doi.org/10.1080/09720529.2020.1728904 https://doi.org/10.1016/j.compedu.2019.103666 https://doi.org/10.1016/j.jhlste.2020.100267 https://doi.org/10.1016/j.jhlste.2020.100267 https://doi.org/10.1016/j.indmarman.2016.12.005 https://doi.org/10.1007/s10639-020-10394-y https://doi.org/10.1007/s10639-020-10394-y https://doi.org/10.1016/j.procs.2018.08.187 https://www.online-journals.org/index.php/i-jim/article/view/21137 https://www.online-journals.org/index.php/i-jim/article/view/21137 https://doi.org/10.1016/j.chb.2020.106392 https://doi.org/10.1080/09720529.2020.1728904 https://doi.org/10.1016/j.entcom.2020.100402 https://doi.org/10.1080/09588221.2018.1447490 https://doi.org/10.1080/09588221.2018.1447490 Paper—A Review of Gamification Impact on Student Behavioral and Learning Outcomes [71] Hwang, W.-Y., Shih, T. K., Ma, Z.-H., Shadiev, R., & Chen, S.-Y. (2016). Evaluating lis- tening and speaking skills in a mobile game-based learning environment with situational contexts. Computer Assisted Language Learning, 29(4), 639–657. https://doi.org/10.1080/0 9588221.2015.1016438 [72] Legaki, N.-Z., Karpouzis, K., Assimakopoulos, V., & Hamari, J. (2021). Gamification to avoid cognitive biases: An experiment of gamifying a forecasting course. Technological Forecast- ing and Social Change, 167, 120725. https://doi.org/10.1016/j.techfore.2021.120725 [73] W. S. Alhalafawy and M. Z. Zaki, ‘The Effect of Mobile Digital Content Applications Based on Gamification in the Development of Psychological Well-Being’, International Journal of Interactive Mobile Technologies (iJIM), vol. 13, no. 08, Art. no. 08, Aug. 2019, Accessed: May 30, 2021. [Online]. Available: https://www.online-journals.org/index.php/i-jim/article/ view/10725 7 Authors Ir. Muhammad Nurtanto, M.Pd., IPM, is an assistant professor in Department of Mechanical Engineering Education, Universitas Sultan Ageng Tirtayasa, Banten, Indonesia. Research interests in the fields of professional learning, learning media, STEM education, gamification, and teacher quality in vocational education. Email: mnurtanto23@untirta.ac.id Nur Kholifah, M.Pd, is a lecturer at the Department of Clothing and Food Engi- neering Education, Yogyakarta State University, Yogyakarta, Indonesia. Research interest in the field of empowering vocational communities, learning methods, voca- tional teachers, and gamification. Email: nur.kholifah@uny.ac.id Dr. Erif Ahdhianto, M.Pd., is a lecturer in the elementary school teacher education study program, Universitas Negeri Malang, Indonesia. His research interest is discuss- ing learning in elementary school. Email: erif.ahdhianto.fip@um.ac.id Dr. Achmad Samsudin, M.Pd., a lecturer and researcher at Department of Phys- ics Education, Universitas Pendidikan Indonesia, Bandung, Indonesia. Now a day, my tenure in academic position is Assist. Prof. My research interest is physics educa- tion especially in the misconceptions, conceptual change and understanding. Email: achmadsamsudin@upi.edu Fajar Danur Isnantyo, M.Sc, is lecturer and researcher at Department of Civil Engi- neering Education, Sebelas Maret University, Surakarta, Indonesia. Research interest in the field of green TVET, vocational character building and project-based learning including teaching factory in vocational education. Member of Asian Academic Soci- ety for Vocational Education and Training (AASVET) University Road, Douliu City, Yunlin County 640, Taiwan (R.O.C.). Email: isnantyo@staff.uns.ac.id Article submitted 2021-05-30. Resubmitted 2021-08-05. Final acceptance 2021-08-06. Final version published as submitted by the authors. 36 http://www.i-jim.org https://doi.org/10.1080/09588221.2015.1016438 https://doi.org/10.1080/09588221.2015.1016438 https://doi.org/10.1016/j.techfore.2021.120725 https://www.online-journals.org/index.php/i-jim/article/view/10725 https://www.online-journals.org/index.php/i-jim/article/view/10725 mailto:mnurtanto23@untirta.ac.id mailto:nur.kholifah@uny.ac.id mailto:erif.ahdhianto.fip@um.ac.id mailto:achmadsamsudin@upi.edu mailto:isnantyo@staff.uns.ac.id