 Proceedings of Engineering and Technology Innovation, vol. 10, 2018, pp. 08 - 12 A Study on the Development of Game-based Mind Wandering Judgment Model in Video Lecture-based Education Jaechoon Jo, Yeongwook Yang, Heuiseok Lim * Department of Computer Science and Engineering, Korea University, Seoul, Korea Received 02 March 2018; received in revised form 30 May 2018; accepted 16 July 2018 Abstract Although video lecture materials are very efficient learning materials, they are likely to be unilateral learning materials by the lecturer. It is easily degraded to be one-sided learning, which has been considered as a problem of online education, and it is difficult to judge whether learners are actually learning. Therefore, in this paper, a minimum learning activity judgment model that can automatically determine if they actually learn through mind wandering judgment was proposed to overcome the limitations of previous learning materials, and educational effect verification experiment was performed. Experiment results show that the video lecture class using the minimum learning activity judgment system was effective in improving the academic achievement. Keywords: online education, e-assessment, flipped classroom, judge system, minimum learning activity 1. Introduction Education has taken a path of computerization, as industry and technology have developed and the society has been computerized. Beginning with the remote education in the 1900s, e-learning using electronic media emerged in 2000 as the Internet developed. E-learning has made many changes in the target, place, time, medium, and method of education. People can receive the desired education regardless of their age or status. They can have education at the place and time and with the medium they want. E-learning has increased the demand and satisfaction of many learners in a revolutionary way without the limitation of time and space. With the development of information technology, even college classes can be taken online. It has led to the creation of an environment like ‘massive open online courses (MOOC)’ [1]. MOOC is an online learning process aiming at the environment where anyone who wants to study can participate and openly access the web. MOOC provides students with the opportunity to earn regular credits or degrees through e-learning lectures as well as to acquire the necessary knowledge [2]. However, the environment of e-learning can make a lack of communication between the teacher and leaners because it is a non-real-time and indirect learning environment comparing to face-to-face classes [3]. Of course, there has been much effort in MOOC to enforce communication between the teacher and learners stepping further from the unilateral remote education. However, in many online learning programs including MOOC, the problems such as insufficient concentration, necessity to compose a self-study environment and lack real-time feedback still remain [3-5]. To overcome such problems, teachers have proposed online quizzes or tasks to learners in online education environment to induce concentration and judge if they have actually learned. However, such methods make teachers use more time and money and learners have additional burden. Another automatic method to check if the learners have actually learned is video playback time. However, this method is only * Corresponding author. E-mail address: limhseok@korea.ac.kr Tel.: +82-2-3290-2396 Proceedings of Engineering and Technology Innovation, vol. 10, 2018, pp. 08 - 12 Copyright © TAETI 9 based on the playback time of the video lecture, so it is difficult to judge whether the learner is really learning or he/she has concentrated on the lecture [6]. Many online education systems currently use pop-up events or online quizzes in the middle of video lectures to determine whether learners are watching the video lecture, or allow the learners to go to the next lecture only when the special event in the middle of the lecture is clicked to verify if they have actually participated in the learning process. However, those existing methods may interrupt the learning flow of learners, or may create ineffective events even in the parts which are not necessary to be learned. Although video lecture materials are very efficient materials, they are likely to be unilateral learning materials by the instructor. It is because the video lecture is not based on the communication with the lecturer but a unilateral lecture of the lecturer. The lecturer has no means to check if the learning contents have been successfully delivered to leaners. The lecturer can check the learning status of students only when he/she asks questions to each student about the contents of the lecture or give them a quiz. However, it is very inefficient if the lecturer has to take pains to make questions and to mark in order to check whether they have learned from the video lecture. Therefore, such judgment on learning status is rarely made. The concerns relating to online education such as likelihood to be unilateral learning and difficulty in the judgement of learning status should be supplemented using computing technology. Accordingly, in this paper, the minimum learning activity judgment model to check the status of learning of learners was proposed to overcome the limitations of existing methods and an educational effectiveness test was conducted. 2. Minimum Learning Activity Judgment Model In this paper, a minimum learning activity judgment model to automatically determine the learning status of learners through mind wandering judgment in video lecture based online education was proposed. The minimum learning activity is defined as the activity to watch a video without mind wandering. It is called minimum learning, as it does not matter how much the learners have understood what they watched. Minimum learning activities focus more on learning efforts (learning activities) than on understanding or understanding the level of the learning contents. In order to automatically determine the minimum learning activity, it should be judged if there was mind wandering while they watched the video lecture. Therefore, the word frequency effect theory of language linguistics is applied to determine mind wandering. The word frequency effect is a theory that there is a difference in cognitive responses between high frequency words and low frequency words, and there is also difference in cognitive responses between words that people actually know and words they do not know. The minimum learning activity judgment model determines the minimum learning activity by calculating the difference in the word-recognition reactions of the learners. To do this, a word game was developed. In other words, a minimum learning activity judgment system applying word game was developed. Fig. 1 Structure of word game Fig. 1 shows the word game structure of the system. The word game is an O/X type game. It automatically extracts high frequency words from a video lecture that a learner has watched and mixes them with words that have never appeared in the video lecture. The learner will choose ‘O’ for the word that appeared in the video lecture, and ‘X’ for the word that has not appeared in the lecture. The system determines whether the minimum learning activity has occurred by calculating the number of correct answers and the recognition response time. Proceedings of Engineering and Technology Innovation, vol. 10, 2018, pp. 08 - 12 Copyright © TAETI 10 The minimum learning activity is judged as pass or fail through the number of correct answers and recognition response time. According to language recognition discipline, it takes 0.7 to 1.2 seconds when people see a word that they know. This was applied to the response time to judge the recognition. As the word game in this paper was implemented in a web environment, +- 0.4~0.5 second was given as tolerance considering the web loading environment. Accordingly, 0.3~1.7 seconds was set as recognition response time. In other words, if a leaner chose the right answer within the response time, it is considered that the learner has watched the video lecture without mind wandering. In order to set the standard score to judge the minimum learning activity, the minimum learning standard score was obtained through an experiment. Sixty domestic college students participated in the experiment. Participants watched the video lecture and checked whether they were focused on a 5-point scale. After watching the video lecture, they were divided into two groups using word game results. Those who had 11 correct answers out of 14 (80% correct answers) were classified as the group that focused on the video lecture, and those who had 7 correct answers out of 14 (50% correct answer) were classified as non-focused group. In other words, the hyper-parameter of the judgment standard score was set as 0.8 based on the experimental result. 3. Experiments and Results To investigate the educational effect, this experiment was conducted on 123 5th grade elementary school students for 5 weeks. The experimental group consisted of 62 students (30 males and 32 females) and the control group consisted of 61 students (30 males and 31 females). The homogeneity of two groups was investigated through the result of the pre-test comparing academic achievements of students in two groups. The results of four students in the control group who did not take the academic achievement test were excluded. To compare the academic ability of the experimental group and the control group, Korean, English and Math questions were presented and they were asked to take the test under the same conditions. In order to verify the difference of academic ability between groups, the independent t test was conducted. In the t-test, there was no significant difference between groups in Korean (t = -0.602, df = 113, p = 0.548), mathematics (t = 0.239, df = 113, p = 0.812) and English. (t = 1.963, df = 82, p = 0.053. In English, the equal variance assumption was significant, but it was not significant with the corrected t value (t=1.963, df = 82, p = 0.053). The total score of the three subjects, which can be regarded as the average academic ability, was not statistically different between the experimental group and the control group (t=0.799, df = 113, p = 0.426). As shown in Table 1. Before the experiment, they took the academic achievement test. After the pre-test, the system using the minimum learning activity judgment was administered to the experimental group and the conventional learning system using the traditional online video lecture was administered to the control group (no word game, no learning judgment function). In order to investigate the change in academic achievement during the experimental period for 5 weeks, they had interim and post-test at the third week and the fifth week. As an academic achievement test tool, diagnostic evaluation of the elementary school curriculum was used. Table 1 Result of academic achievement Group Mean SD N Pre Test Experimental 67.290 14.8880 62 Control 67.123 16.8820 57 Total 67.210 15.8065 119 Mid Test Experimental 85.774 12.3081 62 Control 76.193 13.2953 57 Total 81.185 13.6127 119 Post Test Experimental 86.484 11.8719 62 Control 68.298 16.5583 57 Total 77.773 16.9178 119 Proceedings of Engineering and Technology Innovation, vol. 10, 2018, pp. 08 - 12 Copyright © TAETI 11 There was a statistically significant difference in academic achievement between the groups (f(1,117)=16.997, p<0.001). There was no statistically significant difference between the control group and the experimental group in the pre-test (f(1,117)=0.003, p=0.954), but both groups showed statistically significant difference in interim and post-test (f(1,117)=16.665, p<0.01, f(1,117)=47.977, p<0.01). In the experimental group, there was a statistically significant difference between the pre- and interim and between pre- and post-test (f(1,117)=16.665, p<0.01, f(1,117)=47.977, p<0.01), but there was no statistically significant difference between interim and post-test (f(1,61)=0.393, p=0.533). In control group, there was a statistically significant difference between the pre- and interim and between interim (f(1,56)=25.766, p<0.01, f(1,56)=17.101, p<0.01), but there was no statistically significant difference between pre- and post-test (f(1,56)=0.278, p=0.60). 4. Conclusions This study proposed a minimum learning activity judgment model to judge the learning status of learners through mind wandering in the video lecture-based education. To administer a comparative experiment of educational effectiveness, video lecture learning was carried out using the system developed for the 5th grade of elementary school students for 5 weeks. Additionally, academic achievement tests were given as pre -, interim, and post-test. Experimental results showed that the video lecture class using the minimum learning activity judgment system was effective on the improvement of the academic achievement. In the case of the experimental group, the improvement between the pre-test and the interim test, and between the pre-test and post-test was statistically significant. In the case of the control group, the improvement between pre-test and interim test and between interim test and post-test was statistically significant, but the improvement between the pre-test and the post-test was not significant. The academic achievement of the two groups was similar in the pre-test but there became significant difference from the interim test (the achievement result at the 3rd week evaluation). Especially, in the academic achievement tests, the control group showed score drop between the interim test and the post test. On the other hand, the experimental group showed a continuous increase. Given that the difficulty of learning increases over time from the pre-test to post-test, it can be considered that the minimum learning activity system has not only short-term and simple effect but also long-term effect on learners. In addition, according to the interviews with learners, the experimental group students who used the minimum learning activity judgment system did focus more on the video lecture for the word game. And they had interest and tension thinking of their ranking in the game after playing the game. It seems that these interests and tensions have positive effects on learning, as Keller's ARCS theory. This concentration leads to the concentration of learning time. The result of judgment if students have actually learned enables the lecturer to give them fast and efficient feedback. This can be thought of as enhancing learning achievement. In the control group, they showed improvement until the interim test as the introduction of the new method made them concentrate at the beginning of the experiment. However, in the long term, the effect of the method administered to the control group is negligible. In other words, learners can concentrate on video lecture and the class contents more, when a video lecture is applied with follow-up treatment rather than a simple video lecture without such treatment. It can be considered that the minimum learning activity judgment system can give learners satisfaction with learning by changing the characteristic video lecture from unilateral delivery to be-directional interaction. Acknowledgement This research is supported by Ministry of Culture, Sport and Tourism(MCST) and Korea Creative Content Agency(KOCCA) in the Culture Technology(CT) Research & Development Program 2018. (No. R2016030031) Proceedings of Engineering and Technology Innovation, vol. 10, 2018, pp. 08 - 12 Copyright © TAETI 12 References [1] D. F. O. Onah and J. Sinclair, “Learners expectations and motivations using content analysis in a MOOC,” EdMedia 2015 - World Conference on Educational Media and Technology, vol. 2015, no. 1, June 2015, pp. 185-194. [2] C. J. Bonk, M. M. Lee, T. C. Reeves, and T. H. Reynolds, MOOCs and open education around the world, United Kingdom: Routledge, 2015. [3] J. Jang, I. Ko, and C. Jeong, “The impact of e-learning system characteristics on learner commitment and satisfaction,” Journal of Information Technology Applications & Management, vol. 15, no. 1, pp. 99-116, 2008. [4] D. S. Zhang, L. Zhou, R. O. Briggs, and J. F. Nunamaker Jr. “Instructional video in e-learning: assessing the impact of interactive video on learning effectiveness,” Information & management, vol. 43, no. 1, pp. 15-27, January 2006. [5] M. T. H. Chi, “Active‐constructive‐interactive: A conceptual framework for differentiating learning activities,” Topics in cognitive science, no.1, no. 1, pp. 73-105, January 2009. [6] P. Pham and J. T. Wang, “AttentiveLearner: improving mobile MOOC learning via implicit heart rate tracking,” Proc. International Conf. Artificial Intelligence in Education, Springer Press, June 2015, pp. 367-376.