International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 14, No. 3, 2020 Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning https://doi.org/10.3991/ijim.v14i03.11722 Chien-Hung Lai (), Bin-Shyan Jong, Yen-Teh Hsia Chung Yuan Christian University, Taoyuan City, Taiwan soulwind@cycu.org.tw Tsong-Wuu Lin Soochow University, Suzhou, Taiwan Abstract—Flipped classroom emphasizes the following. First, the instructor uses a digital platform to store course contents, including video lectures. Then, students retrieve these contents from the platform and learn by themselves before class. Finally, the instructor helps the students acquire their knowledge by asking questions in class, thereby helping the students to attain their learning goals. In previous studies, it was found that students might be affected by their peers and their own personalities, causing them not to express their opinions and views; this in turn might cause the students not to learn well; in some extreme cases, some students might even give up entirely. To solve the problem that students may hesitate to ask questions in class, this research developed a mobile anonymous question-raising system (MAQ) and used it to help students raise questions. By conducting an educational experiment, it was found that the use of MAQ in class could indeed help students to learn better in a flipped classroom setting. Keywords—Anonymous question-raising, blended learning, flipped classroom, problem-based learning. 1 Introduction Researchers have pointed out that combining classroom learning with online learning can bring a positive effect on the learning motivation of students [3] [9] [26]. Flipped classroom is a more recent teaching strategy. Compared to traditional didactic teaching, flipped classroom is versatile in that a variety of teaching strategies can be used in conjunction with digital learning. Students first use a digital platform to study course materials online and gain an understanding of both course contents and reasons why they need to know such contents. In the process, they also identify what they don’t understand well. The instructor then uses questions in class to guide students in their learning. This forces the students to reflect on their understanding of the topics and make the necessary adjustments. As a result, students are able to develop independent ways of thinking and improve their problem-solving abilities. By studying course materials before class and finding acceptable answers to questions (raised by the 66 http://www.i-jim.org https://doi.org/10.3991/ijim.v14i03.11722 Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning instructor and other students) in class, students are able to gain a deeper understanding of course materials [2] [14] [15] [24] [25]. Even though flipped classroom can help students understand more about course contents and improve the interactions between the instructor and students, not all students can benefit from the use of this teaching strategy. As some researchers suggest, if students do not interact well with the instructor in class, it is likely that they do not learn well. The use of flipped classroom is based on the premise that peers are met face to face [29]. This means that everyone knows what everyone else is asking. Therefore, students may be influenced by peer pressure and their own personalities, causing them to compare with one another and also have the fear of losing face. Consequently, they may not dare to ask questions owing to cognitive pressures such as fear, disgust and worrisome [8] [17]. Therefore, researchers have developed mobile learning systems for students to interact with their instructors in class; this includes the use of texting and voting to ask or answer questions [23]. It was found that the use of mobile devices to communicate with instructors could not only improve learning motivation but also lower learning pressure [20]. Therefore, it is not hard to see that with a good teaching strategy, mobile devices can be a very good assisted learning tool to use [19]. However, the effect of just answering questions from the instructor or using texting to send questions to the instructor may be limited. If there is a way to help students conduct peer discussions in an anonymous way, it can help to lower the fear of peers [16]. In addition, anonymous discussions can help students to find answers to questions and also improve their learning efficiency [1]. Based on the above discussions, this research investigates the effects of using a mobile, anonymous question-raising system (MAQ) for in-class learning assistance. This system was developed to help students asking questions in class, but do so in an anonymous way. The goal was to find out whether such a system is able to solve the problem of students not asking questions in class due to shyness or peer pressure. In other words, the goal was to investigate whether the use of a mobile system to ask questions in an anonymous way can effectively encourage students to raise their questions without having to care too much about peer pressure and personalities. To further enhance the students’ impression of course contents, questions raised, and answers to these questions, the system used in this research also includes functionalities for reviewing question-answering records. Students can review course contents, questions raised, and answers to these questions after attending classes, and they can also do so before quizzes and exams. Not only can each student review the questions he/she himself/herself has asked in class and the answers to those questions, the student can also review what others have asked in class and what the answers to those other questions were. The hope was that by going through these question-answering records, a better learning effect is achieved. This research aimed at finding answers to the following three research questions:  Is there a significant difference in learning achievement between the experimental group, which uses MAQ in class, and the control group, which does not use MAQ in class? iJIM ‒ Vol. 14, No. 3, 2020 67 Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning  For those who do not belong to the experimental group but nevertheless attend the same classes as members of the experimental group, will they be affected in their learning achievement?  Will those who do not belong to the experimental group but nevertheless attend the same classes as members of the experimental group feel somewhat being discriminated upon or isolated? 2 Related Works 2.1 Problem-based learning In 1965, McMaster University Medical School, newly established in Canada, adopted a new curriculum called problem-based learning for teaching and evaluation [7]. There are several characteristics with problem-based learning [7]. Learning is student-centered: Each student has to be responsible for his/her own learning, and he/she must also understand and manage the knowledge he/she needs and the questions he/she has. Learning occurs in small student groups: In the early days, problem-based learning was such that students were randomly assigned into groups of five to eight; upon the completion of each course unit, the students were regrouped. This was intended to help students to work in a serious and more effective way when they work with different people. Teachers are facilitators or guides: When the instructor discusses with students, he/she asks various kinds of questions without directly giving them answers. In other words, the instructor serves only as a guide on the side and helps students to reflect on their own thoughts and manage their own problems. Problems for the organizing focus and stimulus for learning: To put group discussions in focus and to stimulate thinking, the instructor can give appropriate questions for a group to find answers to. Problems are a vehicle for the development of clinical problem-solving skills: To develop clinical problem-solving abilities, one has to keep on solving problems. New information is acquired through self-directed learning: Students should actively acquire knowledge and professional ability by themselves. They need to work with their peers; in so doing, they should also discuss about and review what they have learned. Problem-based learning not only helps to establish students’ learning motivation but also help them to improve in their active learning ability and their ability to comprehend knowledge. Through the use of problem-based learning, students get improvements in inter-personal relations and their ability to work together; they also get improvements in intellectual development and integrated learning [4] [5]. Having participated in problem-based learning, students considered it more interesting than the traditional classroom learning, and they became more used to sharing with other group members, causing the collective intellectual ability of their respective groups to 68 http://www.i-jim.org Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning increase [10]. Spaulding [27], for example, observed that interns of a hospital became more eager to work with patients to solve problems together. 2.2 Flipped classroom Flipped classroom is a form of blended learning. It combines online learning and face-to-face teaching [3] [9]. The instructor first digitizes course materials and lectures and asks students to learn it online before they attend classes. Then, during class time, the instructor discusses questions with the students and guides them in problem solving. By helping students to solve problems and do practices, students learn by working on the four upper levels of (revised) Bloom’s taxonomy [14]: application, analysis, evaluation, and creation. Bristol [26], for example, let students stay home and learn medical knowledge online; then, he lectured about practical clinic experiences during class time. In retrospect, Bristol [26] considers flipped classroom helpful in helping students to learn more. On the one hand, it (flipped classroom) saved time for learning basic medical knowledge in class. On the other hand, the instructor and students could use class time for discussing clinical practices. 3 A Mobile Anonymous Question-Raising System for Assisting Flipped-Classroom Learning To encourage those who are shy in asking questions to ask questions in class and also to help students understand course contents, this research utilizes the convenience and immediateness of mobile devices and developed a mobile, anonymous question-raising system (MAQ) that assists flipped-classroom learning. By using MAQ, students can freely raise questions in class, and the instructor gets to know what the students do not understand when the class is in process. Students can use MAQ to do three things: 3.1 Raise questions in class As Figure 2 shows, when a student does not fully understand what the instructor talks about in class, he/she may want to ask questions. However, if the student did not do an adequate preview before class, he/she may not know what to ask. Even if the student knows what he/she does not understand, he/she may also have problems to express himself/herself clearly. Therefore, MAQ provides its user with two alternatives for asking questions. The first alternative is to choose from a list of likely questions, a list that was prepared beforehand by the teaching assistants; these questions are the ones that were raised in the past by those taking the same course. The second alternative is to directly enter one’s own questions. The ideal scenario then is the following. When a student is puzzled, he/she first checks the list of available questions and sees if there is any item in the list that matches with his/her puzzlement. If there is such a question in the list, he/she just selects it and clicks “send.” If there are no questions in the list that iJIM ‒ Vol. 14, No. 3, 2020 69 Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning correspond to his/her puzzlement, he/she then directly enters his/her question in the space provided and clicks “send.” Fig. 1. Question-raising functionality of the system 3.2 View questions raised by others in class As Figure 3 shows, it is also possible that when a student is puzzled, he/she cannot find his/her question(s) in the given list, nor can he/she directly specify his/her question. When this is the case, it may be helpful for this student to see what others are asking. For some students, it may also be of some comfort to know that “at least there are others who have the same question(s) as mine.” By allowing each student to look at what others are asking, it helps the student to identify what he/she does not understand. 70 http://www.i-jim.org Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning Fig. 2. Viewing questions raised by others 3.3 Review all questions raised in the course As Figure 4 shows, students can review, after class, what they have asked in class. This helps to strengthen their impression of what they have learned in class. By reviewing their own questions (along with the answers given in class), students can better prepare for the exams. It is also a way for students to make better use of their time when they study. For example, they can try to find better or more convincing answers to their own questions. iJIM ‒ Vol. 14, No. 3, 2020 71 Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning Fig. 3. Review all questions 4 The Experiment 4.1 Course used in the experiment and participants of the experiment The course used in this experiment was Systems Programming, offered by Department of Information and Computer Engineering of Chung Yuan Christian University, Taiwan. This course introduces three subjects to students enrolled in the department, and the three subjects were, respectively, database systems, software engineering, and operating systems. Teaching assistants of this course prepared a list of questions that the instructor of this course had asked over the years, and they put these questions in MAQ so that students could quickly go over relevant questions whenever the instructor talked about some specific topic in class. Participants of this experiment were 99 students taking that course. As Table 1 show, there were two classes, an “MAQ class” and a “normal class,” all taught by the same instructor. The normal class had 54 students and was designated as Control Group A. The MAQ class had 45 students and was divided into two groups. 18 students of the MAQ-class had Android based smart phones and were designated as the experimental group. 27 students of the MAQ-class, whose cell phones were not Android based, were designated as Control Group B. Members of the experimental group could use MAQ to raise questions in class and 72 http://www.i-jim.org Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning review questions after class, whereas Control Group A and Control Group B did not have MAQ to use. Table 1. MAQ class and normal class Class Group Number Use MAQ MAQ Class Experimental Group 18 O Control Group A 27 X Normal Class Control Group B 54 X 4.2 Procedure The experiment lasted for four weeks. The instructor used the strategy of flipped classroom of blended learning to teach. When teaching any topic in class, he used problem-based learning to guide the students in their learning. The students were also encouraged to ask their own questions in class. Prior to the experiment, all participants took a pretest to evaluate their learning achievement; the teaching assistants also taught members of the experimental group how to install MAQ in their smart phones. During these four weeks, members of the experimental group could freely raise questions (in an anonymous way and also in real time) in class. The instructor used (the instructor part of) MAQ to see what the students’ puzzlements were and also guide them in finding appropriate answers to their questions (along with other questions that the instructor had prepared for the class beforehand). The two Systems Programming classes were arranged in a way so that the experimental group (and Control Group B) always took their classes before Control Group A did; when the instructor later taught Control Group A, he always included in his question-answering session questions raised by the experimental group earlier in a separate class session. After the experiment, all participants took a posttest to evaluate their learning achievements; they also took part in a questionnaire survey to give feedback. 5 Result and Discussion 5.1 Learning achievement analysis Table 2 shows ANOVA analyses of pretest and posttest results of the three groups. By making pair-wise comparisons, it can be seen that for pretests, the F-value was less than the critical value for any two groups that are being compared. This means that all three groups had the same learning achievement before the experiment began. Table 2 also shows that there is a significant difference between the experimental group and Group A in posttest results. This means that the use of MAQ helps students to improve in their learning achievements. By using MAQ, students can raise questions in class in an anonymous way; this helps them to better understand what was taught in class. Even though members of Control Group A could still raise their hands and ask questions in class, they were nevertheless influenced by peer pressure and personalities in asking questions, causing them to understand less about what was taught in class. iJIM ‒ Vol. 14, No. 3, 2020 73 Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning Interestingly, there is no significant difference in posttest results between the experimental group and Control Group B. This means that students with no MAQ to use were influenced by students who used MAQ when the two groups took classes together. Even though members of Control Group B were not able to use MAQ to raise questions, they still benefited from questions raised by the experimental group. To some extent, knowing a question asked by the instructor actually comes from a peer (and not from the instructor himself) provides a stimulus for students to learn better in finding an appropriate answer to the question. This seems to be the case for all students in the class, even if some students of this class do not have MAQ to use. To find out more about what had happened to members of Control Group B, ten samples of these students were randomly selected for individual interviews. Section 5.3 analyses the results of these interviews. The average score of Control Group B is higher than the average score of Control Group A. However, there were no significant differences. This suggests that even though members of Control Group B benefited from knowing some questions were raised by the experimental group, they nevertheless have some questions of their own that remained unanswered (since they did not ask their own questions due to shyness or peer pressure), causing them not to understand fully about what was taught in class. Table 2. ANOVA analyses of pretest and posttest results of the three groups Stage Group Average Variance F Critical value α Pretest MAQ class 60.80 145.891 1.013 3.939 0.05 “normal” class 57.85 264.280 Experimental Group 63.56 156.026 1.582 4.067 0.05 Control Group B 58.96 136.114 Experimental Group 63.56 156.026 1.845 3.978 0.05 Control Group A 57.85 264.280 Control Group B 58.96 136.114 0.100 3.962 0.05 Control Group A 57.85 264.280 Posttest MAQ class 55.18 690.331 4.138 3.939 0.05 “normal” class 44.35 699.289 Experimental Group 60.61 725.075 1.291 4.067 0.05 Control Group B 51.56 660.103 Experimental Group 60.61 725.075 5.058 3.978 0.05 Control Group A 44.35 699.289 Control Group B 51.56 660.103 1.360 3.962 0.05 Control Group A 44.35 699.289 5.2 Analysis of questionnaire results about system usage To investigate how members of the experimental group felt about MAQ and how satisfied they were about its use, a questionnaire was designed. There are 13 items in the questionnaire. With the exception of the first question, which is a yes-no question, and the last question, which requires a simple answer, the rest are all questions that are answered using a four-point Likert scale, ranging from “strongly agree” to “strongly disagree.” 74 http://www.i-jim.org Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning After the experiment was over, a questionnaire survey was conducted for the experimental group. 18 responses were collected. Cronbach’s Alpha is 0.945. Therefore, the feedback provided by users of MAQ is trustworthy. Table 3 shows the questionnaire results. More than half of the subjects considered themselves to be of a shy type (Question 1). Therefore, by using an anonymous mechanism to raise questions, students that were shy had a way of asking the instructor questions. It also helped to improve the instructor-student interactions (Questions 2, 3, and 4). This is in agreement with the findings of earlier research, which suggested that the use of interaction tools could raise the level of interaction among the instructor and students, thereby helping to improve instructor-student relations [6] [11] [12] [18] [21] [22]. In general, students found MAQ useful in finding what they wanted to ask in the pre-arranged list of questions (Question 5). However, there were also some that preferred to phrase the questions by themselves (Questions 6 and 7). Most students agreed that the “questions raised by others” functionality of MAQ was useful (Question 8). This means that even though some might not know what their questions were, they could nevertheless identify their own puzzlements from questions raised by others in class. In addition, the students generally agreed that by reviewing questions that were raised in class, it helped them to review what was taught in class (Questions 9 and 10). Finally, from Questions 11 and 12, it can be seen that students generally gave approval to the use of MAQ in class and they also wished to use MAQ to raise questions in future classes. To further improve the working of MAQ and also to know how the subjects felt about MAQ, Question 13 allowed the subjects to write down their opinions about system use. Basically, the subjects thought that it might be better to have more platforms that support MAQ (since the current research only provides MAQ for Android-based smart phones). They also gave various suggestions regarding system use. For example, when the system is about to send a question to the instructor, there ought to be a confirmation dialog so that a student does not mistakenly send a question to the instructor. As another example, it may be desirable for each student to edit questions (along with the instructors’ answers to these questions) that were asked in class so as to rephrase the questions and answers in the student’s own words. This could help students to better prepare for quizzes and exams when they later review what were asked and answered in class. iJIM ‒ Vol. 14, No. 3, 2020 75 Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning Table 3. Questionnaire results for the use of MAQ Item number Item Options (%) Strongly agree Agree Disagree Strongly disagree 1 I consider myself a shy student. Yes (55.56) No (44.44) 2 I am glad to use MAQ to interact with the instructor. 22.22 61.11 16.67 0 3 I consider it very interesting to interact with the instructor through the use of MAQ. 22.22 72.22 5.56 0 4 When I use MAQ to raise questions, it will not show my name. This serves as a stimulus for me to use MAQ to ask questions. 16.67 66.67 11.11 5.56 5 The “I have a question” functionality of MAQ provides us with a list of prepared questions. It helps me to very quickly identify the question I want to ask. 11.11 83.33 5.56 0 6 I like to raise questions through the use of MAQ. 22.22 66.67 11.11 0 7 Compared to the use of “I have a question” functionality of MAQ, I like it better to directly key in the question I want to ask. 22.22 66.67 11.11 0 8 The “questions raised by others” functionality of MAQ helps me to see whether my questions are also others’ questions. 11.11 83.33 5.56 0 9 The “review of all questions raised” functionality of MAQ helps me to learn more about questions I did not quite have answers to in class. 27.78 66.67 5.56 0 10 The “review of all questions raised” functionality of MAQ helps me to review what I did not learn well in class. 27.78 66.67 5.56 0 11 On the whole, I like MAQ. 27.78 66.67 5.56 0 12 I hope we can still use MAQ in the next semester. 27.78 66.67 5.56 0 13 Please write down your comments or suggestions about the MAQ system itself and the use of it. 5.3 Interview results for control group B To understand how members of Control Group B felt about taking the same class with the experimental group but without MAQ to use, ten students of this group were randomly selected for interviews. The purpose was to see whether there were any negative effects on members of Control Group B and whether they felt being somewhat “left out.” Members of Control Group B took their classes with members of the experimental group. Did this help them to learn better? The actual question asked in the interview was: “Some of your classmates could use Android based smart phones to ask questions in an anonymous way. Then, they got answers from the instructor. Among the questions they raised, there might be questions that you also wanted to ask. Did you find it helpful when the instructor answered their questions in class?” Student 01: Yes. Student 02: Yes, indeed. 76 http://www.i-jim.org Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning Student 03: Yes (it helped). Even if I knew the answer to that question, the instructor’s explanation might help me to understand the whole thing better. Student 04: It was very helpful. Since I might not have thought of those questions, the fact that they asked those questions helped me to learn better. Student 05: Yes. Student 06: Yes, because sometimes I did not know how to ask. When that was the case, it helped when my classmates asked what I wanted to ask (but did not know how to ask). Student 07: There is a prerequisite for getting help from what they asked, and the prerequisite is that the one who asks questions (using MAQ) must have seen the (instructor’s) film first; otherwise, he/she would not know the right question to ask. If what the person asked was what I did not understand, it could be very helpful. Student 08: I think this is very helpful, because I could not use my iPhone (with iOS) to ask. Student 09: It is very helpful for catching up with the progress of the course. Sometimes, the questions asked happened to be what I did not understand, and I got to know the answers. But if I did not understand the questions in the first place, it would not be helpful. Anyway, when the instructor explained, I more or less got to learn some extra-curricular knowledge that was helpful. Student 10: I think MAQ is a nice system, because it makes it possible for those who usually do not ask questions to raise their questions. It also helped us to notice questions that we might not pay attention to. From the students’ answers to this question, it can be seen that most students who were interviewed felt that even if they themselves did not have MAQ to use, they nevertheless benefited from the fact that members of the experimental group used MAQ to raise questions. This may be the reason why Group B also had a higher learning achievement when compared to their counterparts in Group A. Did students feel somewhat isolated, since they did not have MAQ to use (while some of their classmates had MAQ to use)? The actual question asked in the interview was: “Some classmates of yours had MAQ to use, and they did use MAQ to raise questions to the instructor. Do you feel that this is unfair to other classmates like you who did not have MAQ to use (to raise questions)?” Student 01: No! Because we are always free to ask questions anytime. For example, if I know what stack overflow is and others do not know it, then I have an edge over others. The resource (i.e., the instructor who can answer questions) is there. It is just a matter of whether you know that it is there and whether you know how to use this resource. Student 02: Yes! Because those who had a question, did not have MAQ to use, and also did not dare to ask could not ask their questions (and get the answers they needed) in time. Student 03: No. If anyone has a question, he/she can always go to the instructor and ask. Student 04: I think it is OK. There are many ways to raise one’s questions to the instructor (and MAQ is only just one of the ways). iJIM ‒ Vol. 14, No. 3, 2020 77 Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning Student 05: Actually, it is OK. If one is serious in asking questions, he/she can do it without the use of MAQ. Student 06: I don’t feel that it is unfair. If you cannot use smart phones or pad computers to raise your questions, you can also raise your hand (and ask questions) in class; or you can also go to the instructor during class breaks and ask your questions. I believe the instructor is always happy to answer your questions. In other words, one can still ask questions without using smart phones or pad computers. Student 07: Even though I did not use MAQ to raise questions, I do not feel that it is unfair. Student 08: No. Because we can also raise our questions on i-learning. Student 09: Actually not. Because when they got answers from the instructor for the questions they raised, we also benefited from it. Besides, I don’t think there is so much difference between directly asking questions (to the instructor) and using MAQ to raise questions. Student 10: No. It is just that those without Android-based smart phones will have to borrow Android-based smart phones from their classmates in order to raise their questions using MAQ. This is a little bit troubleso me. Corbeil & Valdes-Corbeil [13] pointed out that students without mobile devices had a feeling of being isolated. However, this is not the case with Control Group B in this experiment, because they also benefited from their classmates who used MAQ to raise questions. From the results of the interview, it can be seen that members of Control Group B did not feel that it was unfair, because they had other alternatives of asking questions. The use of MAQ in this experiment not only helped those in the experimental group to learn better by asking questions, it also helped those in Control Group B to learn better, because members of Control Group B were also affected when their MAQ-counterparts raised questions in class. Therefore, it may not be necessary for everyone in a class to use MAQ to raise questions. When those with Android based smart phones use MAQ to raise questions, those without Android based smart phones (including those who had no smart phones) could be stimulated, thereby causing all students of the same class to learn better. 6 Conclusion This research associates MAQ, an anonymous question-raising system, with flipped classroom, a form blended learning. Central to this approach is problem-based learning. The instructor uploads video lectures and course-related materials on the Web, making it possible for students to learn course content online by themselves before they come to class. Then, the instructor helps the students understand course topics by discussing questions in class. In so doing, the instructor provides guidance to students and interacts with them so that the students learn by solving problems [4] [5] [7] [10] [27]. One problem with this approach is that some students may be shy in raising questions to the instructor, causing them not to learn well [28]. This is what MAQ was developed for. With the use of MAQ, students that are shy or in fear of peer pressure can freely raise questions in class, without having to reveal their true identities. 78 http://www.i-jim.org Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning A related educational experiment demonstrated that not only could MAQ help students that are shy to raise questions in class, the use of MAQ could also help to alleviate learning achievement. It was also found that, so far as learning achievement is concerned, there is no need for all students to have MAQ to use. When students with MAQ raised questions in class, those without MAQ could benefit from their understanding that their classmates had raised such questions, causing these latter students to also learn well. 7 Acknowledgement Wei-Ching Lin helped perform the experiment and collect and analyze data. 8 References [1] B.S. Jong, C.H. Lai, Y.T. Hsia, T.W. Lin (2013). Effects of Anonymity in Group Discussion on Peer Interaction and Learning Achievement, IEEE Transactions on Education, 56(3), 292-299. https://doi.org/10.1109/TE.2012.2217379 [2] C. Juwah (2002). Using communication and information technologies to support problem-based learning, Article of Higher Education Academy. [3] C. Redmond (2014). Flipping Primary Literacy Instruction. [4] E. Bridges, P. Hallinger(1991). Problem-based learning: A promising approach for preparing educational administrators. UCEA Review, 32(3), 3-7. [5] H. Mandin, P. Harasym, C. Eagle, M. Watanabe (1995). Developing a clinical presentation curriculum at the University of Calgary. Acad Med 70:186-193. https://doi.org/10. 1097/00001888-199503000-00008 [6] G. Jancke, J. Grudin, A. Gupta (1999). Presenting to local and remote audiences. Technical ReportMSR-TR-99-71, Microsoft Research. https://doi.org/10.1145/332040.332461 [7] H.S. Barrows (1996). Problem-based learning in medicine and beyond: A brief overview. New Directions for Teaching and Learning, 68, 3-12. https://doi.org/10.10 02/tl.37219966804 [8] H. Smith, A. Cooper, & L. Lancaster, (2002). Improveing the quality of undergraduate peer assessment: a case for student and staff development. Innovations in Education and Teaching International, 39, 71-81. https://doi.org/10.1080/13558000110102904 [9] H. Staker, M.B. Horn (2012). Classifying K-12 blended learning, Innosight Institute. [10] J. Biggs (2000). Teaching for quality learning at university. Buckingham, UK:Open University Press. [11] J. Roschelle, R. Pea (2002). A walk on the WILD side: How wireless handhelds may change CSCL. In Proceedings of Computer Supported Collaborative Learning 2002. https://doi.org/10.3115/1658616.1658624 [12] J.L. Fitch (2004). “Student Feedback in the College Classroom: A Technology Solution.”, Educational Technology Research and Development. ETR&D, 52(1), 71-81.https://doi.org/10.1007/BF02504773 [13] J.R. Corbeil, M.E. Valdes-Corbeil (2007). Are You Ready for Mobile Learning? EDUCAUSE QUARTERLY Number 2 2007. https://doi.org/10.1504/IJMLO. 2008.018717 [14] L. W. Anderson, D. R. Krathwohl, Peter W. Airasian, Kathleen A. Cruikshank, Richard E. Mayer, Paul R. Pintrich, James Raths, & Merlin C. Wittrock (2000). A Taxonomy for iJIM ‒ Vol. 14, No. 3, 2020 79 https://doi.org/10.1109/TE.2012.2217379 https://doi.org/10.%0b1097/00001888-199503000-00008 https://doi.org/10.%0b1097/00001888-199503000-00008 https://doi.org/10.1145/332040.332461 https://doi.org/10.10%0b02/tl.37219966804 https://doi.org/10.10%0b02/tl.37219966804 https://doi.org/10.1080/13558000110102904 https://doi.org/10.3115/1658616.1658624 https://doi.org/10.1007/BF02504773 https://doi.org/10.1504/IJMLO.%0b2008.018717 Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, published by Pearson. [15] P. Pusawiro (2012). Hybrid teaching and Flipped Classrooms, article of Thailand THE NATION, retrieved. [16] M.L. Wen, C.C. Tsai, C.Y. Chang (2006). Attitudes towards peer assessment: a comparison of the perspectives of pre-service and in-service teachers. Innovations in Education and Teaching International, 43, 83-92. https://doi.org/10.1080/14703290500467640 [17] M. R. Paquet, & J. E. Des Marchais (1998). Students’ acceptance of peer assessment, Education for Health, 11, 25-35. [18] M. Ratto, R.B. Shapiro, T.M. Truong, W.G. Griswold (2003). The Activeclass Project: Experiments in Encouraging Classroom Participation. Computer Support for Collaborative Learning (CSCL). S. 477-486. https://doi.org/10.1007/978-94-017-0195-2_57 [19] N.A. Gromik (2012). Cell phone video recording feature as a language learning tool: A case study. Computers & Education, 58(1), 223–230. https://doi.org/10.1016/j. compedu.2011.06.013 [20] P.L.P. Rau, Q. Gao, L.M. Wu (2008). Using mobile communication technology in high school education: Motivation, pressure, and learning performance. https://doi.org/10. 1016/j.compedu.2006.03.008 [21] R. Kern(1995). Restructuring classroom interaction with networked computers: Effects on quantity and quality of language production. Modern Language Journal, 79, 457-476https://doi.org/10.1111/j.1540-4781.1995.tb05445.x [22] R. Cadima, C. Ferreira, J. Monguet, J. Ojeda, J. Fernandez (2010). Promoting social network awareness: A social network monitoring system. Computer & Education, 54(4), 1233-1240. https://doi.org/10.1016/j.compedu.2009.11.009 [23] R. Shen, M. Wang, W. Gao, D. Novak, L. Tang (2009). Mobile Learning in a Large Blended Computer Science Classroom: System Function, Pedagogies, and Their Impact on Learning. IEEE Transactions on Education, 52(4), 538-546. https://doi.org/10.1109/TE.2008.930794 [24] S. Khan, (2011). Let's use video to reinvent education, filmed in TED. https://doi.org/10.1037/e530962011-001 [25] S. Khan (2012). Khan Academy: The future of education? interview with CBSNEWS 60 Minutes. [26] T. Bristol (2014). Flipping the Classroom. Teaching and Learning in Nursing, 9(1), 43–46. https://doi.org/10.1016/j.teln.2013.11.002 [27] W.B. Spaulding (1991). Revitalizing Medical Education. McMaster Medical School in the Early Years, 1965-1974. Philadelphia: Decker. [28] A. Thatcher and G. Mooney (2008). Managing Social Activity and Participation in Large Classes with Mobile Phone Technology. Internation Journal of Interactive Mobile Technologies (iJIM), 2(3), 41-51. [29] K. Umam, T. Nusantara, I N. Parta, E. Hidayanto, & H. Mulyono (2019). An Application of Flipped Classroom in Mathematics Teacher Education Programme. Internation Journal of Interactive Mobile Technologies (iJIM), 13(3), 68-80. https://doi.org/10.3991/ ijim.v13i03.10207 9 Authors Chien-Hung Lai is an assistant professor in the Department of Electronic Engineering, Chung Yuan Christian University. His research interests include computer-aided education. Email: soulwind@cycu.org.tw 80 http://www.i-jim.org https://doi.org/10.1080/14703290500467640 https://doi.org/10.1007/978-94-017-0195-2_57 https://doi.org/10.1016/j.%0bcompedu.2011.06.013 https://doi.org/10.1016/j.%0bcompedu.2011.06.013 https://doi.org/10.%20%0b1016/j.compedu.2006.03.008 https://doi.org/10.%20%0b1016/j.compedu.2006.03.008 https://doi.org/10.1111/j.1540-4781.1995.tb05445.x https://doi.org/10.1016/j.compedu.2009.11.009 https://doi.org/10.1109/TE.2008.930794 https://doi.org/10.1037/e530962011-001 https://doi.org/10.1016/j.teln.2013.11.002 https://doi.org/10.3991/%0bijim.v13i03.10207 https://doi.org/10.3991/%0bijim.v13i03.10207 ../KK%20Comments/soulwind@cycu.org.tw Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning Bin-Shyan Jong is a professor in the Department of Information and Computer Engineering, Chung Yuan Christian University. His research interests include computer graphics and computer aided education. Email: bsjong@ice.cycu.edu.tw Yeh-Teh Hsia is now a professor in the Department of Information and Computer Engineering, Chung Yuan Christian University. His research interests include computer aided education and artificial intelligence. Email: hsia.yenteh@gmail.com Tsong-Wuu Lin is a professor in the Department of Computer Science and Information Management, Soochow University. His research interests include computer graphics and computer aided education. Email: twlin@csim.scu.edu.tw Article submitted 2019-09-22. Resubmitted 2019-12-08. Final acceptance 2019-12-08. Final version published as submitted by the authors. iJIM ‒ Vol. 14, No. 3, 2020 81 ../KK%20Comments/bsjong@ice.cycu.edu.tw ../KK%20Comments/hsia.yenteh@gmail.com mailto:%20twlin@csim.scu.edu.tw mailto:%20twlin@csim.scu.edu.tw