Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 173 engaging and assessing students through their electronic devices and real time quizzes Ferrándiz, E.*, Puentes, C., Moreno, P.J., Flores, E. Department of Economics, Universidad de Cádiz *esther.ferrandiz@uca.es; Received: 2016-01-23; Accepted: 2016-03-13 Abstract This paper describes a teaching experience using Socrative, a third party electronic tool, for real-time questioning in lectures of Econometrics. Econometrics is a theoretical-practical subject, but traditionally a large proportion of our students tend to focus on the practical and discard the theory, often skipping classes on theory and avoiding studying its content, probably motivated by its complexity. As a consequence, students’ marks obtained in the theoretical part of the exam are usually low. In this context, we put forward a change in our teaching methodology to include the use of Socrative, a freely available app, that allows students to answer teachers’ short, true/false, or multiple choice questions posed during each class using their smartphones (or other electronic devices with Internet connection). The objectives of this project are twofold: 1) to engage students and increase attendance at lectures; 2) to improve feedback on the learning process. The results of a survey of a sample of 186 students reveal that Socrative has been an effective tool for achieving these objectives. Keywords Students response system, clicker, quizzes, active learning, student engagement, Socrative, gamification http://dx.doi.org/10.4995/muse.2016.6375 mailto:esther.ferrandiz@uca.es Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 174 1. Introduction Information technology (IT) offers a powerful opportunity for increasing student engagement (Roblyer & Wiencke 2003). Prominent examples of technology include student response systems (SRSs),1 which allow an instructor to pose questions to the class, allow students to enter their answers in some kind of device (e.g., clicker or smartphone) and instantly summarize and graphically represent results for the instructor (Beatty 2004), who can decide whether the responses are anonymous or not. Terrion and Aceti (2012) have highlighted the importance of student engagement because unengaged students do not listen, process, or attend to the learning process. Several papers have analysed the role of SRSs as a means of engaging students and making them active in the classroom, among other benefits (for a review see Aljaloud et al. 2015). Although much of the research in SRS has traditionally been related to the use of clickers, the development of cloud-based software that enables any device to become an SRS overcomes some of the criticism of clickers such as cost, and allows educators to exploit the potential of personal electronic devices, mostly smartphones, for educational purposes. Although education is not among the main uses of these devices yet, this use is expected to grow in the coming years as new applications are developed. Companies such as Apple have already foreseen this possibility, developing iTunes U, an application store especially focused on education. The highest rates of smartphone ownership are among the richer economies and the top countries in the rank include South Korea (88%), 77% of Australia (77%), Israel (74%), USA (71%) and Spain (71%), with a greater presence among young people (Pew Research Center, 2016). Gaming (65%), social networking (54.3%), and instant messaging (48.3%) are among the most common uses minors and younger people have for their smartphones’ advanced features (INTECO, 2011). These statistics are in line with Yusof et al. (2012), who argue that smartphones are mainly used for entertainment and social networking. Therefore, the potential of smartphones for education (m-learning) has yet to be unlocked. 1 SRSs are also refered to as classroom performance systems, audience response systems, personal response systems, classroom communication systems, electronic response systems, electronic voting systems, polling systems, or clicker systems (Aljaloud et al. 2015). http://dx.doi.org/10.4995/muse.2016.6375 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 175 Additionally, the subject of Econometrics taught in the Business and Administration Degree program relies on theoretical reasoning and computer practices that students often find difficult to understand. This, jointly with the limited time available in lectures and the complexity of the subject, hinder the learning, comprehension, and motivation of students when they deal with Econometrics. Motivated by all the above-mentioned reasons, instructors of Econometrics at the University of Cadiz have proposed an innovation project based on Socrative, a cloud-based SRS. The main objectives of the project were two: 1) increase attendance and engagement of students in theoretical classes and 2) improve the feedback process to students regarding their learning processes. This paper is structured as follows. Section 2 summarizes the advantages and disadvantages of SRSs. Section 3 describes the methodology and the implementation of the study. Section 4 describes the results; and finally, in section 5 we draw some conclusions and comment on future work. 2. Benefits and drawbacks of using student response systems Kay and LeSage (2009a) and, more recently, Aljaloud et al. (2015) review the literature on the benefits of using SRSs, which are related not only to engagement, but also to interactivity and academic performance. Focusing on the former, previous research has shown that SRSs enhance student engagement in learning by creating a fun learning environment. Further, the researchers found increased attendance and positive student attitudes and an increase in the students’ desire to improve their performance by identifying areas of improvement. Because of increased engagement, an SRS has the potential to improve academic performance. Finally, SRSs improve the process of instructor feedback since the instructor has more information about the learning processes of their students. However, there are also some drawbacks, such as time required for teachers to formulate good questions and deal with technical problems. Other often-cited inconveniences of clickers, such as cost or time to deliver the clickers to students in each class, are overcome thorough the use of cloud-based software such as Socrative, which enables any electronic device (such as students’ smartphone or tablets) to be used as a clicker. http://dx.doi.org/10.4995/muse.2016.6375 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 176 3. Methodology This project focused on 385 students enrolled in Econometrics classes in the Business and Administration Degree program at the University of Cadiz. Econometrics is taught in the second semester to third-year students on three different campuses, involving seven teachers. The average number of participants was 160, spread among five classes (i.e., 32 students per class). The project mainly consisted of the regular use of Socrative quizzes during lectures. The procedure to develop the activity followed several steps: 1. Design and training of teachers. This phase includes the following tasks: a) Elaboration of questions in Socrative for each of the 10 lessons of the syllabus. Each quiz had 3-4 questions and was focused on relevant contents explained in class. We included multiple choice and true/false questions, bearing in mind that the most effective questions to be used in SRS are those (Kay and LeSage 2009b) that allow students to apply knowledge recently acquired; are higher level; focus on process and reasoning as opposed to factual content; identify and help resolve misconceptions; and support a comprehensive review of a specific set of concepts. b) Statement of the rules of the game. - The participation with Socrative is individual unless the teacher says the opposite, and is limited to students who have attended all the classes. - The use of electronic devices is only allowed for responding to teachers’ questions. - Students are responsible of bringing a working electronic device with Internet connection. The University provides free working Wi-Fi for students in their facilities. c) Selection of the reward. Students who regularly attend class and provide correct answers to at least 75% of the questions would get 0.5 extra points added to their final mark. d) Training of teachers. The coordinator of the project developed a training session for teachers and elaborated a visual guide on how to use Socrative, covering some of the following aspects: - How to create a personal account in Socrative.com - How to import a quiz previously elaborated by the coordinator and how to start it. Regarding the latter, we used the following options:  Students pace the quiz, which allows them to navigate through it at their own pace. http://dx.doi.org/10.4995/muse.2016.6375 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 177  Disable immediate feedback. We chose this option because we wanted to keep the correct answer concealed until the entire class had submitted their answers to make cheating less attractive.  Random questions and answers. We used these options to make cheating more difficult. - How to review the class’ answers, save reports and get bar graphs to give feedback to the students. - 2. Information session for students and system testing. In the first class of the course, each teacher introduced the subject to his/her class, including the methodology and the evaluation system. Regarding the latter, the instructor introduced Socrative and presented the rules of the game. Subsequently, we made a trial test in class to check that the Internet connection worked well in the classroom and to get students familiar with the system. In order to facilitate the link between student names and their marks in Socrative, avoiding misspellings, students were asked to log in with their identity card number instead of their names. 3. Classes, quizzes and exit tickets. Once students and teachers knew how to use Socrative, we began the course. In each session, after the teacher explained the theoretical content, he/she gave the students a quiz. By the end of the class, they had to obtain an “exit ticket” that included the following questions: How well did you understand today’s material? What did you learn in today’s class? and, optionally, a teacher’s open question. The latter often polled the student’s desire to explain a point or points again or to develop more deeply some specific materials or concepts that were not clear enough for them. After completing each quiz, the instructor should review and project the aggregated performance of the class for each question, comment on the correct answer, and save the report automatically generated by Socrative. The total estimated in-class time devoted to quizzes and exit tickets should be 10-12 minutes. Before the next class, lecturers had to review the exit tickets to check students’ needs and their requests in the open question. When all the quizzes had been run, teachers had to merge the marks of each quiz with the students’ names. 4. Students’ feedback on the project. By the end of the course, we gathered students’ opinion through a survey to evaluate the contribution of Socrative to their learning process. http://dx.doi.org/10.4995/muse.2016.6375 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 178 4. Results In order to get information about students’ satisfaction with the experience, they were asked to answer questions related to the contribution and limitations of Socrative. This survey was answered through Google Forms. We use a Likert scale from 1 to 5, where 1 is strongly disagree, 2 somewhat agree, 3 neutral, 4 somewhat agree, and 5 strongly agree. There were 186 respondents. For several questions, a relatively high percentage of answers was concentrated in the central value, 3, but this is not unusual because central tendency is a common bias when using a Likert scale. Figure 1 shows that 47.31% of students strongly/somewhat agree that Socrative enhances comprehension of content and acquisition of competence related to Econometrics (20.43% strongly/somewhat disagree). Figure 2 shows that 56.99% of students agree that Socrative has motivated them to attend to class versus 23.12% who disagree. Figure 1. Socrative improves comprehension of contents and acquisition of competence related to the subject. Figure 2. Socrative has motivated me to attend class. It is notable that 73.12% of the students agree that Socrative is a motivation for listening to the teacher (Figure 3). Figure 4 shows that 55.91% of the students agree that Socrative has contributed to helping them memorize some basic concepts, versus 20.97% of the students who disagree. According to Figure 5, 65.05% of the students agree that Socrative 3,23% 17,20% 32,26% 38,71% 8,60% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 9,14% 13,98% 19,89% 32,26% 24,73% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 http://dx.doi.org/10.4995/muse.2016.6375 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 179 is helpful in testing their level of understanding of the contents explained by the teacher (13.44% disagree). Figure 6 shows that 50% of the students agree that Socrative has increased their self- confidence for answering teachers’ questions versus 17.74% who disagree. Figure 7 demonstrates that 44.62% of the students agree that Socrative increased their self- confidence in their learning process in contrast with 23.66% of the students, who disagree. Further, as represented in Figure 8, 56.99% of the students prefer Socrative to the traditional system of raising hands (23.66% do not). Figure 3. Socrative has motivated me to listen to the teacher. Figure 4. Socrative has helped me to memorize some basic concepts. 4,84% 4,84% 17,20% 45,70% 27,42% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 6,45% 14,52% 23,12% 34,95% 20,97% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 http://dx.doi.org/10.4995/muse.2016.6375 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 180 Figure 5. Socrative has contributed to testing my level of understanding. Figure 6. Socrative has increased my self-confidence in answering teachers’ questions. Figure 7. Socrative has increased my self-confidence in my learning process. Figure 8. Socrative is better than traditional questioning systems (e.g., raising hands, direct questions from the instructor). 3,23% 10,22% 21,51% 43,01% 22,04% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 8,06% 9,68% 32,26% 34,95% 15,05% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 7,53% 16,13% 31,72% 34,41% 10,22% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 10,75% 12,90% 19,35% 25,27% 31,72% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 http://dx.doi.org/10.4995/muse.2016.6375 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 181 Figure 9 makes clear the perceived contribution to feedback in comparison to other theoretical subjects, with 67.2% of the students agreeing with this item (only 9.14% disagree). Figure 10 shows that 61.29% of the students agree that Socrative facilitates interaction while only 15.05% of students disagree with this statement. Although we do not present a detailed graph here, 68.81% of the students strongly/somewhat agree that Socrative contributes to making classes more pleasant and fun (only 10.75% of the students disagree). Finally, Table 1 presents the results of questions about the importance of some of the inconveniences students could face with the use of Socrative. Our results show that the most relevant inconvenience was related to Internet connections and problems with the device. Our results suggest that using Socrative does not hinder class dynamics and does not prevent students from going back to the lesson and listening to the instructor. From the instructors’ viewpoint, the experience is also positively evaluated, and we are motivated by the results obtained in the students’ survey. Although it represented an extra effort that required teacher commitment and slight changes in the teaching methodology, the system is easy, fast, and convenient and can be used on a regular base. Figure 9. It gives more feedback to the teacher than in other theoretical subjects. Figure 10. It facilitates interaction. 4,30% 4,84% 23,66% 35,48% 31,72% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 6,99% 8,06% 23,66% 37,10% 24,19% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 http://dx.doi.org/10.4995/muse.2016.6375 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 182 However, we also found some drawbacks in the use of Socrative. First, time must be devoted to formulate questions that are relevant and appropriate for use in smartphones and other personal electronic devices. Second, when we linked data from the reports to students’ names, we found some errors (for example, incomplete identity numbers or students who had logged in with their first name). This may be explained because these students only occasionally attended class or they did not check their login information before submitting. For this reason, it is recommended that the system allow teachers to limit access to a list of users enrolled in the course so spelling errors in login information can be eliminated. This feature will be available in the upcoming Socrative Pro version, according to the information provided in the system developer website. We also felt that some more gamification and teamwork would be advisable because when using “space races” that allow team collaboration and competition when answering quizzes, students were more motivated than they were by ordinary quizzes. In this regard, other cloud-based response systems like Kahoot may be explored. Table 1. Rate the importance of the following drawbacks (%) 1 Strongly disagree 2 Somewhat disagree 3 Neutral 4 Somewhat agree 5 Strongly agree Problems with the device (i.e., no battery, system errors) 30.90 17.98 25.28 14.04 11.80 Problems with the Internet connection 17.20 24.19 25.27 16.67 16.67 After using Socrative during class it is difficult to go back and and listen to the teacher again ok 34.41 23.66 27.42 8.60 5.91 5. Conclusions The experience described in this paper mainly consisted of the regular use in lectures on Econometrics of a cloud-based response system, Socrative. The project was mainly aimed at engaging students and improving feedback during the lectures. Socrative offers lecturers the opportunity to quickly and easily enhance the delivery of their lectures or tutorials in a way that increases interaction with students, leading to a better learning experience. http://dx.doi.org/10.4995/muse.2016.6375 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 183 Socrative is free, easy to use, and our results show that it enhances student engagement and improves the feedback process from learner to teacher and vice versa. For example, 27.42% of the students strongly agree (and 45.7% somewhat agree) that using Socrative in lectures has motivated them to listen to the teacher. Further, 24.19% of the students strongly agree that using Socrative enables interaction (37.10% somewhat agree) and 56.98% agree that it has motivated them to attend lectures. Students also positively evaluate the feedback that the system provides them (enabling them to check their level of understanding) and the teacher (providing more feedback than in other subjects). All in all, our results show that the goals of the project have been reached. Therefore, student response systems, and specifically Socrative, are confirmed as an effective means of engaging students and enhancing the feedback process. Although some drawbacks were found such as time required for instructors to develop quizzes and review reports and exit tickets, as well as managing error in the login data, we strongly believe, basing on the survey of students and our own perceptions during lectures, that a student response system is a useful tool for improving our lectures. This is not to say that Socrative is the best option available on the market since there are many cloud-based free apps with distinctive features that instructors could also explore, such as Kahoot, that add more gamification to the learning process. Further research could explore how to increase the effects of response systems on knowledge acquisition and comprehension. http://dx.doi.org/10.4995/muse.2016.6375 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2016.6375 Social and Technological Sciences EISSN: 2341-2593 Ferrándiz et al. (2016) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 3 Nº 2 (2016): 173-184 | 184 References Aljaloud, A., Gromik, N., Billingsley, W. & Kwan, P. (2015) Research trends in student response systems: a literature review. International Journal of Learning Technology, 10 (4), 313–325. http://dx.doi.org/10.1504/IJLT.2015.074073 Beatty, I. (2004). Transforming student learning with classroom communication systems. EDUCAUSE Research. Retrieved June 2016 from: http://cds.cern.ch/record/877215/files/0508129.pdf INTECO, Instituto Nacional de Technologías de la Comunicación (2011). Estudio sobre hábitos seguros en el uso de smartphones por niños y adultos españoles. Retrieved May 2016 from: http://www.inteco.es/file/BbzXMkVkX8VG7-0ggHlozQ Kay, R. H., & LeSage, A. (2009a). Examining the benefits and challenges of using audience response systems: A review of the literature. Computers & Education, 53(3), 819-827. http://dx.doi.org/10.1016/j.compedu.2009.05.001 Kay, R. H., & LeSage, A. (2009b). A strategic assessment of audience response systems used in higher education. Australasian Journal of Educational Technology, 25(2), 235-249. http://dx.doi.org/10.14742/ajet.1152 Pew Research Center, Smith, A. (2016). Smartphone Ownership and Internet Usage Continues to Climb in Emerging Economies. Retrieved May 2016 from: http://www.pewglobal.org/2016/02/22/smartphone-ownership-and-internet-usage- continues-to-climb-in-emerging-economies/ Roblyer, M. D., & Wiencke, W. R. (2003). Design and use of a rubric to assess and encourage interactive qualities in distance courses. The American journal of distance education, 17(2), 77-98. http://dx.doi.org/10.1207/S15389286AJDE1702_2 Terrion, J. L., & Aceti, V. (2012). Perceptions of the effects of clicker technology on student learning and engagement: A study of freshmen chemistry students. Research in Learning Technology, 20:16150 http://dx.doi.org/10.3402/rlt.v20i0.16150 Yusof, S., Goolamally, N., Latif, L.A. & Fadzil, M. (2012). 12th International Conference on information, Kuala Lumpur (MALAYSIA). http://dx.doi.org/10.4995/muse.2016.6375 http://dx.doi.org/10.1504/IJLT.2015.074073 http://cds.cern.ch/record/877215/files/0508129.pdf http://www.inteco.es/file/BbzXMkVkX8VG7-0ggHlozQ http://dx.doi.org/10.1016/j.compedu.2009.05.001 http://dx.doi.org/10.14742/ajet.1152 http://www.pewglobal.org/2016/02/22/smartphone-ownership-and-internet-usage-continues-to-climb-in-emerging-economies/ http://www.pewglobal.org/2016/02/22/smartphone-ownership-and-internet-usage-continues-to-climb-in-emerging-economies/ http://dx.doi.org/10.1207/S15389286AJDE1702_2 http://dx.doi.org/10.3402/rlt.v20i0.16150