Language Value http://www.e-revistes.uji.es/languagevalue July 2017, Volume 9, Number 1 pp. 132-159 ISSN 1989-7103 Articles are copyrighted by their respective authors DOI: http://dx.doi.org/10.6035/LanguageV.2017.9.6 132 Up2B2: Playing English grammar games at the B2 level Irina Argüelles Álvarez irina.arguelles@upm.es Margarita Martínez Núñez margarita.martinez@upm.es Ana Belén García Hernando anabelen.garcia@upm.es Antonio da Silva Fariña antonio.dasilva@upm.es Universidad Politécnica de Madrid, Spain ABSTRACT The study presented herein is framed within the Mobile Assisted Language Learning (MALL) approach and presents the results from the use of an original mobile app, developed by the research team, on the part of 73 students. The app prototype consists of multiple choice questions and answers that assess varied aspects of the grammar, vocabulary and use of English at level B2 according to the Common European Framework of Reference for Languages (CEFRL). The application presents motivating gamification components that aim at enhancing students’ participation and regular use of the game. Some of these gamification features are response time, use of competition and training tests, and a score ranking based on nicknames. The study uses different indexes in order to describe the use of the platform and qualitative and quantitative indicators to reach positive conclusions related to students’ increased motivation and improvement in their grammar and vocabulary competence levels. Keywords: Applications in subject areas; architectures for educational technology system; evaluation of CAL systems; post-secondary education; mobile assisted language learning I. INTRODUCTION Universities and other tertiary level centres or institutions are today experiencing a global revolution for which internationalization is a priority if they are to survive. Bilingual programmes across Europe, degrees taught in English as a Medium of Instruction (EMI programmes) or Erasmus+ student and teacher mobility imply the need to demonstrate an adequate level of proficiency in English which most universities across Europe have established as B2, according to the Common European Framework of Reference for Languages (CEFRL). A B2 level has been established as the minimum standard for engineering students at the Universidad Politécnica de Madrid (UPM) to obtain their degree, although placement tests carried out across the university for the last seven years place most of them at an A2/B1 level. This means, for many students, two levels below the required one. To deal with this and other similar situations, some official programmes at different European universities have incorporated subjects that mailto:irina.arguelles@upm.es mailto:margarita.martinez@upm.es mailto:anabelen.garcia@upm.es mailto:antonio.dasilva@upm.es Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 133 aim to prepare the students to obtain the required B2 certification. Others have opted for an extracurricular offer with or without the participation of the language centres in the same universities when such language centres exist. In the last few years, the UPM has offered both optional subjects in its degrees as well as free-elective European credits through extracurricular training, in order to prepare students preferably for the “Test of English for International Communication” (TOEIC), but more generally for most external certifications included in the list provided by the CRUE (Conferencia de Rectores de Universidades Españolas – Spanish Universities Rector Conference) as official certifications (CRUE Acreditación en idiomas) http://www.acreditacion.crue.org/. Furthermore, several innovation projects which address the learning of English i have been developed by groups under innovation university internal calls for internationalization. Our research was developed under one of these innovation projects. A game, called Up2B2, was presented to enhance and motivate the practice of English grammar and vocabulary at a B2 level on the part of the students (Argüelles Álvarez et al. 2015). The project was framed within Mobile Assisted Language Learning (MALL) (Viberg and Grönlund 2012, Chinnery 2006), which is increasingly associated with mobile gaming, as Kukulska-Hulme (2009: 159) puts it, the basic idea being that students use their mobile devices to learn languages through games. The concept of gamification (Kapp 2012, Werbach 2014) has therefore been directly associated with MALL. More specifically, the aim of our research was to design and test a multiple-choice question and answer game to review different aspects of grammar, vocabulary and use of language at the B2 proficiency level. The game was considered a complement to other possible regular or extracurricular English courses that the students at the Escuela Técnica Superior de Ingeniería y Sistemas de Telecomunicación could be attending. It was offered to 73 students who were at either higher A2 or B1 English level. All these students had been admitted and were enrolled in the groups preparing for the B2 certification exam at the UPM. Initially, the aim was twofold. On the one hand, it intended to study the degree of improvement, if any, in the proficiency level of those students who used the app regularly as compared with those students who did not. On the other hand, the aim was to reach conclusions regarding motivational aspects which Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 134 could derive from the use of the gamification features in the app to learn vocabulary and grammar. Several studies point to the contribution of the MALL approach to learning. Viberg and Grönlund (2012) revise research carried out in the field of MALL from 2007 to 2012 and conclude that MALL enhances the learning of foreign languages. Burston (2013) presents annotated bibliography of implementation studies in MALL, many of them comparing groups in pre- and post-tests with positive results in listening skills, vocabulary learning, grammar knowledge, reading comprehension and writing skills (spelling, grammar, punctuation, editing or re-drafting) apart from other more general gains in motivation to learn, e.g. reflection on language usage or use of strategies and positive interaction. Rico et al. (2015) also highlight the benefits of MALL and game- based learning. Motivation has been found to be vital to increase students’ participation and success (Milligan et al. 2013). Individual motivation is related to the reasons to do something (Ryan and Deci 2000). An individual is intrinsically motivated to carry out an activity for the mere satisfaction inherent in the activity, whereas they will be extrinsically motivated by the impulse to complete an activity to get a desired result. Gamification combines these two types of motivation: it provides extrinsic rewards such as levels, points or badges, which improve engagement, in addition to raising feelings of mastery achievement, autonomy and sense of belonging (Muntean 2011). Several studies have demonstrated the added value of motivation to the learning process when incorporating typical features of games (Osma Ruiz et al. 2015a, 2015b). Others have shown that students prefer to use games as a means to learn grammar or vocabulary not only because they find it more enjoyable and challenging, but also because they find the learning more fruitful and long-lasting (Lui 2014). Furthermore, benefits of learning languages through the use of apps that integrate gamification features are reported in Sauerland et al. (2015) or Figueroa (2015). Mosavi-Miangah and Nezarat (2012) add to these conclusions and the well-known concept of “anytime/anywhere” (ubiquitous learning or u-learning in Jung 2014) other advantages and also disadvantages of mobile language learning. The pros include a better use of the students’ free time, while some of the cons would be reading Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 135 difficulties on small screens, problems to complete specific tasks mostly due to an inappropriate initial design – activities that take too long to complete on the mobile devices and/or lack or cost of Internet access (Stockwell 2008) – or, most frequently, issues related to usability and accessibility (Jordano de la Torre et al. 2013 cited in Pareja et al. 2016). The study of the aspects specifically related to the use of a mobile app to learn are therefore also of great interest. This is the reason why, in our study, we added a third aim to the two already mentioned: to analyse parameters that characterize some of the user patterns, so as to reach some preliminary conclusions on ubiquitous learning. II. CONTEXT The app prototype, Up2B2, was developed by the Multidisciplinary Educational Innovation Group (GIEM) ii in the academic year 2014-2015 (Argüelles Álvarez et al. 2015). The repository of questions and answers comes from former pen and paper proficiency exams that had been carried out for the previous three academic years (six semesters) across the University and had afterwards been validated by members of the group. Validity and reliability issues were addressed in (Argüelles Álvarez 2013) and (Argüelles Álvarez and Martínez Núñez 2015), based on more than 2,000 answers to each of the questions and answers that made up the final bank of 500 questions and answers for the app. As described in the introduction section, Up2B2 presents a gamification component which seeks to motivate students using it. Our hypothesis is that the competitiveness features of the activities will favour the regular use of the game and the students’ interest in the app (Ryu 2013, Sykes and Reinhardt 2013, Osma Ruiz et al. 2015a, 2015b). As mentioned earlier, the aim of our research is threefold: firstly, different indexes have been defined for monitoring the platform and to reach conclusions regarding the extent to which a MALL methodology influences an improvement in proficiency, if any. Secondly, we aim to study the motivational factors related to the use of MALL together with the inclusion of several gamification features in the learning process. Lastly, ubiquitous learning is preliminary studied based on some patterns of Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 136 use, as these are important components in any mobile-based non-formal learning process. Based on previous research on motivation by Stockwell (2013) or Ushioda (2013), questionnaires were purposefully designed (Appendix I) to look for students’ perceptions and perspectives in relation to mobile language learning. The results of these questionnaires were analysed qualitatively by a frequency Likert scale and correlation spearman’s coefficient. These questionnaires were prepared and administered together with the post-test that 73 students completed after a period of four months’ training preceded by a pre-test. The pre/post-tests, as noted above, aimed to assess improvements in students’ proficiency levels, and gave rise to the quantitative results that will be presented later. III. DESCRIPTION OF THE APP III.1. Application requirements and design The system that realizes the Up2B2 application has been designed to meet a set of requirements. In the following we summarize these requirements together with the rationale behind them:  The game has to be playable by using the widest possible range of mobile/non- mobile devices (client devices in what follows), including smartphones (Android, iOS or others), tablets and personal computers. Since students are going to use their own equipment to play the game, this requirement makes it easier for more students to enter the system no matter what specific device they have.  The players are included in a ranking based on their score. The ranking is made public but each player is shown by their nickname instead of their real name. This requisite defines the Up2B2 gamification features the effects of which are evaluated in this work.  The database of multiple-choice questions together with their associated meta- information (correct answer, category, level of difficulty) is maintained on a server. The client device should not store the questions but download them online just at the Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 137 time when they are needed to play. This enables the questions to be updated without having to update all the locally installed apps and it also saves memory space on the client devices.  Stemming from the previous requisite, the client device needs to have an operative Internet access whenever the player wants to do a test. Any additional technical requisite on the client device should be kept at a minimum, for the same reasons set out in the first point above.  The server also has to store the parameters that define the main characteristics of the game: the ranking of the different players, the maximum time to perform each test, whether there is a maximum time of inactivity after which the user loses points, etc. Again, this allows the game parameters to be updated without having to force all the users to update the app on their devices.  The server should provide the administrator with the possibility of updating both the questions in the database and the configuration of the game parameters so that all the changes take place at the same time. Given the requirements presented, we chose web technologies to develop Up2B2 and it was implemented as a multiplatform WebAPP. The client-side, which implements the user interface, has as the sole requisite a web browser capable of executing JavaScript code and applying CSS (Cascading Style Sheets) rules. The vast majority of modern devices fulfil these basic requirements. Moreover, we produced native versions of the app for Android and iOS users, although the web-based access is always available on any platform. Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 138 Figure 1. Main technologies used by the Up2B2 app. The application was designed to be adaptive, suitable for different screen sizes and services. Visually attractive CSS components were used, which greatly resemble those on the most popular platforms and contribute to a distinctive look-and-feel aspect, very similar to native applications. The server-side, i.e. the back end of the application, is deployed on a specific server accessible via the Internet and provides all the necessary management of users and application parameters and hosts the questions database. Figure 1 shows a schematic representation of the main elements involved in Up2B2 and their associated technologies. We would like to refer the reader interested in more technical details to (Argüelles Álvarez et al. 2015). Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 139 III.2. Introduction of gamification features: competition tests As mentioned in the introduction to the study, the aim of the gamification features included is to increase the students’ motivation in the use of the application, thus favouring their training in the language aspects that can be exercised through it. We designed a system by which users obtain points after having completed competition tests. These points serve to order the users on the participants’ global ranking so that everyone can check their position on it. The number of points that each user gets results from their language competence, on the one hand, and the speed when answering the competition tests, on the other. Furthermore, a hysteresis component has been introduced to consider not only the last test completed but also the previous ones, with a weight that decreases exponentially. This means that whenever the user completes a new competition test, the score obtained is weighted equally (50% each) with their previous score, in order to obtain the updated number of points of the player. Every player starts the game with 0 points. The number of points obtained in a specific competition test is calculated as shown in Table 1. Table 1. Operations made to obtain the score in a competition test. P_test = P_competence + P_speed. P_competence = 9 x (num. of correct answers) – 4 x (num. of mistakes). P_speed = +10 (if the test is completed in less than half the maximum time established) or -10 (in the opposite case). From the way the score is calculated and bearing in mind that every competition test consists of ten questions, it can be deduced that the number of points of each student is always between -50 and +100. The weighting derived from the hysteresis favours that the “new stars”, who improve fast from lower scores, get more points in absolute value than those who have a longer history of excellent results. This aspect is not by chance, as there are in fact many seasonal competition sports (e.g. tennis) that establish, in a similar way, a ranking mechanism that favours those players who are situated in a lower position and improve significantly. This scoring procedure increases competitiveness and forces the participants who are situated in higher positions to maintain a sustained effort in order to defend their position in the ranking. Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 140 The platform also provides the option to subtract points if the user spends a given amount of time without doing any competition test (e.g. 48 hours) and so regularity in the use of the game is promoted through the game itself. Nevertheless, for the current academic year we have considered it more suitable not to activate this option to control a possible negative effect in the case of those users who experience loss of points in the ranking due to non-regular use. The users were introduced into the system by their real names and email addresses. They were provided with an initial random password but they could (and were recommended to) change both their username (their nickname) and password the first time they entered the application. Therefore, the user ranking shows nicknames instead of real names, as can be seen on the right-hand side of Figure 2. Figure 2. Feedback given to students through training tests (left and centre) and players ranking associated to competition tests (right). III.3. Feedback and specific categories review: training tests It should not be forgotten that the main aim of the Up2B2 application is to offer further training in the English language at a B2 level. The users can present different needs to reinforce specific aspects of the language (e.g. word order), and they could find it useful to receive feedback regarding the categories where they perform better or worse according to the historical record maintained in the server. Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 141 Pursuing this aim and in addition to the competition tests, the user can also select training tests. The goal of these tests is to prepare the students before the actual use of the competition ones. The results on training tests do not modify the score obtained by the user in competition as the objective of the training tests is only of a formative nature. This modality allows the user to select the category he/she wants to train (Grammar, Vocabulary, Word order or Verbs) or to opt for a random mixture of categories in a way similar to what they are expected to do in competition tests (Assorted category). The training tests have fewer time restrictions than the competition ones; besides, for every question answered, the results are clearly reported, and in the case of not having succeeded, the correct option is highlighted (see Figure 2, left-hand side and centre). IV. RESEARCH METHODOLOGY IV.I Methodology followed for the study From the research team’s experience in previous innovation projects, students usually report that they feel overloaded when asked to collaborate in innovation research. The main reason they give is that they have to deal with too many resources in their regular courses and researchers ask them to use a new one, an app in our case. Thus, they often ask for structured environments, clear instructions and a high degree of orientation when it comes to incorporating new tools into their learning. Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 142 Up2B2 v0: First prototype of App and learning contents Results of pilot test: inputs for the second phase of App and materials Up2B2: Final App and materials ready for final evaluation Final results and conclusions Testing of first prototype in pilot environment Second phase of: • App.enhancement. • Learning mat. completion. First phase of: • App. design & implem. • Learning materials. Final evaluation Start: Figure 3. Methodology followed for Up2B2. The design, development, validation and test methodology that we have followed for the Up2B2 system follows a spiral model (see Figure 3). Two phases are considered to be necessary for the design and implementation in order to better adapt the app to the needs of real users. In the first phase (Argüelles Álvarez et al. 2015) the prototype was tried out by a small group of students in a closed and controlled environment to analyse its usability, since this is one of the aspects considered important from the initial design, as highlighted in previous research (Ali et al. 2015, Jordano de la Torre et al. 2013 cited in Pareja et al. 2016). This first phase allowed us to develop the second version of the app, as well as to improve the server database with additional questions, up to around 500, as stated earlier. It also made us aware of the importance of clearly informing the users on the specificities of the app in order to encourage its usage before the evaluation. Thus, we followed an ordered list of stages for the second phase of Up2B2, paying special attention to the information given to the students, not only before the evaluation period but also during it. Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 143 The study was carried out during the second semester (spring semester) of the academic year 2015-2016. Firstly, in-class information was given to the students at the beginning of the academic semester. Either one or two teachers from the research team gave a brief talk to the groups of students that were offered the possibility of using Up2B2. In this way, students learnt about the basics of the app and they knew beforehand that they would receive an automatic mail with the necessary information to start the process. All potential users had previously been given access to the aforementioned Moodle site, and the automatic email was sent to invite them into this platform. Once they had logged in, students could find the instructions to officially enrol in the Up2B2 activity. Those students who decided to participate had to fill in a short questionnaire that was useful for characterizing the sample (demographical data, studies-related data and information on their previous experience in using apps to learn English). Secondly, they had to do a placement test (i.e. the pre-test) to estimate their initial level of English. We used the Oxford Quick Placement Test v2 for this purpose. Once they had finished, they received a second e-mail, this time personalized, which provided each student with their initial username and password to access the Up2B2 app. Students could contact the Up2B2 team throughout the competition period by using a specific email address in case they had any doubts or problems. After this period had expired, participants were again asked to log into the Moodle site in order to repeat the placement test (i.e. the post-test) and to fill in a final survey. The results of this final survey would be analysed by the research team to measure their satisfaction while studying English with the app. IV.II Characterization of the sample and statistical analysis The number of students selected for the study was 73. The group was randomly chosen among the individuals who were enrolled in the subject Introduction to English for Professional and Academic Communication I and II, two B2 level preparatory subjects that are taught in different engineering degrees at the UPM. The sample was split into two: the experimental group (63 individuals) and the control group (10 individuals). Since most students wanted to use the app, the researchers opted for a small control group (Fidalgo-Blanco et al. 2016). Nevertheless, this control group would have enough Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 144 subjects to ensure that both the experimental and the control groups were statistically homogeneous according to the Levene statistic, as shown in Table 2. Table 2. Characterization of the sample. CHARACTERIZATION OF ENGLISH LEVEL AND STUDY HABITS GROUP % (nº) TOTAL % (nº) INITIAL LEVEL CONTROL EXPERIMENTAL A2 30% (3) 26.98% (17) 28.76% (21) B1 50% (5) 55.55% (35) 54.80% (40) B2 20% (2) 17.46% (11) 16.44% (12) STUDY TIME 1 : Less than 1 hour 30% (3) 39.68% (25) 38.35% (28) 2 : Between 1 and 3 hours 50% (5) 36.50% (23) 38.35% (28) 3 : Between 3 and 5 hours 20% (7) 19.05% (12) 19.18% (14) 4 : More than 5 hours 0% (0) 4.76% (3) 4.12% (3) TECHNOLOGY ADOPTION Uses smartphone apps to learn English 20% (2) 33.33% (21) 31.51% (23) Uses web apps to learn English (not mobile devices) 20% (2) 36.51% (23) 34.25% (25) INDIVIDUALS CHARACTERIZATION Male 70% (7) 73.015% (46) 72.60% (53) Female 30% (3) 26.99% (17) 27.40% (20) Between 20 and 26 years old 100% (10) 100% (63) 100% (73) Studies: engineering degree 100% (10) 100% (63) 100% (73) The statistical analysis reported in this paper was carried out using IBM’s SPSS software version 21. Data follow normality, based on the Shapiro-Wilk test. From the data presented in Table 2, and the Levene statistic for variance homogeneity, in the initial analysis (Levene C = 0.389 and level p = 0.679), we could state that both groups of items are equivalent for the study, i.e. they are statistically homogeneous regarding the characterization of the sample. Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 145 V. RESULTS AND DISCUSSION V.1. Students’ marks The study of the improvement in academic results in the course was conducted with the previously defined groups. The group that used the application had a mean score of 38.49 (SD = 7.92) in the post-test, while the control group had a mean score of 33.90 (SD = 6.707). In this first analysis we studied the improvement in learning attained by using the app Up2B2. We carried out a 2 (group type) x 1 (examination) analysis of covariance (ANCOVA). The student’s initial level grades were controlled by a pre-test. We considered the exams as intra-subject variables and the group type as the variable between the subjects. A significant effect between both variables was found: for tests, F = 103.90, p<0.001, η 2 = 0.597, for group type (i.e. control vs. experimental), F = 5.885, p < 0.018, η 2 = 0.78, and for interaction F = 6.256, p <0.015, η 2 = 0.082. After having used the application for the competition period, the two groups exhibited statistically significant differences in the test results, the marks obtained by the experimental group being higher than those of the control group. The app performance was measured through the level of improvement in the learning results in the post-test. This improvement has been also corroborated by different studies on English language learning with mobile devices (Saran et al. 2012, Chen and Hsu 2008, Cavus and Ibrahim 2009), all of them based on instructional learning. Positive knowledge gains are mainly explained by more frequent practice and repetition (Saran et al. 2012, Thornton and Houser 2005). Our study, even within the same context of instructional learning, incorporates gamification features that were not present in the aforementioned articles. The incorporation of gamification features, as stated earlier, brings with it a more interactive learning experience and increased motivation. V.2. Students’ motivation For the analysis of students’ motivation and expectations, the students’ responses were examined after the application use period had expired by applying a standardized evaluation instrument (Jung 2014). We used a Likert scale (1-5), the analysis being http://www.sciencedirect.com/science/article/pii/S0747563216303843#bib82 Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 146 focused on students’ responses to questions concerning the following points: 1) to what extent the characteristics of mobile learning (ubiquity, innovation, ease of use, usefulness and contents) are elements that motivate learning, and 2) how much they perceive that they have learned and how high their expectations are to continue learning. The total number of respondents was 63. From the 9 elements analysed and shown in Table 3 below, a Cronbach alpha of 0.836 was obtained. When analysing the results we must highlight that students do not consider the app as the only instrument to improve their level of English (Q7) with a mean of 2.94 (SD = 1.01). However, the use of an app for learning “is more fun” (Q5) with an average of 3.97 (SD = 0.86) and it is a challenge they welcome (Q3), with a mean of 3.73 (SD = 0.87). Table 3. Percentages of participant responses regarding their attitudes towards motivation and expectations with Up2B2. Frequency (%) Mean SD Strongly disagree (1) Disagree (2) Neither agree nor disagree (3) Agree (4) Strongly agree (5) (Q1) Using different working modalities whenever I feel like it has helped me design my own learning strategy. 1.59 20.63 28.57 33.33 15.87 3.41 1.04 (Q2) I like using new and different methods to study English. 0.00 3.17 12.70 41.27 42.86 4.24 0.80 (Q3) Learning English using an app is an interesting challenge. 0.00 7.94 30.16 42.86 19.05 3.73 0.87 (Q4) Using an app to learn English makes me invest more time in my learning. 0.00 7.94 30.16 42.86 19.05 3.65 0.94 (Q5) Using an app to study English makes it more fun. 1.59 4.76 14.29 53.97 25.40 3.97 0.86 (Q6) Using an app to study English makes me feel good. 4.76 9.52 26.98 38.10 20.63 3.60 1.07 (Q7) I can improve my competence in English up 9.52 23.81 31.75 33.33 1.59 2.94 1.01 Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 147 to the level I need only with an app. (Q8) The combination of my previous knowledge in English and the information I get through an app can help me improve my marks in English. 0.00 9.52 31.75 50.79 7.94 3.57 0.78 (Q9) After this experience, I will use other mobile apps to study English. 4.76 7.94 22.22 50.79 14.29 3.62 0.99 Almost 85% of the students who used the application demand new learning methodologies and agree or strongly agree that they like to use new and different methods to study English (Q2). However, less than 50% identify these technologies as a tool to design their own learning strategy (Q1). These results indicate that, although students consider this app as a novel and innovative learning tool, they are not willing to incorporate this tool into their personal learning strategy in the long term. 65% of the students plan to continue using mobile applications to study English (Q9). In addition, approximately 59% agree that this app helps to improve their competence in English (Q8). Motivational factors are linked to gamification as the basis to understand motivation to play. Five motivational adaptive factors to use the application have been selected from the questions in the survey based on Personal Investment Theory (PIT) (Shilling and Hayashi 2001, McNamara et al. 2009) (see Table 4). Table 4: Selected motivational adaptive factors. IMMEDIACY (Q10) The app has allowed me to obtain information to learn English quite immediately. INNOVATION (Q11) I am a person who is in the habit of trying new things before others. USEFULNESS (Q12) I consider this application useful to learn English. EASINESS (Q13) Learning English has been easier with this app. INTERACTIVITY/ CONNECTIVITY (Q14) The app let me interact with other students. The correlation coefficients of these motivational factors are shown in Table 5. Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 148 Table 5. Correlation Spearman’s coefficient. Innovation Usefulness Easiness Interactivity/ Connectivity Immediacy Correlation coef. (bilateral significance) -.096 (p = .455) .390 ** (p = .002) .468 ** (p = .000) .332 ** (p = .008) Innovation Correlation coef. (bilateral significance) .069 (p = .592) .074 (p = .563) .272 * (p = .031) Usefulness Correlation coef. (bilateral significance) .795 ** (p = .000) .420 ** (p = .001) Easiness Correlation coef. (bilateral significance) .424 ** (p = .001) ** Correlation is significant at the 0.01 level (bilateral) * Correlation is significant at the 0.05 level (bilateral) Whereas innovation factors do not seem to be important as a motivational factor to use the application, connectivity and interaction of the application influences the rest of the motivational factors (immediacy, innovation, usefulness, easiness). As connectivity and interaction motivational factors are provided by the gamification elements, we can conclude that incorporating gamification elements increases students’ motivation to study. The positive feedback they get pushes students forward and they become more interested and stimulated to learn (Muntean 2011). Gamification has been frequently used to provide incentives seeking to modify specific negative behaviours such as discouraging interruptions or distraction. Previous studies provide convincing arguments for what is viewed as the core of mobile learning: the facilitation of learning across different contexts, as defined by Sharples et al. (2007) or Pachler et al. (2010). V.3. Students’ usage profile Learning outside the classroom places the responsibility for time management on the students themselves. Ubiquity provided by mobile devices is especially useful for non- formal learning because it is integrated into the users’ real life. Undertaking learning activities, for example, while in transit, is only made possible by using mobile devices. Generally speaking, there are two ways of using mobile devices to support mobile language learning: as a regular, habitual pattern of activity or in a spontaneous, unplanned way (Kukulska-Hulme et al. 2012). Regular patterns revolve mainly around opportunities in daily routines, such as at breakfast, lunch times or last thing before http://www.sciencedirect.com/science/article/pii/S0747563216303843#bib75 http://www.sciencedirect.com/science/article/pii/S0747563216303843#bib60 Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 149 going to bed, whereas spontaneous learning is determined by sudden available time and student’s mood for learning. In this research, ubiquitous learning and use patterns are preliminary studied from the histogram of times at which students use the app in a day or during the week as well as the time spent playing with the app per day. The actual usage pattern is valuable information in order to make future improvements to the app usability and, consequently, to improve the students’ learning experience. For example, implementing specific feedback for students who follow a planned app usage schema, e.g. every night before going to bed, will be easier. In what follows, results are discussed in terms of students’ activity profile. Figure 4 shows the tendency in the times at which the app is used and supports the idea of a greater degree of freedom to use personal time, thus personalizing the way and the context in which the student actually studies. This is a clear example of anytime/anywhere learning patterns that MALL technologies make easier (Saleh and Bhat 2015). Figure 4. Daily usage profile. Figure 4 shows the histogram of times at which students accessed the Up2B2 system throughout the day. The server logs revealed that there are high activity levels around midnight, from 11:30 pm to 2:00 am, which falls drastically after this time. This can be considered a good use of time available after having dinner and before going to bed. Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 150 Activity resumes again at 6:00 am and starts increasing from 8:00 am, which can be considered breakfast time. Another moment that can be related to common daily activities is seen at 13:30, lunch time in Spain, when there is a peak of activity. Students might be using the app while waiting in the cafeteria self-service queue. Regarding Saturday and Sunday, many people try to plan time at the weekend for homework or extra activities and, therefore, these could be considered regular planned activities. However, in our study, as shown in Figure 5, most of the students prefer not to carry out learning activities with the app at the weekend. Even Mondays exhibit low figures, only around 10% of test activity. The bulk of activity occurs in the middle of the week, with a high concentration of activity around Tuesday, Wednesday and Thursday. Figure 5. Weekly usage profile. Finally, it is important to know how long the students are engaged doing tests. As shown in Figure 6 most of the days (35%) students do just one test. 55% of the days students did between two and 10 tests per day. There are a residual number of situations (10%) where students do more than 10 tests per day. Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 151 Figure 6. Students tests per day histogram. VI. CONCLUSIONS AND FUTURE WORK In this article we have presented the fundamental characteristics of an application consisting of a question-answer competition game for the learning of English language at a B2 level. The application has been oriented to its use from mobile terminals in accordance with the MALL approaches, and has been designed integrating a strong gamification component which aims at motivating its users to use it regularly. In addition, as an innovative contribution, the application has been fed with a wide battery of questions and answers, previously validated by their use in official examinations for the accreditation of the B2 level in English at the UPM. The app has been evaluated through an ANCOVA analysis by measuring the students’ improvement in their results in an English test. Two groups of statistically homogeneous students were selected, i.e. the experimental and the control groups. After the competition period with the app, the experimental group obtained higher marks in the post-test than the control group. Regarding motivation, the accomplishment of competition tests and the subsequent positioning on a mastery ranking has proved to be a challenging practice. On the other hand, training tests allow the students to exercise specific language skills. Students’ motivation and expectations have been studied by means of a survey, in which a standardized evaluation instrument based on a Likert scale has been used. The use of the app has been found to be positive for the objective of learning English, even though it is not considered as the only instrument neither is it seen as a tool to be included in a Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 152 personal learning strategy in the long term. Five motivational factors have been evaluated in the survey and the correlation coefficients among them have been extracted. From the results, we have found that connectivity and interactivity are the main motivational factors that influence the rest. These two factors are directly related to gamification features, and we can conclude that gamification has played an important role in fostering motivation to study English. Study habits are influenced by the “anytime / anywhere” possibilities that mobile apps present. In our analysis we have found that students tend to make a good use of, otherwise idle, periods (just before going to bed or waiting in the cafeteria queue), although weekends are not their preferred time to study. Regarding frequency of use, only 10% of the students do more than 10 tests per day, 1 or 2 tests per day being the most usual frequency. Up2B2 offers university students interested in improving their competence in English anytime and anywhere a convenient way to study, provided they have access to a smartphone. Moreover, working with a technology that they use on a daily basis and with which they feel confident, favours acceptance and learning becomes easier. From the results obtained, MALL should be considered as another resource alongside the others in the educational toolbox. The relationship between personal differences or learning styles and proficiency in English is an area worth investigating. The results derived from this analysis could be very useful for improving the app in order for it to better adapt to the students’ needs and preferences. As a secondary future objective, we will aim at receiving feedback on from the students using the app regarding aspects more related with the technology and its usability. These are interesting from the point of view of the app functionality and the users’ experience in order to improve the prototype in further stages of the research. The evaluation of this experience and the students’ sensitivity to the app from a more technical point of view will also allow us to reach conclusions regarding possible individual differences among users. Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 153 Acknowledgements The Project ILLLAB: Mobile Application for Language Learning "Up2B2" (IE1415-59002) has been funded by the UPM annual call for Teaching Innovation Projects 2014-2015. Notes i Among these projects we highlight the following: ILLLAB: Mobile Application for Language Learning "Up2B2" (course 2014/2015, UPM code: IE1415-59002), and TechEnglish (course 2014/2015, UPM code: PT1415-01000). ii Escuela Técnica Superior de Ingeniería y Sistemas de Telecomunicación (School of Telecommunications and System Engineering) at the UPM. REFERENCES Ali, A., Alrasheedi, M., Ouda, A.H. and Capretz, L.F. 2015. 2A study of the interface usability issues of mobile learning applications for smart phones from the users perspective”. International Journal on Integrating Technology in Education (IJITE) 3(4), December 2014. Argüelles Álvarez, I. 2013. “Large-scale assessment of language proficiency: Theoretical and pedagogical reflections on the use of multiple-choice tests”. International Journal of English Studies, 13(2), 21-38. Argüelles Álvarez, I. and Martínez Núñez, M. 2015. “Identifying learning patterns in the upper-intermediate level of English through large-scale testing”. Procedia - Social and Behavioral Sciences, 173(13) February, 5-10. Argüelles Álvarez, I., M. Martínez-Núñez, A. B. García Hernando, A. Da Silva Fariña, M. L. Mouronte López and M. González Martín. 2015. “Aprendizaje Asistido por Dispositivos Móviles: UP2B2 Mobile Assited Language Learning: UP2B2”. Proceedings III Congreso Internacional sobre Aprendizaje, Innovación y Competitividad (CINAIC), Madrid, 14-16 October. Burston, J. 2013. “Mobile-assisted language learning: a selected annotated bibliography of implementation studies 1994–2012”. Language Learning & Technology, 17(3), 157–225. Cavus, N. and Ibrahim, D. 2009. “m‐Learning: An experiment in using SMS to support learning new English language words”. British Journal of Educational Technology, 40(1), 78-91. Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 154 Chen, C.M. and Hsu S.H. 2008. “Personalized intelligent mobile learning system for supporting effective English learning”. Educational Technology and Society, 11, 153–180 Chinnery, G.M. 2006. “Emerging Technologies Going to the MALL: Mobile Assisted Language Learning”. Language Learning & Technology, 10(1), 9-16. Conferencia de Rectores de Universidades Españolas (CRUE). Acreditación en idiomas http://www.acreditacion.crue.org/. Fidalgo-Blanco, A., M. Martínez-Nuñez, O. Borrás-Gené, and J.J. Sánchez- Medina. 2017. “Micro flip teaching‒An innovative model to promote the active involvement of students”. Computers in Human Behavior, 72, 713-723. Figueroa, J. 2015 “Using gamification to enhance second language learning”. Digital Education Review, 27, http://greav.ub.edu/der/ Jordano de la Torre, M., Pareja-Lora, A., Read, T. and Rodrigo San Juan, C. 2013. “The design of accessible mobile and ontology-based applications for ubiquitous foreign language learning”. In Proceedings of the UNED – ICDE International Conference, 1-14, Madrid, Spain. Jung, H. 2014. “Ubiquitous learning: determinants impacting learners’ satisfaction and performance with smartphones”. Language Learning & Technology, 18(3), 97– 119. Kapp, K.M. 2012. The Gamification of Learning and Instruction: Game-Based Methods and Strategies for Training and Education. New Yersey, Wiley. Kukulska-Hulme, A. 2009. “Will mobile learning change language learning?” ReCALL, 21(2), 157–165. Kukulska-Hulme, A. 2012. “Language learning defined by time and place: A framework for next generation designs”. In: Díaz-Vera, Javier E. (ed.) Left to My Own Devices: Learner Autonomy and Mobile Assisted Language Learning. Innovation and Leadership in English Language Teaching, 6. Bingley, UK: Emerald Group Publishing Limited, 1–13. Limayem, M. and Cheung, C.M.K. 2008. “Understanding information systems continuance: the case of Internet-based learning technologies”. Information & Management, 45(4), 227-232. http://greav.ub.edu/der/ Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 155 Lui, S. 2014. “Use of gamification in vocabulary learning: A case study in Macau”. 4th CELC Symposium Proceedings. McNamara, D.S., Jackson, G.T. and Graesser, A.C. 2009. “Intelligent tutoring and games (ITaG). Proceedings of the Workshop on Intelligent Educational Games at the 14th Annual Conference on Artificial Intelligence in Education”, Brighton, UK, 1-10. Milligan, C., Littlejohn, A. and Margaryan, A. 2013. “Patterns of engagement in connectivist MOOC”. Journal of Online Learning and Teaching, 9(2), 149-159. Mosavi Miangah, T. and Nezarat, A. 2012. “Mobile-assisted language learning”. International Journal of Distributed and Parallel Systems (IJDPS)3(1), 309-319. Muntean, C.I. 2011. “Raising engagement in e-learning through gamification”. 6th International Conference on Virtual Learning ICVL, 323-329. Nov, O., Naaman M. and Ye, C. 2010. “Analysis of participation in an online photo sharing community: a multi-dimension perspective”. Journal of the American Society for Information Science and Technology, 61(3), 555-566. Osma-Ruiz, V.J., Argüelles-Álvarez, I., Sáenz-Lechon, N., Gutiérrez-Arriola, J.M., Fraile, R., Villar-Miguélez, C. and Guerrero-Vaquerizo, I. 2015. “Past and future of gamification in the learning of English as a foreign language”. Proceedings of the 9th International Technology, Education and Development Conference (INTED) Madrid, Spain. 2-4 March. Osma-Ruiz, V.J., Sáenz-Lechon, N., Gutiérrez-Arriola, J.M., Argüelles-Álvarez, I., Fraile, R. and Marcano-Ganzo, R. 2015. “Learning English is fun! Increasing motivation through video games”. Proceedings of the 8th International Conference of Education, Research and Innovation ICERI, Seville, Spain 16-18 November. Pachler, N., Bachmair, B. and Cook, J. 2010 Mobile Learning: Structures, agency, practices. Dordrecht: Springer. Pareja-Lora, A., Rodríguez-Arancón, P. and Calle-Martínez, C. 2016. “Applying information and communication technologies to language teaching and research: an overview”. New perspectives on teaching and working with languages in the digital era. Research-Publishing Net. Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 156 Rico, M., Agudo, J.E. and Sánchez, H. 2015. “Language learning through handheld gaming: A case study of an English course with Engineering students”. Journal of Universal Computer Science, 21(10), 1362-1378. Ryan, R.M. and Deci, E.L. 2000. “Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being”. The American psychologist, 55(1), 68-78. Ryu, D. 2013. “Play to learn, learn to play: Language learning through gaming culture”. ReCALL, 25(2), 286-301. Saleh, S.A. and Bhat, S.A. 2015. “Mobile learning: A systematic review”. International Journal of Computer Applications, 114(11), 1-5. Saran, M., Seferoglu, G. and Cagiltay, K. 2012. “Mobile language learning: contribution of multimedia messages via mobile phones in consolidating vocabulary”. Asia-Pacific Education Researcher, 21 (2012), 181-190 Sauerland, W., Broer J. and Breiter, A. 2015. “Motivational impact of gamification for mobile learning of Kanji”. EdMedia World Conference on Educational Media and Technology, 1, 1571-1580. Schilling, T.A. and Hayashi, C.T. 2001. “Achievement motivation among high school basketball and cross-country athletes: A personal investment perspective”. Journal of Applied Sport Psychology, 13 (1), 103–128. Shin, D.H. 2009. “Towards an understanding of the consumer acceptance of mobile wallet”. Computers in Human Behaviour, 25, 1343-1354. Sharples, M., Taylor, J. and Vavoula G. 2007. “A theory of learning for the mobile age”. R. Andrews, C. Haythornthwaite (Eds.), The Handbook of E-learning Research, London: Sage. Stockwell. G. 2008. “Investigating learner preparedness for and usage patterns of mobile learning”. ReCALL, 20(3), 253–270. Stockwell, G. 2013. “Technology and motivation in English-language teaching and learning”. In E. Ushioda (Ed.), International perspectives on motivation: Language learning and professional challenges, Basingstoke: Palgrave Macmillan, 156-175. Sykes, J. and Reinhardt, J. 2013. “Language at play: Digital games in second and foreign language teaching and learning”. ReCALL 25(2), 286-301. Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 157 Thornton, P. and Houser, C. 2005. “Using mobile phones in English education in Japan”. Journal of Computer Assisted Learning, 21, pp. 217-228 Ushioda, E. 2013. “Motivation matters in mobile language learning: A brief commentary”. Language Learning & Technology, 17(3), 1-5. Valarmathi, K.E. 2011. “Mobile assisted language learning”. Journal of Technology for ELT. 2(2). Viberg, O. and Grönlund A. 2012. “Mobile assisted language learning: A literature review”. In M. Specht, M. Sharples, and Multisilta J. (Eds.), mLearn 2012: Proceedings of the 11th International Conference on Mobile and Contextual Learning, Helsinki, Finland, 9-16. In http://goo.gl/mPOFs2 [Access: Dec 2016]. Wasko, M. and Faraj, S. 2000. “It is what one does: why people participate and help others in electronic communities of practice”. Journal of Strategic Information Systems, 9(23), 155-173. Werbach, K. 2014. “(Re)defining Gamification: A Process Approach. Persuasive Technology”. Proceedings 9th International Conference, PERSUASIVE, Padua, Italy, May 21-23, 266-272. Irina Argüelles Álvarez, Margarita Martínez Núñez, Ana Belén García Hernando and Antonio da Silva Fariña Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 158 APPENDIX 1 Cuestionario posterior a la utilización de la aplicación Valore las siguientes cuestiones relacionadas con su experiencia durante la utilización de la app UP2B2 para la mejora de sus conocimientos de inglés. Evalúe cada una con un número entre 1 (nada de acuerdo, muy poco) y 5 (muy de acuerdo, mucho). 1 2 3 4 5 La app me ha permitido utilizar contenido para aprender inglés cuando y donde lo he necesitado.       La app me ha permitido obtener información para aprender inglés de manera inmediata.       He podido estudiar el contenido necesario para el aprendizaje del inglés a través de un proceso de aprendizaje auto dirigido.       He podido acceder al contenido necesario para el aprendizaje del inglés directamente a través de la app.       La posibilidad de utilizar varios modos (competición, entrenamiento) para trabajar le da un carácter más personal a la app.       El poder utilizar las distintas modalidades de trabajo cuando yo quiera me ha permitido diseñar mi propia estrategia de aprendizaje.       Me gusta utilizar métodos nuevos y diferentes para estudiar inglés.       Soy una persona que suele probar cosas nuevas antes que otros.       Encuentro el contenido de la aplicación completo.       La aplicación funciona bien.       Esta aplicación me ha resultado útil para aprender inglés.       Con esta aplicación me ha resultado más fácil aprender inglés.       La app me ha permitido interactuar con otros estudiantes de inglés.       La utilización de una app facilita la comunicación entre estudiantes de inglés y proveedores de contenidos.       Aprender inglés utilizando una app es un reto que me agrada.       Utilizar una app para aprender inglés me hace invertir más tiempo en este aprendizaje.       Utilizar una app para estudiar inglés lo hace más divertido.       Utilizar una app para estudiar inglés me hace sentir bien.       Estoy satisfecho con el uso de una app para el aprendizaje del inglés.       Prefiero el uso de una app a otros métodos de aprendizaje del inglés.       Puedo mejorar mis conocimientos de inglés hasta el nivel que necesito a través de una app.       La combinación entre mis conocimientos previos de inglés y la información que puedo adquirir a través de una app me puede ayudar a mejorar mi nota de inglés.       Después de esta experiencia pienso seguir utilizando aplicaciones móviles para estudiar inglés.       Up2B2: playing English grammar games at the B2 level Language Value 9 (1),132–159 http://www.e-revistes.uji.es/languagevalue 159 Received: 10 December 2016 Accepted: 19 February 2017 Cite this article as: Argüelles Álvarez, Irina, Martínez Núñez, Margarita, García Hernado, Ana Belén, Da Silva Fariña, Antonio. 2017. “Up2B2: playing English grammar games at the B2 level”. Language Value 9 (1), 132-159. Jaume I University ePress: Castelló, Spain. http://www.e- revistes.uji.es/languagevalue. DOI: http://dx.doi.org/10.6035/LanguageV.2017.9.6 ISSN 1989-7103 Articles are copyrighted by their respective authors http://www.e-revistes.uji.es/languagevalue http://www.e-revistes.uji.es/languagevalue