Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 54 Can we improve academic performance and student satisfaction without additional time cost for teachers? Evidence from a blended methodology in Microeconomics Anett Erdmann, ESIC Business and Marketing School, anett.erdmann@esic.edu, Alfonso Jesús Torres Marín, ESIC Business and Marketing School, alfonso.torres@esic.edu Abstract Purpose: The main goal of this paper is to determine if the use of a blended methodology can improve performance and satisfaction of the students, with no additional time cost for teachers. As a second objective, the article attempts to explain observed differences across students in the effect of the methodology on study time based on the theory of optimal decision making. Finally, we sketch a simple cost benefit analysis for the digital learning platform (DLP) used. Design: The teachers combined the traditional classes methodology with the adoption of a DLP in two courses of Microeconomics at an undergraduate level at ESIC Business & Marketing School. Subsequently it is analyzed the impact of this methodology on student satisfaction and performance, as well as on student’s study time using different analytical tools. Findings: students’ grades, at the final exam, increased in a significative way as they spent more time with the DLP and/or when they do more digital assignments at home. Their satisfaction with the blended methodology, and the use of the DLP was quite high for most of students. Their feedback on working time relative to traditional methods showed two mailto:anett.erdmann@esic.edu mailto:alfonso.torres@esic.edu Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 55 extremes, either studying much more or much less. We provide a theoretical explanation for this observation, based on Microeconomic theory. A cost-benefit analysis of the DLP tool at an institutional level suggests that its economic costs are more than justified by the economic benefits of the tool in terms of student’s satisfaction, brand reputation and teachers time saving. Contribution: This document provides a methodology to measure the benefits of an innovative learning methodology using relevant indicators and employing advanced statistical techniques as regression analysis. It also helps us to understand student’s behavior in the face of an educational innovation based on technology. The findings are in line with economic theory. Keywords: Blended Learning, Technology-Enhanced Learning, virtual labs, educational innovation, computer software on education. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 56 1. BACKGROUND Despite the importance of innovation in the pre-university stage (Sangrá, 2016) (Rodríguez Miranda, 2009) (Tornero & Pi, 2013) and in certain university areas such as engineering, mathematics or biology (Beard, 2017) (Bravo, Guerrero and López, 2011), in Economics there are not many contributions of innovation in the university environment. Table 1 lists elements that may affect blended education in Economics. As shown in Table 1, there are new elements to consider in Economics education, that can be used to improve results from a blended perspective. Among them we can consider: (1) various types of instruments, (2) simulations, (3) transversal applications (4) use of different places. Table 1: New elements to consider in Economics teaching and the learning outcomes of ESIC students INSTRUMENTS that can be used for Economics training in Spanish Universities and specifically in ESIC. New teaching and learning tools and new technologies. Physical means of support for learning adaptation of the classrooms. Use of multimedia tools, social networks, blogs and virtual platforms. New teaching methodologies such as gamification, metacognition, flipped classroom. SIMULATIONS in the learning of various disciplines of Economics Learning supported by professional scenography’s. Experiences of professional praxis Exercises based in the resolution of problems Business games simulators Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 57 TRANSVERSALITIES to improve the learning outcome of Economics Teaching synergies with other areas of knowledge such as sociology, geography, business, demography, law, etc. Teacher synergies with other public and private institutions. Teacher synergies with professors or students from other Spanish and foreign Universities. PLACES of learning about the economy. Classrooms, libraries, homes and any other place with internet access. Source: (Torres, Gonzalez & Bordonado, 2018) As shown in Table 1, there are new elements to consider in Economics education, that can be used to improve results from a blended perspective. Among them we can consider: (1) various types of instruments, (2) simulations, (3) transversal applications (4) use of different places. New tools and learning technologies should be considered among the new instruments to be used. Multimedia tools, social networks, virtual platforms, as well as methodologies such as gamification, metacognition or flipped classroom stand out in this first category. In simulations in the learning, it can be included learning systems based on professional scenography, professional practice experiences, problem-based learning or business games simulators. Cross-cutting techniques should also be considered to improve learning outcomes in economics. For this, can be used synergies with other areas of knowledge - such as sociology, geography, business administration, demography, advocacy, etc. -, with other institutions -publics or privates- or with professors or students from other Spanish or foreign universities. Finally, different places can be used to learn about economics – classrooms, libraries, homes, and any other place with internet access. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 58 In many occasions some subjects in Economics are not the most preferred ones for students, they find them difficult and consequently they are not very much motivated to study (Becker & Greene, 2001). Sometimes these courses are criticized for being overly theoretical, uncreative, non-innovative and far from the real world (Liu & Xie, 2019). When this happen students simply get lost and fail. They do not acquire the knowledge and skills they should. In other cases, the courses become too mechanic, like receipts that people apply without understanding what they are really doing. To improve the previous situation, it is very helpful the use of innovative methodologies and techniques, that help students to improve their learning results and scoring through a learning by doing methodology using digital interactive tools (Lisitsyna, Efimchik, & Izgareva, 2017). The use of new technologies allows students to better understand concepts that are difficult to explain, such as the level of confidence of the impact of an advertising campaign on sales, the sensibility of the return of one stock to the changes in the market returns or the impact of education over salaries (Hattinger, Engeström, & Sannino, 2018). Blended Learning is a method commonly used by students from all over the world, either because the professor proposes it in class or autonomously as a support to the teacher's explanations (Medina, Medina, & Rojas, 2016). In any case, the use of digital platforms, which contain specific material by areas or blocks, has spread globally, with the use of information and communication technology (ICT) as a key element for learning (Pérez, Urbano, & Onías, 2017). The easy access to the different digital platforms, the volume of contents they provide, the immediacy of the information, as well as the guided learning and the support and control of the teacher, make the mixed methodology very effective for an achievement of the objectives of learning established in any course (Aguirre, Quintana, & Miranda, 2015). Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 59 Currently only in Moodle (Moodle.org, 2018) there are 130,037,486 registered users, of which 8,197 are registered in Spain, and in Miriadax (the largest online platform of Spanish language courses worldwide) there are 4,071,438 registered students worldwide (Miríadax_, 2018). Nowadays we also observe the implementation of other learning management systems in Spain, like for instance Canvas, implemented in the current academic year by a Business School (Cunef). On the other hand, publishers have gradually adapted to the new reality by incorporating digital materials into their catalogue, although in many cases, this adaptation usually includes the digitalization of the manual following a traditional structure, enriched with self-evaluating activities and additional resources (Castro -Rodríguez, De Castro Calvo, & Hernández Rivero, 2017). The editorial Pearson for instance uses different learning management systems. While the English platform MyLab – which we use in this teaching experiment – has been developed 100% by Pearson, the Spanish version of MyLab (iMI) is based on Sakai (Source: Pearson office). Another characteristic of this type of training is the degree of involvement required by the students. They can organize and plan their work, identify and solve problems, decide their work time, as well as evaluate their own progress (Fernández, 2005). This is significant considering that the labor market in Spain calls for university education, not only a more practical training aimed at the company, but a proactive and decisive attitude towards work, with notable disinterest and the tendency to mechanize the tasks assigned to young graduates (Alonso, Fernández Rodríguez, & Nyssen, 2009). The DLP is a teaching innovation initiative that apply the technological advances to interactive teaching based on the learning by doing methodology (Efimchik, Ivaniushin, Kopylov, & Lyamin, 2017). Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 60 For all the previous reasons, it is not surprising that the use of this type of methodologies is increasingly implemented in the university sphere (Brown, 2016). In this research we focus on a practical case of enabling and enhancing blends methodologies (Bonk & Graham, 2012) carried out at ESIC. The theory of optimal decision making and the impact of new educational technologies on study time As we will see in the findings of this research, there are strong evidences about two extreme behaviors of students with respect to their working time. Some of them suggest that they have work much more, meanwhile some others have work less than under traditional methods. How can we explain this pattern? In this section, we will base on the theory of optimal decision making to explain how the impact of new methodologies on study time may differ that much across individuals Study effort is in general difficult to measure. One possibility to think of working effort is in terms of the time spend on the studying for a subject. An alternative way of quantifying working effort is the amount of work, in terms of exercises, that have been realized, to which we will come back later. For the moment let us focus on the study time. In the following paragraphs we will give a short overview on the existing theory and empirical evidence on working time and performance, and later we will formulate our hypothesis about the effect of the DLP based on the theory of optimal decision making. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 61 First, let us consider evidence based on standard teaching methodology and the corresponding theory. In general, the literature on the effect of study time on grades find only a week relationship that sustains the common perception that studying more translates into a higher grade (Gortner Lahmers & Zulauf, 2000). For a detailed overview of these studies see Plant et al. (2015), who analyze the effect of the choice on study time of university students at Florida State University on the official university records. The authors identify study-time as a significant predictor of the grade, when controlling for other variables like attendance, study environment or planning which are supposed to determine deliberate practice. In general, the relationship between practice and performance is sketched by Ericcson (2002) as a process of practice that takes time to yield effects in performance. Figure 1 illustrates the expected performance of professionals, e.g. musicians, increasing with practice quite fast but once a certain experience has been acquired additional practice increases performance at a lower rate until a certain limit and may even turn down once physical maturity has been reached. While the representation by Ericsson refers to the career path over the working age, a similar relationship holds when considering the performance within the university career or at an even smaller scale at the course level. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 62 Figure 1. Relationship between performance and practice Source: Ericcson (2002), Figure 2.2, p. 25 Figure 2 represents an adaptation of the relationship between performance and practice to the academic performance within a university course, in line with the model of optimal study time by The Core Team (2017) and applied to our research question. The figure sketches the expected relationship between the exam score and the time studied under standard teaching methodologies. If students dedicate some time to study, the learning effect is large and is expected to be reflected in the exam score. However, as students dedicate more time to study, the effect of studying an additional hour on the exam score becomes smaller as dedicated hours increase, which is a standard learning effect. This outlined relationship between exam score and study time or number of assignments realized can be conceptually interpreted as a “score-production function”, that is, the production of exam score as a function of the study time or an alternative measure of study effort. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 63 Figure 2. Adaptation to academic performance based on study time Source: self-made Note that there are limits of improving performance. For instance, studying a few hours in a row without break, you will notice how your concentration decline and reach a point where you cannot assimilate more content. Likewise, the innate capability of each students for a particular task determines the limit where studying additional time will not be reflected in the performance. Second, based on the existing empirical and theoretical evidence on the trade-off between study-time and exam grade as well as the self-reported information by the students, we set up our hypothesis about the effect of the introduction of MyLab as complementary learning tool. In the next section we will test the hypothesis with objective data. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 64 Figure 3. Adaptation to academic performance based on study time: Expected Effect of DLP method Source: self-made Consider working with a DLP tool, such as MyLab, which provides direct feedback and offers walk-through options such that a certain performance can be reached with in less time. Analogue individual tutorial hours. Figure 3 illustrates this argument through a shift of the score-production function upwards, which is represented as a dashed line. In this context, the expected effect of working with the DLP tool MyLab is interpreted as a technology improvement in learning techniques which allows to improve performance. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 65 This hypothesis is in line with the recent literature on deliberate practice. For instance, Ericsson (2002), who analyzes the effect of the amount of high-quality practice accumulated during individuals’ careers on the performance, with reference to fields like practice of musicians, sportys, mathematicians etc. Third, given the relationship between work effort and exam score, let us consider the decision a student face. We all have a time constraint of 24 hours a day and we have to make a choice how to employ our time to achieve our personal objectives. Given that time is scarce, this implies for students that spending more time on a subject in order to increase performance – as outlined before – students have to scarify time for other activities, for instance studying for another course or simply enjoying free time. This trade- off between study time for a subject and free time (or alternative activities) is a key concept in economics in optimal decision making. An intuitive and narrative introduction of this trade-off as a first model of optimal decision making can be found in the open- source economic book by The Core Team (2017). The optimal decision about effort of studying - which translates according to the “score- production function” into a higher exam score - is an individual cost-benefit analysis by the students. While the potential benefits (e.g. improved understanding, complementary contents, better performance in exams) are easy to identify, there have been no monetary costs for students to use the platform. However, given the time constraint of students, they incur implicit costs known as opportunity cost, the best alternative activity they give up when dedicating time on studying with MyLab. This implies that the implementation of the platform has two effects on students: Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 66 1. “Time-WIN Effect”. The efficiency of learning liberates time. This additional free time can be employed to study more and climb up the learning curve and improve performance. 2. “Substitution effect”. The increased efficiency of studying with the platform implies an increase of the opportunity costs of study time. The substitution effect leads to less study time achieving the same performance as under traditional methods (“least-effort”). Which of the two effect dominates is an empirical question, that we analyze in the findings section of this article. 2. PURPOSE The main goal of this paper is to determine if the use of a blended methodology can improve performance and satisfaction of the students, with no additional time cost for teachers. As a second objective, the article attempts to explain observed differences across students in the effect of the methodology on study time based on the theory of optimal decision making. Finally, we sketch a simple cost benefit analysis for the digital learning platform (DLP) used. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 67 3. METHODOLOGY We adopted the Digital Learning Platform MyLab from Pearson for two groups of the course of Microeconomics in the degree of Business Administration at ESIC Business and Marketing School. This tool helps to create hybrid courses which require online homework as well as digital content delivery (Ahmed, Ohkubo, Limaye, & Ballard, 2017). It provides students with a larger number of practical exercises, with simultaneous guidance and in time feedback during the learning process. Simultaneously, it generates other benefits as dedicate more time in class to some other activities with more added value for students – as discussions of current topics, real-world application videos, simulations, experiments, critical thinking, work group and group discussions and questions - and to free more time for professor to dedicate to other professional activities. The tools offered by the platform are very flexible, and could be personalized by the professor so that: a) Students may dedicate a personalized number of hours to work and solve exercises, according to their needs. b) You can offer them different level of help, modify the deadline, or the scoring of the different problems. c) You may use them for practice, to consolidate some ideas (without scoring the students) or for testing and scoring them. d) You may do it at home, or in class, e) You have complete control: you may know individual results, and group result for individual questions, sections or topics. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 68 f) You can implement different help programs directed to individuals that do not understand something (they had poor results in some questions) or to solve general doubts when an important proportion of the class is having a poor performance in some specific questions or topics. Students have immediate feedback in case they need help. (“walk through options, “help”) As a final consequence of all the above-mentioned advantages, we expect students to be better oriented and learn more in quantity and/ or in quality, and this way improve their results, their learning experience and satisfaction and all this without increasing the time dedicated by the professor. Respective the time dedicated by students, we were not sure about the effect of MyLab on the time dedication of students. On the one hand, different features - like the instantaneous feedback, the walk-through options and the help option - allow students to solve problems and avoid being stuck, such that they may engage with this feature and use the additional free time to study more and improve performance – that could be denominated “time effect”-. On the other hand, a more efficient learning increases the opportunity costs of the time dedicated to learning and the substitution effect leads to less effort achieving the same result as under traditional methods. Which of the two effects dominate depends on student individual preferences. We adopted MyLab in 2 different groups, in total 50 second year students, during the first semester of 2018-2019, taught by different professors. In group 2 the professor assigned 6 compulsory homework’s in MyLab and two partial exams were realized in the platform under observation in computer rooms, such that 95% of the evaluation of the course was realized through MyLab. In group 1 the professor assigned only 4 compulsory Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 69 homework’s, but students were motivated to realize additional training using the individual study plan offered by the platform before doing the assignments. In the latter group the exams were realized on paper. Implementation The software is used by students working at home (and in class occasionally) on a personal computer on the cloud. Students use My Lab for understanding content, applying principles to the real world, homework assignments, and testing. Our goals by assigning tasks through My Lab are to complement the teaching of new concepts in the classroom by providing homework and practice opportunities, help students assess their own understanding of the course material, offer the possibility to deepen knowledge in topics they are especially interested in, and track their progress, and finally identify students who are at high risk of failing (Hinerasky, Fahr, & Sliwka, 2017). As course instructors our role is to assign content, homework and assessment in My Lab and provide support and remote monitoring to students using the program at home while the feedback and walk-through explanation of concrete exercises is directly and instantaneously provided by MyLab. While Students are often working remotely, the course is not self-paced as defined due date are firm. We encouraged students to be responsible for their own learning experience. After finishing each topic students were supposed to do some study planning (practice and quiz with no scoring), and subsequently the assignments that were scored. To encourage students to do assignments, their evaluation represents 10% of the final course scoring. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 70 4. FINDINGS To analyze the results two kinds of dataset have been used: (1) A first group of data has been directly extracted from MyLab and (2) a second group proceed from an individual student survey at the end of the semester. 4.1 THE STUDENTS EXPERIENCE This section presents the qualitative results from the individual student survey passed to all the students at the end of the semester. Responses from a voluntary survey of students indicate (see table 1) that most students recognize the value of My Labs - which reinforces the hypothesis established in this paper about the satisfaction of the students with the methodology and the tool. Specifically, we want to highlight the following overall results from that survey respective the DLP: a) 78% of students would recommend the use of the platform to other students of microeconomics. b) 80% of students strongly agree or agree that they liked the direct and instantaneous feedback for the digital assignments. c) 76% of students strongly agree or agree that the additional resources of the platform have helped them to learn more than in a traditional course. d) 80% of students agree or strongly agree that their understanding of the teacher presentation has increased because of using the digital platform. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 71 The detailed results of the survey are provided in Table 2. Table 2: Students opinion about the use of the digital platform in Microeconomics Questions Strongly disagree Disagree Agree Strongly Agree 1.- My digital understanding of the teacher presentations has increased because of using the digital platform 12,2% 7,3% 61,0% 19,5% 2.- Because of working with the digital platform I have worked more 14,6% 17,1% 24,4% 43,9% 3.- Using the digital platform provided additional resources to learn more than I would have from a traditional course 7,3% 17,1% 39,0% 36,6% 4.- I found it easy to get used to MyLab (e.g. web navigation, drawing tools for assignments) 12,2% 22,0% 29.3% 36,6% 5.- I liked the direct and instantaneous feedback for the digital assignments 2,4% 17,1% 29,3% 51,2% 6.- Because of working with the digital platform I have liked the subject more 22,0% 31,7% 26,8% 17,1% 7.- The use of Pearson digital platform has impacted positively my exam score 22,0% 14,6% 24,4% 39,0% 8.- I would recommend using Pearson digital platform to other students taking this course 9,8% 12,2% 34,1% 43,9% Source: self-made In general, the results from the survey suggest that students are satisfied with the DLP tool, which should not be confused with the fact whether they like the subject, which in turn doesn’t seem to be altered by the use of the digital tool. Especially the direct and instantaneous feedback by the platform seems to be valued by the student, which is reinforced by the student comments on the additional question what they liked most about using the digital platform. In the following we provide some of the representative student statements: Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 72 • “I like that when some question is wrong, it is clearly explained with instant feedback” • “Explanations when exercises are not done in the correct manner” • “I could work whenever I wanted and that if I had a question it could be answered whenever I had the question” • “Available to give us the correct answer and explain it in the moment. Easier and more handful” All these observations are in line with Pérez et al. (2017), who state that some of the advantages of digital platforms is the immediacy of information and guided learning. Our survey highlights these two aspects when using MyLab in terms of direct and instantaneous feedback for assignments and emphasize the “walk-through” help function. Respective the usage of the DLP in terms of working time, the results from the survey suggest differences across groups. At the overall level, 43,9% of all students strongly agree that they have worked more using the platform and 14,6% of all students strongly disagree. Considering the results at the group level, it calls attention that especially in group 1 the answers are polarized, which is represented in Graph 1. While 42% of the students disagreed or strongly disagreed on having worked more, 58% agreed or strongly agreed to have worked more. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 73 Graph 1. Answer to question: Because of working with the digital platform I have worked more. Group 1. Source: self-made Additional to the scale-questions in Table 2, we asked for the student’s willingness to pay for the use of the platform (which is analyzed in detail in section 4) and about the usage of book references. Respective the literature used in the course, the answers reveal that 66% of the students accessed the online book provided in MyLab often (at least 5 times during the term) while only 10% of the students had a look at the printed version of the book. All the previous results strongly suggest that our hypothesis about the positive effect of new educational technologies (DLP) on student performance may be confirmed. 0,00 0,10 0,20 0,30 0,40 0,50 I strongly disagree I disagree I agree I strongly agree Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 74 4.2. THE STUDENT PERFORMANCE As said before, one of the main goals of this paper is to determine if the use of a blended methodology can improve performance of the students. More specifically we claim as a relevant hypothesis that the average score of exams depends on the training time spend in the study plan or the number of assignments realized. In order to test our hypothesis about the score-production function and to identify the opportunity cost, we make use of the data extracted from MyLab, in particular, we analyze the number of assignments that the students did during the semester – as a means of quantity worked – and the time spent in the study plan – as a means of the time dedicated to study. Table 3 provides some summary statistics. Given that there are many potential differences between the two groups, the analysis is realized at a group level. The analysis with two groups can be understood as sensitivity analysis and is expected to reinforce the results based on small groups. Note that the average exam grade in both groups is similar, as well as the average Pearson score (score obtained in the MyLab assignments). However, this is not the case for the way students made use of the platform. In group 1, 25 out of the 28 students that composed it made active use of the study plan and realized on average 2,96 out of 4 compulsory assignments. In group 1, only 3 of the 22 students made use of the study plan but students were assigned more compulsory exercise sets and they realized on average 4,73 out of 6 assignments in total (including a first practice assignment) and 3,90 compulsory assignments out 5. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 75 Table 3. Summary Statistics MyLab. Variable Group 1 Group 2 Number of enrolled students 28 22 Average time spend in the study plan 7,96 hours (12,21) na Average number of compulsory assignments realized 2,96 (1,45) 3,90 (1,26) Average Pearson score (over 10) 5,69 (2,54) 6,30 (2,29) Average Exam score (over 10) 7,10 (1,34) 7,36 (1,14) *Standard deviations in brackets. Source: self-made The graph 2 shows the observations of time spent in the study plan and the corresponding exam grade (and Pearson grade respectively) and the adjusted score-production function as a fractional polynomial prediction plot for group 1. The data suggest a positive relationship between the time spent in the study plan by every student and the score in the final exam, with a steep initial increase of performance as study time increases but only a small performance effect if the time spend in the study plan is larger than 14 hours. Note that these observations are in line with the theoretical framework from the previous section suggesting a decreasing marginal effect of study time on the exam grade. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 76 Graph 2. Relation between score and time dedicated previously to do exercises in the study plan. Group 1 Source: self-made Given the small class size of 25 students, the results have to be interpreted with caution but can be understood as a first experimental result. In order to quantify the impact of the time spent in the study plan (for students of the group 1) on the student performance - or which is the same, the opportunity costs of study time using MyLab -, we provide a simple OLS analysis, which main results we explain in the following lines. The model that was estimated respond to a simple regression as reflected in the next equation: 𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 = 𝛽𝛽0 + 𝛽𝛽1(𝑇𝑇𝑇𝑇𝐸𝐸𝑠𝑠 𝑇𝑇𝑖𝑖 𝑀𝑀𝑀𝑀𝑀𝑀𝐸𝐸𝑀𝑀 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑀𝑀 𝑝𝑝𝑝𝑝𝐸𝐸𝑖𝑖) + ԑ Equation 1.1 Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 77 The Results obtained are (with standard errors in brackets): 𝐸𝐸[𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠] = 6,74 + 0,04(𝑇𝑇𝑇𝑇𝐸𝐸𝑠𝑠 𝑇𝑇𝑖𝑖 𝑀𝑀𝑀𝑀𝑀𝑀𝐸𝐸𝑀𝑀 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑀𝑀 𝑝𝑝𝑝𝑝𝐸𝐸𝑖𝑖) (Equation 1.2) (0,31) (0,02) The coefficients of the model show us that the time spent in the study plan significantly increases the exam grade. Spending one additional hour in the study plan increases the score in the exam by 0,04 points. The determination coefficient of the model (R2) is 15,1%, which may be interpreted as follow: the time dedicated to the study plan explains 15.1% of the variations in the final grade of the students with respect to the average. In none of the groups we find a significant effect of the score in the assignments on the exam score, which suggests that it is really the training that matters – in terms of the number of problems solved or the time spend in the study plan – rather than a matter of general analytical ability. Complementary, to verify the sensitivity of our results, we consider the relationship between the number of assignments realized during the course and the performance in the exams. As indicated previously, due to differences in the implementation of the DLP approach by the different professors, in group 2 almost no student used the study plan, but students were assigned more homework’s such that they practiced directly with the scored online assignments. Hence, for this group we measure work effort in terms of the number of assignments realized before the exam. We expect a positive relationship Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 78 between the number of exercise sets realized and the exam score. To test this hypothesis, likewise we use a simple OLS regression as follows: 𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 = 𝛽𝛽0 + 𝛽𝛽1𝐴𝐴𝑠𝑠𝑠𝑠𝑇𝑇𝐴𝐴𝑖𝑖𝐸𝐸𝑠𝑠𝑖𝑖𝑠𝑠𝑠𝑠 + ԑ (Equation 2.1) The previous equation helps us to quantify the impact of number of assignments done by students of group 2 during the semester on their performance The model that was estimated respond to a simple regression as reflected in the equation below. The figures between brackets represent the standard deviation . 𝐸𝐸[𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠] = 5,46 + 0,46𝐴𝐴𝑠𝑠𝑠𝑠𝑇𝑇𝐴𝐴𝑖𝑖𝐸𝐸𝑠𝑠𝑖𝑖𝑠𝑠𝑠𝑠 (Equation 2.2) (0,84) (0,19) We found that the exam score (average of the two partial exams in MyLab) significantly improves with the number of assignments realized, which confirms our motivation of the implementation of MyLab. The coefficients of the model show us that on average, the realization of an additional compulsory assignment improves the exam score in 0,46 points (out of 10). The determination coefficient of the model (R2) is 29,5%, which may be interpreted as follow: the number of assignments done explains 29,5% of the variations in the final grade of the students with respect to the average. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 79 The comparison between the determination coefficients of equation 1.2 and 2.2, (15,1% versus 29,5%) seems to suggest that the explanatory power of the number of assignments performed exceeds the correspondent to time dedicated to carrying out study plans), although we cannot discard some differences due to the diverse methodological approach followed by each one of the teachers. 4.3 THE EFFECT OF THE DIGITAL LEARNING PLATFORM MYLAB ON STUDY TIME This section presents the effect of the Digital Learning Platform MyLab on study time, helping us to verify the hypothesis about the different impact of the new educational technologies on study time across individuals. The descriptive data of the student survey, represented in Graph 1, suggest that, as mentioned between our hypothesis, there is heterogeneity between students, that is, preferences about study time and scoring differ across students. Moreover, given the differences between groups we believe that the instructors can influence the dominance of one effect or other through the implementation of the features offered in one way or another, which remains to be investigated further. The answers to the survey made to students – already mentioned in epigraph 4.1 - suggest two extremes of reported working time: (1) Strongly agree with having worked more or (2) Strongly disagree with having work more (which means working the same or less) than under traditional methods, which is especially pronounced for group 1 (see graph 1). Around one fourth of the students in the group 1 indicated that they strongly disagree with the statement that said that they have work more using the new technological Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 80 solution - and additional 15% more disagree-. These results can be explained over the bases of the optimal decision-making theory. In case the Time-Win Effect is dominating to the Substitution Effect, students will dedicate a part of their additional free time to study more and climb up the learning curve to improve their performance. However, if the Substitution Effect is the dominant, the increased efficiency of studying with the platform will lead to less study time achieving the same performance as under additional methods. Which of the two effect dominates is an empirical question? It is also convenient to mention that there is a small group of students that dedicate less time to study, simply because they are averse to the use of new technological educational platforms and they simply get discourage using it. In these circumstances they do the minimum effort to pass the course. 4.4. COST BENEFIT ANALYSIS FROM THE UNIVERSITY PERSPECTIVE Virtual platforms may either be directly adopted by the institutions, used as external services for application or integrated in existing platforms. In general, the implementation or transition of such platforms implies a high fixed cost for the institutions. Hence, in some respects, it may be profitable to rely on third party providers for student applications which merely imply variable costs in terms of student licenses. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 81 It could be interesting for any university to conduct a Cost-Benefit Analysis (CBA) of a the third-party provided virtual platform, and for that reason in this section it is sketched one CBA of MyLab for ESIC. In this article we defined the Net Benefit or Economic Rent of the use of MyLab as the sum of the net benefits for students, which is interpreted as student satisfaction, the net benefits for professors and the institutional costs. Net Benefit = (BenefitsStudent – CostStudent) *Students + Net Benefits Prof – Monetary Costs ESIC 1.- Students. The survey results evaluate several potential benefits and costs of the complementary use of the digital platform. Between the direct benefits we are considering the following: (1) The students are Increasing their understanding of the subject and learning how to better apply it; (2), They are learning more and contextualizing better; (3) The learning impact positively on their performance, generating better scores. To these benefits we should add the implicit ones, as for example, the need of working less time to pass the course, as one of the possibilities already mentioned in the epigraph 4. We cannot forget the implicit cost (or opportunity cost for the students). They will need some time to get used to it, which reduce their time for other activities (either study or leisure), and, they may have more working time (in this case normally to increase their scoring). Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 82 Based on these features and others, students make their judgement to calculate their economic rent (or net benefit), which implies the willingness to pay for the MyLab access. The survey data suggest that the most accepted price is 10 € (see graph 3), following a value-based pricing approach. It does not imply that a license should be priced at 10€. The institution needs to be considered as well. Graph 3.- Willingness to pay for MyLab use in Microeconomics Source: self-made 2.- ESIC. We consider that the main implicit benefits for the institution and the professors are the time saving, that can be used for the creation of class material, check of evaluation, grading, for other subjects or in general for other activities of high added value such as research, or congresses attendance. Other benefits for the institution may be better brand reputation, or image because of the use of new technologies in class, that can generate 0 1 2 3 4 5 6 7 8 9 0 5 10 15 20 40 60 80 100 Group 1 Group 2 Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 83 new student want to come ESIC in the future. However, we are not including it in this simplified CBA, given the complexity to estimate those values. The costs included in this analysis are the direct monetary costs of the licenses and the implicit costs of other activities (e.g. visiting professors or practioneers). Respective the monetary cost, the cost of licenses is calculated in 24,5€ (before VAT) per student, amounting a total 1.227,4€ for 50 licenses. Hence the net benefit, or economic rent, from an institutional point of view can be formulated as follows: Economic rent = 10€ * 50 + Net benefit prof * 2 – 1227,40€ To be profitable, the net benefit of each professors (without considering ESIC brand reputation or image) has to be at least 363,75€. Considering only the time saving in grading of the assignments and exams, we feel that this amount is justified. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 84 5. LIMITATIONS We need to consider, as the main limitation, that this project was conceived as a pilot with a total number of 50 students which implies that the results need to be interpreted with caution and give rise to further research in this area. 6. CONCLUSIONS The main purpose of this article is to provide an analysis to determine if the implementation of a blended methodology could improve performance and satisfaction of the students, with no additional time cost for teachers. As a second objective, the article attempted to explain observed differences across students in the effect of the methodology on study time, based on the theory of optimal decision making. Third, a simple cost benefit analysis for the digital learning platform (DLP), is used to analyze the implementation from a perspective of profitability for institutions. To address these questions, teachers at ESIC Business & Marketing School combined the traditional classes methodology with the adoption of DLP in two courses of Microeconomics at an undergraduate level. To analyze the results two kinds of dataset were used: (1) A first group of data extracted from MyLab and (2) a second group proceeding from an individual student survey at the end of the semester. The followings are the most remarkable conclusions obtained from this research. Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 85 We find evidence that student`s grades at the final exam increase in a significative way as they spent more time with the DLP and/or when they do more digital assignments at home. Spending one additional hour in the study plan increases the score in the exam by 0,04 points and the realization of an additional compulsory assignment improves the exam score in 0,46 points (out of 10). This suggests that it is really the training that matters rather than a matter of general analytical ability. There is no additional cost for the teacher to offer additional guided learning time, since the use of the study plan of the DLP and the “walk through option” substitutes individual tutorial hours. Standard tutorial hours with the professors were offered but almost not used at all. Likewise, the correction DLP method allows to increase the number of assignments implying for the teacher only a marginal cost in selecting a set of exercises and keeping track on performance, since the feedback and grading is automatized by the platform. Hence, we conclude that the use of the DLP provides students with opportunities to increase their academic performance without additional time cost for teachers. However, we find that students make use of this opportunities in different ways. The student feedback on working time in contrast with traditional methods showed two extremes, either studying much more or much less. A theoretical explanation for this observation, based on Microeconomic theory is provided. We argue that those students who work more than under standard methods take advantage of the more efficient Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 86 learning with the DLP to climb up the learning curve, while those students who work less take advantage of the more efficient learning time to reduce working time on the subject and liberate time for other activities. Which effect predominates depend on individual preferences. Given our experience with two groups, taught by different professors, our conjecture is that instructors can influence the dominance of one effect or other through the implementation of the features offered in one way or another, which remains to be investigated further I any case, we find that most students show a high level of satisfaction with the blended methodology and the use of the DLP. Responses from a voluntary survey of students indicate that most of them recognize the value of My Labs. Students recognize that the DLP help them to improve their understanding of the teacher presentation, and that the additional resources of the platform have helped them to learn more than in a traditional course. As a consequence, they liked more the subject, and it has impacted positively in their exam score. For all these reasons, most of them would recommend the platform to other students of microeconomics. All the previous results strongly suggest that our hypothesis about the positive effect of new educational technologies (DLP) on student satisfaction may be confirmed. A sketch of a cost-benefit analysis of the DLP tool at an institutional level suggests that its economic costs are more than justified by the economic benefits of the tool in terms of student’s satisfaction, brand reputation and teachers time saving, that can be dedicated to other activities with higher added value. As a summary, this article contributes with the provision of a methodology to measure the goodness of an innovative learning tool, using relevant indicators and employing Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 87 advanced statistical techniques as regression analysis. It also helps us to understand the key factors for the success of this blended methodology and why students may behave differently in the face of an educational innovation based on technology. The findings are in line with economic theory. REFERENCES Aguirre, A.C., Quintana H.P. & Miranda, R.T. (2015). Aplicación de las TIC en la educación superior como estrategia innovadora para el desarrollo de competencias digitales. Campus Virtuales, 3 (1), 88-101. Ahmed, N., Ballard, A., Ohkubo, S., & Limaye, R. (2017). Global Health eLearning: Examining the effects of blended learning models on knowledge application and retention. Alonso, L. E., Fernández, C. J., & Nyssen, J. M. (2009). El debate sobre las competencias. Una investigación cualitativa en torno a la educación superior y el mercado de trabajo en España. Madrid: ANECA. Beard, L. H. (2017). ‘Incentivised reading’-Using an online VLE to measure engagement and attainment in student learning. International Journal for Innovation Education and Research, 5(11), 74-86. https://doi.org/10.21125/edulearn.2017.0006 https://doi.org/10.21125/edulearn.2017.0006 Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 88 Becker, W. E., & Greene, W. H. (2001). Teaching statistics and econometrics to undergraduates. Journal of Economic Perspectives, 15(4), 169-182. https://doi.org/10.1257/jep.15.4.169 Bonk, C. J., & Graham, C. R. (2012). The handbook of blended learning: Global perspectives, local designs. John Wiley & Sons. Bravo, L. E. C., Guerrero, K. G., & López, H. J. F. (2011). Uso de las TIC y especialmente del blended learning en la enseñanza universitaria. Revista Educación y desarrollo social, 5(1), 151-160. Brown, M. G. (2016). Blended instructional practice: A review of the empirical literature on instructors' adoption and use of online tools in face-to-face teaching. The Internet and Higher Education, 31, 1-10. https://doi.org/10.1016/j.iheduc.2016.05.001 Castro-Rodriguez, M., De Castro Calvo, A., & Hernandez Rivero, V. M. (2017). Analysis of Spanish commercial digital educational platforms for Primary Education. Revista Latinoamericana de Tecnologia Educativa-RELATEC, 16(2), 49-62. Efimchik, E. A., Ivaniushin, D. A., Kopylov, D. S., & Lyamin, A. V. (2017). A Technique for Applying RLCP-Compatible Labs on Open edX Platform. In E-Learning, E-Education, and Online Training (pp. 12-18). Springer, Cham. https://doi.org/10.1007/978-3-319-49625-2_2 Ericsson, K. A. (2002). Attaining excellence through deliberate practice: Insights from the study of expert performance. https://doi.org/10.1257/jep.15.4.169 https://doi.org/10.1016/j.iheduc.2016.05.001 https://doi.org/10.1007/978-3-319-49625-2_2 Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 89 Fernández March, A. (2005). Nuevas metodologías docentes. Valencia: Universidad Politécnica de Valencia, Valencia. Gortner Lahmers, A., & Zulauf, C. R. (2000). Factors associated with academic time use and academic performance of college students: A recursive approach. Journal of College Student Development, 41, 544-556 Hattinger, M., Engeström, Y., & Sannino, A. (2018). From contradictions to transformation: a study of joint Work-Integrated Elearning between Industry and University. Journal of Engineering Education. Hinerasky, C., Fahr, R., & Sliwka, D. (2017). Learning outcomes, feedback, and the performance effects of a training program. Universitätsbibliothek. Lisitsyna, L. S., Efimchik, E. A., & Izgareva, S. A. (2017, June). RLCP-compatible Virtual Laboratories with 3d-models and Demonstration Mode: Development and Application in e-learning. In International Conference on Smart Education and Smart E-Learning (pp. 75- 81). Springer, Cham. https://doi.org/10.1007/978-3-319-59451-4_8 https://doi.org/10.1007/978-3-319-59451-4_8 Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 90 Liu, Y., & Xie, T. (2019). Machine learning versus econometrics: prediction of box office. Applied Economics Letters, 26(2), 124-130. https://doi.org/10.1080/13504851.2018.1441499 Medina, J. M. C., Medina, I. I. S., & Rojas, F. R. (2016). Uso de objetos virtuales de aprendizaje ovas como estrategia de enseñanza–aprendizaje inclusivo y complementario a los cursos teóricos–prácticos. Revista educación en ingeniería, 11(22), 4-12. Miríadax (2018, 3 de mayo). Nuestros números. Recuperado de https://miriadax.net/web/guest/nuestros-numeros Moodle.org. (2018, 3 de mayo). Course and content. Recuperado de https://moodle.net/stats/ Mosbeck, M., Markum, G., Platschek, A. & Jantsch, A. (2018). E-Learning System for Automatic Generation and Evaluation of Per-Student Customized Tasks for Hardware Modeling Courses, World Academy of Science, Engineering and Technology, International Jour. https://doi.org/10.1109/NORCHIP.2018.8573455 Sangrá, L. M. (2016). La nueva brecha digital. El futuro de las nuevas tecnologías en Primaria desde la formación del profesorado. Revista electrónica interuniversitaria de formación del profesorado, 19(2), 301-313. https://doi.org/10.1080/13504851.2018.1441499 https://doi.org/10.1109/NORCHIP.2018.8573455 Multidisciplinary Journal for Education, https://doi.org/10.4995/muse.2019.11869 Social and Technological Sciences ISSN: 2341-2593 Erdmann and Torres-Marín (2019) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 6 Nº 2 (2019): 54-91 | 91 Pérez, P., Urbano, J. & Onías, A. (2017). Los recursos didácticos digitales en la calidad del aprendizaje significativo en el área de matemáticas, (Bachelor's thesis, Universidad de Guayaquil). Plant, E. A., Ericsson, K. A., Hill, L., & Asberg, K. (2005). Why study time does not predict grade point average across college students: Implications of deliberate practice for academic performance. Contemporary Educational Psychology, 30(1), 96-116. https://doi.org/10.1016/j.cedpsych.2004.06.001 Rodríguez Miranda, F. D. P. (2009). Los coordinadores y coordinadoras de Centros TIC: su importancia como asesores internos para la calidad y el desarrollo del currículo en la Educación Primaria de Andalucía. The Core Team (2017). The Economy: Economics for a changing world. Oxford University Press. Recuperado de http://www.core-econ.org/the-economy/index.html Tornero J.M. & Pi, M. (2013). La integración de las TIC y los libros digitales en la educación. España: Grupo Planeta. Torres, A.T., Gonzalez E. y Bordonado, J. (2018, 2-4, July). Using a digital interactive labs educator tool to improve the scoring and learnig results of marketing students of econometric in ESIC Business & Marketing School. 10th International Conference on Education and New Learning Technologies. Palma Spain. ISBN: 978-84-09-02709-5. Doi: 10.21125. https://doi.org/10.1016/j.cedpsych.2004.06.001 4. FINDINGS Liu, Y., & Xie, T. (2019). Machine learning versus econometrics: prediction of box office. Applied Economics Letters, 26(2), 124-130. https://doi.org/10.1080/13504851.2018.1441499