FLIPPING THE ECON class: Reconsidered 1 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 1 FLIPPING THE ECON CLASS: RECONSIDERED Todd Broker1, Victor Raj2, and Simone Silva3 Abstract This paper examines the effectiveness of flipping the classroom by comparing exam performance in several microeconomics courses taught by the same instructor over the course of one academic year. Overall, we found mixed evidence regarding the effectiveness of exposing students to a flipped classroom environment. While flipping the class may improve exam scores after controlling for numerous independent variables, these results are not robust across specifications, and deeper analysis showed that certain groups of students were actually hurt by the classroom format change. Somewhat contrary to other research, our findings suggest that flipping the classroom puts more responsibility on students and some student subgroups do not handle this change effectively, though course design and other variables can also be relevant factors. Key Words: flipped classroom, achievement, learning, engagement JEL Classification: A20, A22 Introduction Teaching more effectively has been a long-standing pursuit for educators, especially among those who teach what are perceived to be more challenging courses, such as economics. One teaching strategy that has recently grown in popularity is called the “Flipped Classroom.” This format reverses the typical or “Traditional Classroom” format by moving lectures to an online platform for students to absorb individually outside of class. Once class time is free from the constraints of a typical lecture, classes are redirected towards more group-based interactive learning activities (Bishop & Verleger 2013). This study seeks to better understand the effect of flipping the classroom on student performance, particularly with regard to economic education. To this end, we gathered data from four different sections of the introductory microeconomics course at Murray State University, a four-year public regional university in the south during the academic year 2015-2016. Each of these sections had the same instructor and approximately half of the students were exposed 1 Director, Center for Economic Education, Arthur J. Bauernfeind College of Business, Murray State University, 307 Business Building, Murray, KY 42071 (corresponding author) 2 Professor of Computer Science & Information Systems, Arthur J. Bauernfeind College of Business, Murray State University, Murray, KY 42071 3 Associate Professor of Economics, Department of Economics and Finance, Arthur J. Bauernfeind College of Business, Murray State University, Murray, KY 42071 2 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 2 randomly to the flipped format while the other half were exposed to the traditionally formatted class structure. Interestingly, students in the flipped classes performed worse on exams overall than students in the traditionally formatted classes (75.4% avg. compared to 79.7% avg.). However, after controlling for various student-specific factors, this result only remained significant for particular cases. In addition, when pooling all exams in the regression analysis to increase the degrees of freedom, we found some positive but not robust evidence of flipping the classroom. Thus, overall, this experiment’s results suggest that the benefits of flipping a class are not consistent, although this may arise from the small size of the sample as well as some specific student groups suffering a negative impact on their learning. Specifically, the interaction between a student’s prior GPA and flipping the class suggests that this variable, which is typically a significant predictor of students’ future educational success (Larose & Roy 1991), may be less influential when a student is in a flipped class. In addition, students that work more hours on outside jobs also struggled with the change in the classroom format. Possible explanations of these results are explored further below. The literature on the flipped classroom is reviewed in the next section. The experimental design of our study is then laid out. Next we analyze the student data and discuss our findings. Finally, we propose various implications of our results, which we hope will add to the discourse on this relevant pedagogical topic. Review of the Literature The flipped class has been prevalent in education literature at least since 2000 (Lage et al. 2000). Since that time, some educators have concluded that their lectures were most effective if reviewed by the students outside of the classroom. This reversal of format, they reasoned, would free up valuable class time for more interactive lessons (Tucker 2012). What makes the flipped classroom model so appealing? Researchers have noted that it may have numerous benefits including (but not limited to) aiding in the development of life-long learners, increasing students’ engagement with the material, and increasing the quality and quantity of interactions between students and faculty (Bergmann, et al. 2014). Furthermore, flipping the classroom allows instructors more class time to tailor learning experiences to better match each student’s unique learning style, thus improving educational outcomes (Lage, et al. 2000) and may also reduce cheating among students (Hoxie, et al. 2015). Advocates of the flipped class format argue that educational theory has long shown that traditional lectures are essentially ineffective at fostering student learning and that student-centered approaches create better student-learning outcomes (Bishop and Verleger, 2013). For example, research has found that students learn slightly more from visual-based instruction (videos) than from conventional lectures (Cohen, et al. 1981) and that students come to class better prepared when supplied with optional videos to watch about the class content (Falconer, et al. 2009). This 3 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 3 is important, especially considering that students tend to shirk outside class reading assignments (Sappington, et al. 2002). Flipped classroom supporters also point out that successfully flipping the classroom involves a lot more than just adding technology to a set of curricula (Tucker, 2012). Indeed, technology by itself, when implemented for activities such as homework assignments, has a mixed record on improving student learning (Bonham, et al. 2003). Many authors argue that a successful strategy involves changing the instructor’s approach to teaching. This occurs by putting more responsibility for learning in the students’ hands within an engaging student-centered learning environment (Bergmann, et al. 2014). This approach not only takes advantage of technology, but can capture the benefits of cooperative learning (Johnson and Johnson, 2001). However, when implementing a flipped classroom, many educators may find it difficult to fully integrate the new format for a variety of reasons. For instance, some instructors may find it difficult to remove themselves from the primary role of importance, moving from the “sage on the stage” to more of a “guide on the side” (Frydenberg, 2012). Also, if instructors fail to communicate the purpose of the new format, fail to convince the students of its advantages, or are simply unwilling to “let go” of their traditional lecture practices, flipping the classroom may prove to be ineffective (Findlay-Thompson and Mombourquette, 2014). Successful implementation of the flipped classroom format must incorporate these factors and can also benefit from using shorter lecture videos, less than 15 minutes in length (Stone, 2012), as well as creating incentives to review the video content through the use of lecture video assessments (Bishop and Verleger, 2013). Current research about student preferences seems to suggest that students are generally receptive to changing the instructional format to the flipped classroom model (Bishop and Verleger, 2013). Not all students necessarily like the change. Some seem conflicted, because, although they like the interactive class time in the flipped format, they miss the live in-person lectures found in the traditional class (Toto and Nguyen, 2009). Some studies have shown that flipped classrooms require more self-discipline on the part of students and more time spent with the class material than the traditional format. Even though both of these may help students learn the material, neither would likely be well received (Lee and Lee, 2015). Other studies have shown that students prefer the flipped classroom, even when students feel the new format is more challenging (Wilson, 2013). Regarding student learning, many studies suggest that a flipped classroom improves academic outcomes. For example, flipping a large-lecture principles of economics course yields numerous positive benefits for students, including better performance and effort during the semester (Balaban, et al. 2016), even though the difference in average student improvement is often modest between traditional and flipped classes (Olitsky, 2016). Additionally, adopting the flipped format in an introductory biology course has been shown to improve exam performance significantly (Moravec, et al. 2010). The flipped format has also positively affected student performance in senior-level computer classes (Day and Foley, 2006). Also, flipping a large biology class and a smaller genetics class showed that academic results in 4 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 4 the flipped classes were consistently higher than in the traditional format, although the effects were more robust for the smaller more specialized genetics class (Stone 2012). Other studies have shown mixed results with little to no effect on student performance from flipping the classroom. For example, researchers in Canada studied the effects of flipping some university level introductory business courses. They found that student academic outcomes were identical across the different sections of the course, regardless of class structure (Findlay- Thompson and Mombourquette, 2014). Thus, given the current literature, it may be difficult to say definitively whether or not flipping the classroom will be a “home run” for any instructor. Further investigation is needed. Experimental Design This study was structured as an experiment using four different class sections of a microeconomics course taught by the same instructor over the course of one academic year (2015- 2016) at Murray State University, a four-year public regional university in western Kentucky. Two of the course sections were taught in the fall semester and two in the spring. During each semester, one of the sections met during a late morning time slot and the other met during an early afternoon time slot. In each semester, one course section was flipped for this study and the other section continued under the traditional format. Students did not know which course sections were flipped or traditional before registering for the course. Also, the specific time slot for the flipped class varied between semesters; the afternoon class flipped in the fall semester and the morning class flipped in the spring semester. Each section of the course covered identical course content during the year (e.g. identical textbook chapters) and used identical standardized exams. Other features of the course, including grading scheme, attendance policy, etc. were identical. Structurally, the traditional sections were setup with “chalk and talk” lectures given in class twice per week with the aid of PowerPoint and with 2-3 homework assignments on the course’s online homework platform. Very little, if any, class time was devoted to working in groups to collaborate on homework problems or questions in the traditional sections. On the other hand, the flipped sections involved two unique structural differences: 1) All class lectures were recorded by the instructor (interactive presentation and audio) and were required viewing for the students outside of class. 2) Roughly 75% of the newly available class time was spent collaboratively working on chapter-specific, group-based problem sets unique to the flipped class sections. During a typical week in the flipped sections, the first class session was spent conducting a high level overview of the material with an introduction to new content and the use of hands-on worksheets and examples that were completed in small groups. These same worksheets were often used in the traditional classes as well, but with less emphasis, interaction, and time. The second class session during the week for the flipped sections was usually spent assessing students for 5 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 5 watching the online lectures (i.e. lecture quiz) and then allowing the students to collaboratively work on a set of graded problems and questions in groups. During this time, the instructor consistently walked around the classroom to aid students in their understanding and application of the material. There were 30 lecture videos uploaded for the flipped sections, each video ranging between 15 and 30 minutes in length with an average of 24 minutes. The total online video lecture duration (i.e. all the lecture content added together) was about 12 hours.4 To encourage students to watch the online lecture videos, two separate lecture assessments were given, one in class and one online. Exactly how many students watched the video lectures and for how long is unknown, but a simple tally of the number of online views5 was 1,714 shortly after the completion of the academic year.6 This means that on average each online lecture video was viewed only 57 times. Even if we assume that each recorded video view represents a unique student (which is far from likely) this number is still far less than the total number of students enrolled in the flipped classes during the experiment. In other words, it is extremely likely that not all students were watching the online lecture videos. As noted in the literature review, flipping the classroom can be difficult and can be done poorly. Despite the best efforts of the instructor in the experiment, it seems obvious that the video format was not structured effectively, nor were the majority of students given proper incentives to watch the lecture videos outside of class. These could be important factors in the final results. Data During this experiment in academic year 2015-2016, 172 students7 were enrolled in the four sections of microeconomics studied. Of these, 145 agreed to participate in this research (~ 84%). Most of the students who did not participate in the study were not present in class on the day of data collection. Asking students to participate in the study (in terms of data collection) occurred near the end of the course to limit students’ ability to change sections based on their preferences for one teaching method over the other. To our knowledge, no student changed course sections to either avoid or seek out the flipped class structure during the study period. Data on study participants were collected from three different sources. One was a short survey voluntarily filled out by participants near the end of the semester. The survey asked about the students’ attitude towards the course, the instructor’s contribution to the class, and some demographic information (Appendix 1). The second source of data was student activity in class, such as attendance and exam performance. Finally, the third source was individual student data gathered from the university Registrar, such as a student’s prior GPA, etc. 4 This lecture time compares to total traditional class lecture time of between 30-35 hours spread out over about 30-45 class periods in 50-75 minute individual class sessions. In sheer time magnitude, the online videos represented only a fraction of the burden to students that traditional class lectures typically represent. 5 YouTube counts a video view after a user has watched a video for “around” 30 seconds 6 August 2016 7 There were 180 students that began the course in these four sections but only 172 completed the course 6 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 6 Table 1 contains explanations of the variables used in our analysis and Table 2 gives descriptive statistics of the same variables. Table 1: List of variables used in the analysis Variable Description Overall Exam Performance Students’ exam average based on three equally weighted standardized tests given in each class Exam 1 Performance Students’ standardized grade for exam 1 Exam 2 Performance Students’ standardized grade for exam 2 Exam 3 Performance Students’ standardized grade for exam 3 Flipped Classroom: Dummy A dummy variable equal to 1 if the student was in one of the instructor determined “Flipped” classrooms Prior GPA The student's cumulative grade point average prior to taking economics Class Attendance (%) The instructor determined absence percentage for each participating student Perfect Attendance: Dummy A dummy variable equal to 1 if the student had perfect attendance during the semester Race (White): Dummy A dummy variable equal to 1 if the student was White / Caucasian Gender (Male): Dummy A dummy variable equal to 1 if student was male Age Variable for how old the student was in years when enrolled in economics course KY Residence: Dummy A dummy variable equal to 1 if the student’s permanent residence was labeled as in-state (KY) STEM Major: Dummy A dummy variable equal to 1 if the student’s major was labeled as either science, technology, engineering, or mathematics Educational Attainment of Parent / Guardian The student-reported highest educational attainment of their parents / guardians 7 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 7 Parent / Guardian College Degree: Dummy A dummy variable equal to 1 if at least one parent / guardian of the student had earned a college degree Course Rating The student's rating, on a 10-point scale, of how they felt about their economics course overall (10 = highest positive) Instructor Rating The student's rating, on a 10-point scale, of the instructor’s contribution to their economics course (10 = highest positive) Effort Rating A student-reported rating, on a 10-point scale, of how much effort they put forth in their economics course compared to their other courses Use of Online Materials A student-reported rating, on a 10-point scale, of how much they utilized online materials in their economics course Hours Devoted to Course The student-reported number of hours they spent working on material for their economics course in a typical week Hours of Employment The average number of hours the student worked at a job in a given week during the semester Data Sources Student self-reported data came from a voluntary survey and all other data was collected, with permission, from the university registrar 8 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 8 Table 2: Variable descriptive statistics Variables Mean Standard Deviation Minimum Maximum Obser- vations Overall Exam Performance 77.57 12.86 43.33 100.33 145 Exam 1 Performance 78.11 13.95 26.00 103.00 145 Exam 2 Performance 77.71 13.35 34.00 100.00 145 Exam 3 Performance 76.88 15.32 39.00 104.00 145 Flipped Classroom: Dummy 0.50 0.50 0.00 1.00 145 Prior GPA 2.86 0.87 0.00 4.00 145 Class Attendance (%) 0.09 0.11 0.00 0.71 145 Perfect attendance: Dummy 0.26 0.44 0.00 1.00 145 STEM Major: Dummy 0.14 0.35 0.00 1.00 145 Race (White): Dummy 0.85 0.36 0.00 1.00 145 Gender (Male): Dummy 0.50 0.50 0.00 1.00 145 Age 18.72 1.72 16.00 27.00 145 KY Residence: Dummy 0.56 0.50 0.00 1.00 145 Edu. of Parent / Guardian 3.51 1.07 1.00 5.00 144 At Least One Parent / Guardian Has College Degree: Dummy 0.57 0.50 0.00 1.00 144 Course Rating 8.05 1.55 4.00 10.00 145 Instructor Rating 9.12 1.31 4.00 10.00 145 Effort Rating 7.00 1.79 2.00 10.00 145 9 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 9 Use of Online Materials 7.54 2.24 1.00 10.00 145 Hours Devoted to Course 5.72 2.55 1.00 15.00 145 Hours of Employment 11.14 12.24 0.00 50.00 145 Methods and Results The fact that students were not aware of the classroom format when registering should simplify the statistical analysis since, presumably, it avoids the issue of self-selection bias, a fundamental problem in micro econometric evaluation studies (Caliendo and Kopeinig, 2008). However, given the small size of the sample (145 participants), there is a considerable possibility that, even if by coincidence, the two groups of students differ significantly in characteristics relevant to academic performance. For instance, if the students enrolled in the traditional class sections happen to have a stronger work ethic than the students enrolled in the flipped class sections, one may wrongly conclude that flipping the classroom negatively impacts academic achievement, when in reality the lower grades are most likely the result of less effort. This possibility is considered in Table 3, which presents the means of the variables for the flipped and traditional sections. First, note that the students in the flipped sections on average performed worse on the exams than the students in the traditional class sections. We computed the (James, 1954) test of mean equality across groups, which is robust under heterogeneous covariance matrices, and found that this difference is significant at 90 percent in the case of the second and third exams. However, students in the traditional classes also presented significantly higher average GPAs prior to the semester and a larger fraction of them were STEM majors which, according to anecdotal evidence, tend to perform better in the subject. Therefore, although most of the other variables seem balanced across the two sets, this information must be kept in mind when comparing the average grades across groups and in performing multivariate analysis. To investigate the effects of flipping the classroom, we calculated the overall exam performance, the simple average of the three exams’ scores, for each student. Next, we grouped the students by several variables related to academic achievement and calculated the average of the overall performances across groups, also differentiating by classroom structure. The results are presented in Table 4. Note the second to last column, which presents the p-values (F test) of the null hypotheses that, for each group (defined in the rows), the averages of the performances are the same for the flipped and the traditional cases. 10 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 10 Table 3: Means of the variables across flipped and traditional classroom sections Variables Traditional Flipped Total Traditional - Flipped P- value* Exam 1 Performance 79.9 76.3 78.1 3.6 0.121 Exam 2 Performance 80.0 75.4 77.7 4.6 0.037 Exam 3 Performance 79.2 74.5 76.9 4.6 0.069 Gender (Male): Dummy 0.5 0.5 0.5 0.0 0.804 Race (White): Dummy 0.9 0.8 0.8 0.1 0.341 Age 18.7 18.7 18.7 0.0 0.951 Prior GPA 3.1 2.7 2.9 0.4 0.005 STEM Major: Dummy 0.2 0.1 0.1 0.2 0.004 Perfect attendance: Dummy 0.2 0.3 0.3 0.0 0.672 Hours Devoted to Course 5.8 5.6 5.7 0.2 0.563 Hours of Employment 11.8 10.4 11.1 1.4 0.496 At Least One Parent / Guardian Has College Degree: Dummy 0.6 0.6 0.6 0.0 0.739 Course Rating 8.1 8.0 8.0 0.0 0.875 Instructor Rating 9.1 9.1 9.1 0.0 0.845 Effort Rating 7.0 7.0 7.0 0.1 0.782 Sample Size 73 72 145 *Note: Prob. > F: James’ (1954) test for equal means, allowing heterogeneous covariance matrices across by- groups. 11 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 11 Table 4: Average Overall Exam Performance across Students’ Groups Trad- itional Flipped Traditional - Flipped P- value* Obser- vations All Obs 79.7 75.4 4.3 0.045 145 Male 79.9 76.9 3.0 0.337 72 Female 79.5 74.0 5.5 0.062 73 White 80.3 76.3 4.0 0.072 123 Non-White 75.4 71.5 3.9 0.576 22 Up to 18 Years Old 77.5 75.8 1.7 0.512 92 Older than 18 Years Old 83.2 74.7 8.5 0.019 53 High Prior GPA (>3) 84.8 83.4 1.4 0.569 71 Low Prior GPA (0-3) 72.3 70.3 2.0 0.491 74 STEM Major 78.8 78.3 0.5 0.955 20 Not a STEM Major 79.9 75.2 4.7 0.031 125 Perfect Attendance 79.8 79.5 0.3 0.938 38 Less than Perfect Attendance 79.6 73.8 5.8 0.018 107 Often Use of Online Material (9-10) 77.4 74.6 2.8 0.354 59 Less Use of Online Material (1-8) 80.9 76.1 4.9 0.104 86 Hours Devoted to Course (6-15) 74.8 72.6 2.2 0.466 63 Hours Devoted to Course (1-5) 83.1 77.8 5.3 0.063 82 Hours of Employment (>9) 78.9 73.6 5.3 0.093 71 Hours of Employment (0-9) 80.7 76.8 3.8 0.204 74 12 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 12 At Least One Parent / Guardian Has College Degree 81.2 76.6 4.7 0.097 82 Parents/Guardians Have No College Degree 77.3 74.0 3.3 0.316 62 *Note: Prob > F: James’ (1954) test for equal means, allowing heterogeneous covariance matrices across by- groups. When all observations are included, we can see that the hypothesis that the averages are the same in both class structures is rejected: students did worse in the flipped classes. When separating by gender, however, there is statistical evidence that only the female students did worse, indicating that women adapted worse to flipping the classroom. A similar pattern is found regarding white and non-white students. While the sample sizes of male and female participants are almost identical, there are only 22 non-white students, reducing our confidence in this finding. More interesting are the results regarding the following groups: students older than 18 years of age, students that are not STEM majors, and students with less than perfect attendance. Note that the p-values that indicate that students in these groups performed worse in the flipped format are even smaller than the p-value when all observations are included, even though the sample sizes are smaller. Students that devoted less than 5 hours of work per week to the course or worked more than 9 hours per week at a job (i.e. employment) also had issues adapting to the new format, and, somewhat surprisingly, so did students who reported at least one parent/guardian having a college degree. Overall, to the extent that perfect attendance, hours devoted to the class per week and hours spent towards a job/employment (all of which crowd out time to study) are proxies for effort put into the course, it seems that flipping the class has a detrimental effect only on less dedicated students. We get further insights with a multivariate analysis next. Table 5 presents the OLS regression results using the Huber/White/sandwich estimator of variance–covariance matrix to account for the possibility of heteroscedasticity. The dependent variables are the overall performance in the exams with each exam’s grade considered separately. Two models are considered for each case, the second including more controls. Given that these variables are obviously related to each other, most of them being proxies for effort, we checked for collinearity across variables and found no major problem.8 In the last two columns, we pooled 8 Multicollinearity (so long as it is not perfect) does not violate OLS assumptions and the estimates are still unbiased and BLUE (Best Linear Unbiased Estimators). However, it can substantially increase standard errors reducing the level of significance of the results. When we include the interaction between flipping the classroom and age and the interaction between flipping the classroom and prior GPA in the same regression, we detected a variance inflation factor above 10 for these two variables, 10 being the rule of thumb to consider multicollinearity a potential problem. This is expected since both variable are interacted with the same dummy. We redid all our analysis excluding the interaction with age. The results, qualitatively similar (although slightly less significant), are available upon request. 13 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 13 all three exams in order to increase the degrees of freedom. When interpreting these results, however, one should keep in mind that we do not actually have a larger sample, the same control variables are included three times, once for each exam. Table 5: Multivariate analysis Estimates Variables All Exams All Exams Exam 1 Exam 1 Exam 2 Exam 2 Exam 3 Exam 3 Exams Pooled Exam Pooled coef/p- value coef/p- value coef/p- value coef/p- value coef/p- value coef/p- value coef/p- value coef/p- value coef/p- value coef/p- value Flipped Classroom: Dummy 45.265 42.660 47.515 44.990 38.045 34.739 50.236 48.252 45.26** 42.660* (0.210) (0.241) (0.195) (0.234) (0.326) (0.372) (0.218) (0.232) (0.038) (0.051) Gender (Male): Dummy 0.854 0.901 3.404 3.294 -1.682 -1.412 0.840 0.820 0.854 0.901 (0.727) (0.715) (0.207) (0.232) (0.560) (0.617) (0.766) (0.776) (0.592) (0.571) Interaction: Flipped classroom, Male Student 1.963 2.207 1.323 1.851 0.934 1.065 3.633 3.706 1.963 2.207 (0.604) (0.576) (0.757) (0.674) (0.821) (0.799) (0.420) (0.435) (0.420) (0.376) Age 1.227 1.353 0.636 0.933 1.511 1.493 1.534 1.632 1.227 1.353* (0.373) (0.326) (0.622) (0.479) (0.279) (0.290) (0.334) (0.298) (0.124) (0.089) Interaction: Flipped Classroom, Age -1.400 -1.300 -1.679 -1.597 -1.255 -1.078 -1.266 -1.226 -1.400 -1.300 (0.437) (0.477) (0.383) (0.419) (0.489) (0.562) (0.516) (0.527) (0.190) (0.227) Prior GPA 10.38*** 10.13*** 8.65*** 8.29*** 9.08*** 9.13*** 13.41*** 12.98*** 10.38*** 10.13*** (0.000) (0.000) (0.004) (0.006) (0.001) (0.001) (0.000) (0.000) (0.000) (0.000) Interaction: Flipped Classroom, Prior GPA -7.397** -7.113** -6.636* -6.215* -5.477 -5.477 - 10.07*** - 9.647*** -7.397*** -7.113*** (0.019) (0.023) (0.058) (0.077) (0.111) (0.101) (0.003) (0.005) (0.000) (0.000) Perfect attendance: Dummy -0.695 -0.558 -2.422 -2.300 0.639 1.151 -0.302 -0.525 -0.695 -0.558 (0.821) (0.863) (0.523) (0.563) (0.847) (0.743) (0.933) (0.890) (0.723) (0.785) 14 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 14 Interaction: Flipped Classroom, perfect attendance 5.264 4.388 7.274 6.034 1.957 0.530 6.560 6.600 5.264* 4.388 (0.220) (0.334) (0.143) (0.237) (0.669) (0.913) (0.216) (0.251) (0.056) (0.130) STEM Major: Dummy -1.817 -2.091 -4.569 -4.821 0.211 -0.476 -1.093 -0.976 -1.817 -2.091 (0.649) (0.610) (0.276) (0.260) (0.961) (0.915) (0.816) (0.843) (0.467) (0.412) Interaction: Flipped Classroom, STEM 4.780 4.696 10.983 10.474 1.449 2.181 1.906 1.434 4.780 4.696 (0.612) (0.614) (0.154) (0.161) (0.889) (0.835) (0.873) (0.903) (0.405) (0.406) Hours Devoted to Course -1.026** -0.867 -1.124** -0.880 -0.960* -0.834 -0.996* -0.888 -1.026*** -0.867*** (0.037) (0.106) (0.045) (0.152) (0.074) (0.174) (0.075) (0.124) (0.001) (0.009) Interaction: Flipped Classroom, Hours Devoted to Course 0.125 0.146 -0.048 -0.037 0.244 0.277 0.178 0.199 0.125 0.146 (0.886) (0.868) (0.964) (0.972) (0.792) (0.767) (0.856) (0.840) (0.822) (0.791) Hours of Employment 0.081 0.072 0.029 0.013 0.134 0.137 0.079 0.066 0.081 0.072 (0.474) (0.526) (0.823) (0.921) (0.295) (0.283) (0.541) (0.620) (0.261) (0.318) Interaction: Flipped Classroom, Hours of Employment -0.313* -0.307* -0.231 -0.223 -0.330* -0.328* -0.377** -0.372** -0.313*** -0.307 *** (0.051) (0.055) (0.209) (0.221) (0.068) (0.071) (0.038) (0.042) (0.002) (0.002) Race (White): Dummy 2.240 4.041 0.803 1.876 2.240 (0.566) (0.407) (0.841) (0.630) (0.344) Term attempted hours 0.098 0.069 -0.003 0.228 0.098 (0.840) (0.887) (0.996) (0.683) (0.743) At Least One Parent / Guardian Has College Degree: Dummy 1.087 1.120 2.268 -0.127 1.087 (0.609) (0.637) (0.329) (0.959) (0.416) 15 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 15 Constant 30.099 23.658 47.590* 36.749 28.563 25.984 14.144 8.240 30.099* 23.658 (0.299) (0.436) (0.066) (0.198) (0.361) (0.430) (0.677) (0.813) (0.078) (0.186) No. of observations 145 144 145 144 145 144 145 144 435 432 R2 0.268 0.271 0.221 0.228 0.210 0.219 0.280 0.278 0.219 0.221 aic 1,137. 957 1,136. 309 1,170. 566 1,167. 903 1,159. 701 1,157. 336 1,186. 234 1,184. 632 3,466. 636 3,448. 946 bic 1,185. 585 1,192. 735 1,218. 193 1,224. 330 1,207. 329 1,213. 762 1,233. 862 1,241. 059 3,531. 841 3,526. 246 Note: *** p<0.01, ** p<0.05, * p<0.1 The first interesting result is that, after including all controls, the dummy for the flipped classroom is not significant overall (unless we pool the data), but some of its interactions are. That seems to corroborate with the evidence discussed in Table 4, that is, that flipping the classroom is may be disadvantageous to certain groups of students. Specifically, while prior GPA is a very significant and meaningful (given the size of the coefficient) determinant of grades, it is less important when the class is flipped. That can be seen by the coefficient of the interaction between prior GPA and the flipped classroom dummy. Although the effect of prior GPA is still positive for the flipped classroom, the coefficient is roughly reduced from 10 to 3, which is a substantial change. The interaction between flipping the classroom and hours of employment is also significant, especially for the third exam, indicating that students employed with a job performed worse in the flipped classes. Surprisingly, the coefficient of hours devoted to the course, even when significant, has a negative sign. That may be because students who were struggling more with the material had to study more hours. On the other hand, since perfect attendance, the STEM major dummy, hours devoted to course, and hours of employment are all, at least potentially, proxies for effort, it may be difficult to disentangle each of their effects in the regression, which could explain the many insignificant results. However, we experimented running with only one or two of these variables at a time and got very similar results. In addition, the inclusion of other controls in each specification (Columns 2, 4, 6, 8 and 10) does not meaningfully change the results either, except for the variable of hours devoted to course. Thus, the results seem robust. The pooled analysis (columns 10 and 11 in Table 5) did yield a significant positive result for the Flipped Classroom variable. Although this may indicate a positive overall benefit from flipping the classroom, we are only cautiously optimistic about this result since the same coefficient was insignificant across all other specifications, although this can be a result of the sample size. On the other hand, most of our other results remained similar or became stronger after pooling the data, showing the robustness of the negative impact of flipping the classroom on some student groups. One noticeable change was that the interaction between the Flipped classroom 16 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 16 variable and the Perfect Attendance variable became significant after pooling the data, indicating that students who came to class more tended to do better in the flipped classroom environment. This result seems consistent with our other findings suggesting that the students most harmed by the flipped classroom environment are those who are unable or unwilling to put forth the necessary effort to succeed. As a whole, the evidence seems to suggest that, when the classroom is flipped, each student’s past performance, proxied by prior GPA, is less relevant for grade achievement. On the other hand, a students’ current effort does seem to be a determinant, as can be seen in the results presented in Table 4 and, to the extent that students that worked more hours could not put the same amount of effort into the class, as observed in the multivariate analysis. Discussion and Conclusions The results suggest that flipping the classroom can negatively impact student academic achievement among certain student sub-groups. Students who are seemingly less able or less willing to put forth as much effort into their economics studies can suffer academically from a flipped classroom. This seems consistent with other research showing that flipping the classroom puts more responsibility on students and thus requires more effort from them to succeed (Wilson, 2013; Findlay-Thompson and Mombourquette, 2014; Lee and Lee, 2015). The overall results challenge most of the current literature on the subject, which shows broad positive improvement for students in flipped classes. One possible explanation is that flipping the classroom makes it substantially easier to shirk lecture content for students. As mentioned earlier, it seems likely that students were not given proper incentives to watch lecture videos outside of class in this study. Additionally, these videos may not have been structured effectively, leading to numerous students coming to class essentially unprepared with the thought that they would figure out what they needed to know or “wing it” as they worked through the in- class problem sets. If this is true, it is a substantial problem in a flipped classroom, because the lectures serve as the foundation of the learning process and the in-class component is meant to build upon that groundwork. It would be very difficult to build anything substantial (i.e. for substantive learning to occur) if there were no original underpinning. This differs from the traditionally formatted class where, given the right attendance policy (Broker, et al. 2014), students presumably come to class and at least gain some foundational knowledge of the material. Even if students shirked the homework in the traditionally formatted class, they would presumably at least have the foundation gained from sitting through the in-class lecture from which to work. If this were the case, then we would expect to see students with less ability or willingness to apply effort as prime candidates for doing poorly in a flipped classroom environment. In this study, most of our student breakdowns by potential “effort” categories (i.e. more hours of employment, less hours devoted to course, less use of online course materials, and less than perfect 17 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 17 attendance) showed that these students performed significantly worse in the flipped classroom environment, lending at least some credibility to this claim. Some possible methods to improve this outcome are to enhance the flipped classroom format to entice students to want to put forth more effort, especially in watching the video lectures outside of class. This might be done with shorter (10 minutes or less) and more entertaining videos, such as those produced professionally for Marginal Revolution University (http://www.mruniversity.com).9 Applying more weight on watching the videos in students’ final grades could also provide more incentive. Another possibility is that the instructor unknowingly displayed a distinct comparative advantage when teaching in the traditionally formatted classes. This would put the students in the flipped classroom environment at a disadvantage, not because the flipped format is inherently less effective, but because the instructor is personally more effective in a traditionally formatted class. This difference in teaching quality could be the result of experience or natural skills. The students in the study, regardless of class format, gave the instructor high marks for quality teaching (Table 3), so it is difficult to know how much of this teacher quality differentiation was truly present. But, overcoming even perceived differences in teacher ability with respect to different teaching formats could be important. Over time it is possible that any negative outcomes for students resulting from different classroom structures could be improved as those instructors using the flipped format (or some other format) gain more confidence and expertise with the new delivery method. More research on student outcomes in flipped classrooms and analyzing the effects of these adjustments needs to be done. References Balaban, R. A., Gilleskie, D. B., and Tran, U. 2016. “A quantitative evaluation of the flipped classroom in a large lecture principles of economics course” Journal of Economic Education (JEE), Vol. 47, No. 4, pp. 269-287. Bergman, J., Overmyer, J., and Wilie, B. 2014. “The Flipped Class: Myths vs. Reality,” The Daily Riff. . Accessed 27 July, 2016. Bishop, J. L. and Verleger, M. A. 2013. “The Flipped Classroom: A Survey of Research,” American Society for Engineering Education. . Accessed 27 July, 2016. Bonham, S. W., Deardorff, D. L., & Beichner, R. J. (2003. “Comparison of Student Performance Using Web and Paper-Based Homework in College-Level Physics,” Journal of Research in Science Teaching, Vol. 40, No. 10, pp. 1050-1071. . Accessed 27 July, 2016. 9 MRUniversity has built a large online library of free economics education videos that could easily be incorporated into economics classes like the ones in this study. 18 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 18 Broker, T. R., Milkman, M., and Raj, V. 2014. “How Do Instructor’s Attendance Policies Influence Student Achievement in Principles of Microeconomics?” Journal of Economics and Economic Education Research (JEEER), Vol. 15, No. 3, Allied Academies Affiliate Journals. Caliendo, Marco, and Sabine Kopeinig. 2008. “Some Practical Guidance for the Implementation of Propensity Score Matching.” Journal of Economic Surveys, Vol. 22, No. 1, pp. 31–72. Cohen, P. A., Ebeling, B. J., and Kulik, J. A. (1981. “A Meta-Analysis of Outcome Studies of Visual-Based Instruction,” Educational Communication and Technology, Vol. 29, No. 1, pp. 26-36 Day, J. A. and Foley, J. D., 2006. “Evaluating a Web Lecture Intervention in a Human– Computer Interaction Course,” IEEE Transactions on Education, Vol. 49, No. 4, pp.420– 431. Falconer, J. L., DeGrazia J., Medlin, J. W., and Holmberg, M. P. 2009. “Using Screencasts in Che Courses,” Chemical Engineering Education, Vol. 43, No. 4. . Accessed 27 July, 2016. Frydenberg, M. 2012. “The Flipped Classroom: It’s Got to Be Done Right”, The Huffington Post. . Accessed 27 July, 2016. Hoxie, A. B., Shepard, T., and Feyen, R. 2015. “The Flipped Classroom: A Means to Reduce Cheating?” Proceedings from 122nd Annual Conference for the American Society for Engineering Education. James, G. S. 1954. “Tests of linear hypotheses in univariate and multivariate analysis when the ratios of the population variances are unknown.” Biometrika Vol. 41, pp. 19–43. Johnson, D. W. and Johnson, R. T. 2001. “Learning Together and Alone: Overview and Meta‐ analysis,” Asia Pacific Journal of Education, Vol. 22, No. 1, pp. 95-105. Lage, M. J., Platt, G. J, and Treglia, M. 2000. “Inverting the Classroom: A Gateway to Creating an Inclusive Learning Environment,” Journal of Economic Education, Vol. 31, No. 1, pp. 30- 43. Larose, Simon and Roy, Roland. 1991. “The role of prior academic performance and nonacademic attributes in the prediction of the success of high-risk college students,” Journal of College Student Development, Vol. 32, No. 2, pp. 171-177. Lee, G. C., and Lee, P. L. 2015. “Data Structures in Flipped Classroom: Students' Effort and Preference,” Proceedings at the 3rd International Conference on Learning and Teaching in Computing and Engineering, pp. 152-155. Moravec, M., Williams, A., Aguilar-Roca, N., and O’Dowd, D. K. 2010. “Learn Before Lecture: A Strategy That Improves Learning Outcomes in a Large Introductory Biology Class,” CBE- Life Sciences Education, Vol. 9, No. 4, pp. 473–481. 19 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 19 Olitsky, N. and Cosgrove S. 2016. “The better blend? Flipping the principles of microeconomics classroom,” International Review of Economics Education, Vol. 21, Issue C, pp. 1-11. Sappington, J., Kinsey, K., and Munsayac, K. 2002. “Two Studies of Reading Compliance among College Students,” Teaching of Psychology, Vol. 29, No. 4, pp. 272-274. Stone, B. B. 2012. “Flip Your Classroom to Increase Active Learning and Student Engagement,” Proceedings from 28th Annual Conference on Distance Teaching & Learning. . Accessed 27 July, 2016. Toto, R. and Nguyen, H. 2009. “Flipping the Work Design in an Industrial Engineering Course,” 39th IEEE Frontiers in Education Conference, pp. 1–4. Tucker, B. 2012. “The Flipped Classroom,” EducationNext, Vol. 12, No. 1. . Accessed 27 July, 2016. Wilson, S. G. 2013. “The Flipped Class: A Method to Address the Challenges of an Undergraduate Statistics Course,” Teaching of Psychology. Vol. 40, No. 3, pp. 193-199. 20 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 20 Appendix 1 21 |J O U R N A L F O R E C O N O M I C E D U C A T O R S , 1 8 ( 2 ) , 2 0 1 8 21