Regression Results Replicating Ballard and Johnson: 6 JOURNAL FOR ECONOMIC EDUCATORS, 13(1), 2013 PREDICTING STUDENT PERFORMANCE USING ONLINE ONE-MINUTE PAPERS Lee E. Erickson and Patricia A. Erickson 1 Abstract One-minute papers are often used to encourage students to think and write briefly about their own learning, because teachers believe that metacognition and writing help students to learn. The proportion of online one-minute papers that students submit, however, has not previously been used to explain student achievement in economics. This paper shows that the completion rate is a very significant predictor of student performance after controlling for other variables already noted in the literature. Removing small observation categories does not affect the significance or stability of key regression coefficients. Students who complete online one- minute papers more regularly also perform better in Principles of Microeconomics. Key Words: One-minute papers; online surveys; predicting student performance JEL Classifications: A20, A22 Introduction In Economics, the learning of later topics often builds on one’s understanding of earlier concepts. Consistent effort is more effective than cramming, but students tend to procrastinate. Too often students come to class unprepared, sit as passive observers and postpone focused effort until immediately before points of major accountability. One-minute papers encourage students to seek deeper understanding more promptly by engaging in metacognition and writing. The traditional one-minute paper asks students a few open-ended questions at the end of class (Angelo and Cross 1993): What was the most important thing that you learned today? What important question remains unanswered? Mosteller (1989) reported getting better information from students when he asked them “What was the muddiest point in the lecture?” Chizmar and Ostrosky (1998) and Vredenburg (2004) implemented one-minute papers online. Our students had weekly opportunities to complete online surveys through the University’s course management system. These online one-minute papers were available for a limited time after the last class prior to in-class review times for quizzes and tests. Our anonymous surveys asked students the following questions: Did you do the assigned reading before class each day? What is clearest to you? What is least clear? Where are you having trouble? Do you have any other comments, suggestions or questions? Student responses to these online surveys then directed the in-class reviews for quizzes and tests. There are advantages to doing these one-minute papers anonymously, online, outside of class. Responses cannot be traced to individual students, so some may be more honest. Online responses are more legible and may be longer, because some students find typing easier than handwriting. Doing self-appraisals online reduces the dominance of a few vocal students, 1 Lee E. Erickson is a Professor of Economics (leerickson@taylor.edu) and Patricia A. Erickson is an adjunct Instructor of Mathematics (pterickson@taylor.edu); both are at Taylor University, 236 West Reade Avenue, Upland, Indiana 46989. We thank Peter Kennedy, Kenneth Constantine and anonymous referees for helpful comments. mailto:leerickson@taylor.edu mailto:pterickson@taylor.edu 7 JOURNAL FOR ECONOMIC EDUCATORS, 13(1), 2013 because those who are shy in class are less so online (Vredenburg 2004). Some responses may be more thoughtful than if they were done at the end of a class, and the self-appraisal does not require class time. Anonymous responses also have a disadvantage. Students who reflect and write very little receive the same credit as students who reflect and write much more. For example, a student could get credit for doing the assignment by writing “The rain in Spain stays mainly in the plain,” although this may not be relevant or even true. In our experience few students actually completed the surveys without any reflection at all. Doing online one-minute papers regularly requires students to consistently remember to do them. Students were regularly reminded about the survey during class, but recall was needed at the appropriate time after class. Students are more likely to remember the assignment when they are studying for tests and quizzes. A student whose habit is to begin her study for tests and quizzes further ahead of these accountability points is more likely to complete the online one- minute papers during the designated time. Those who cram for tests and quizzes are more likely to miss the deadlines for these assignments. Other authors have compared the performance of students in course sections that completed one-minute papers with those in sections that did not. Chizmar and Ostrosky (1998) found that one-minute papers increased students’ economic knowledge as measured by the Test of Understanding in College Economics (TUCE) after controlling for semester GPA excluding the economics grade. Das (2010) found that students who wrote one-minute papers performed better on a post-test, after controlling for GPA and gender, than students who did not. Stowe (2010) reported that students who completed one-minute papers had higher course grades than those who did not complete them, after controlling for cumulative GPA, SAT scores, absences and gender, but the significance of the results was sensitive to the model specifications. Data and Methodology Rather than using experimental and control sections of a course as others have done, we gave all students the same opportunities to do one-minute papers and noted the differences in student completion rates for this assignment. More specifically, our “self-appraisal percent” variable is the proportion of online one-minute papers that students completed. While this one-minute paper completion rate has not previously been used to predict student performance, many other variables have been studied. Cumulative GPA has been shown to be an important predictor of student success in economics (Park and Kerr 1990; Durden and Ellis 1995; Didia and Hasnat 1998; Ballard and Johnson 2004; Krohn and O’Connor 2005, and Grove, Wasserman and Grodner 2006). Math skills are widely reported to be important predictors of student achievement in economics courses (Anderson, Benjamin, and Fuss 1994; Durden and Ellis 1995; Ballard and Johnson 2004; and Pozo and Stull 2006). Men have been found to outperform women, especially on multiple choice tests (Lumsden and Scott 1987; Anderson, Benjamin, and Fuss 1994; Ballard and Johnson 2004; and Krohn and O’Connor 2005), but others found no significant gender differences (Williams, Waldauer, and Duggal 1992; Lawson 1994; and Swope and Schmitt 2006). Our data represent ten semesters of Principles of Microeconomics courses taught by the same instructor. The fraction of one-minute papers that students completed and their performance on each weekly quiz and each unit test were recorded. Test scores were about 80% of the total possible points and quiz scores accounted for the remainder. 8 JOURNAL FOR ECONOMIC EDUCATORS, 13(1), 2013 Students self-reported gender, ethnic group, class standing, whether Principles of Macroeconomics had been completed, whether a calculus course 2 had been completed, whether the University’s remedial math course had been required 3 , whether the Issues in Economics course 4 had been taken, and whether Principles of Microeconomics was required for the student’s major. Dummy variables for the different semesters, ethnic groups and class levels were created. The total sample size was 476 over this five-year period. Incomplete data for a few students reduced the sample size to 470. To avoid including the Principles of Microeconomics course grade in the GPA variable, cumulative GPA at the beginning of the semester was used. GPA, ACT and SAT scores were retrieved from student records. A few students were excluded from the sample, because as freshmen or transfer students they did not have a prior GPA at our University. SAT scores were converted to ACT scores using concordances (Dorans 1999; ACT 1998). The percentage of points earned could not be used directly as our dependent variable, because it includes the extra credit meant to motivate completion of the online surveys. These extra credit points would have been about 3% of the total points possible, if all of the surveys had been done. The last quiz in each semester was a bonus quiz. So we delete the extra credit points for the self-appraisals and the bonus quiz to construct a new dependent variable. Our “assessment percent” variable is the proportion of test and quiz questions that were answered correctly throughout the semester. This removes the influence of the variability in the length and number of quizzes and tests from one semester to another and measures the course grade without the influence of any extra credit. Descriptive statistics are given in Table 1. Because the self-appraisals were done online outside of class and were not required, the average student completed only 60% of them. Almost all of the students were Caucasian, about two thirds were male and most had taken a calculus course. Ordinary least squares regression was used to estimate the linear relationship between assessment percent and the independent variables. A linear model is appropriate, because the residuals appear to be normally distributed. Results The regression results for the complete data set are given in Table 2. The variables in the full model explain over 60% of the variation in assessment percent according to the adjusted R 2 . There is not a high correlation among the quantitative predictor variables. The largest Pearson r correlation coefficient for any pair of independent variables was 0.59 for the relationship between GPA and ACT composite. 2 Credit for a calculus course could have been earned by passing the AP® exam or by passing a college course. The Advanced Placement Program® (AP®) is administered by the College Board and allows high school students to earn college credit by taking AP® courses and passing AP® exams given at the end of these courses. 3 All students at our University are required to demonstrate mathematics proficiency. They may do this by scoring sufficiently high on the SAT or ACT Mathematics tests or by passing a Mathematics Department proficiency exam covering basic math skills and algebra. Those who do not pass the mathematics proficiency exam are required to complete a remedial math course and re-take the exam until they do pass it. 4 “Issues in Economics” is a general education course intended for students who do not plan to take other economics courses. 9 JOURNAL FOR ECONOMIC EDUCATORS, 13(1), 2013 Table 1: Descriptive Statistics Variable Percent in Category Mean Standard Deviation Assessment percent 0.8 0.1 Self-appraisal percent 0.6 0.3 GPA at the beginning of the semester 3.2 0.5 ACT Composite 25.6 3.9 Basic math quiz score 8.1 1.6 Male 67.4% Principles of Macroeconomics course completed 3.6% Calculus course completed 54.8% Remedial math course completed 8.0% African or African American 3.2% Asian or Asian American 2.1% Caucasian 93.3% Hispanic 1.3% Spring 2004 11.3% Fall 2004 9.0% Spring 2005 10.5% Fall 2005 10.1% Spring 2006 10.3% Fall 2006 10.3% Spring 2007 9.7% Fall 2007 10.3% Spring 2008 9.9% Fall 2008 8.6% Freshman 14.3% Sophomore 63.5% Junior 17.9% Senior 4.4% Issues in Economics course completed 1.5% Repeating Principles of Microeconomics course 3.6% Principles of Microeconomics course required 83.2% 10 JOURNAL FOR ECONOMIC EDUCATORS, 13(1), 2013 Table 2: Full and Reduced Models for the Complete Data Set Full Model Reduced Model Coefficient t-statistic p-value Coefficient t-statistic p-value Constant 0.004 0.10 0.92 0.028 0.79 0.43 GPA 0.133 12.73 0.00 0.131 12.68 0.00 Male 0.042 4.68 0.00 0.044 4.95 0.00 Self-appraisal percent 0.078 4.23 0.00 0.081 4.38 0.00 ACT Composite 0.006 3.87 0.00 0.005 3.60 0.00 Principles of Macroeconomics 0.054 2.37 0.02 0.070 3.25 0.00 Calculus course 0.030 3.38 0.00 0.028 3.13 0.00 Basic math quiz 0.008 2.54 0.01 0.008 2.63 0.01 Remedial math course -0.037 -2.36 0.02 -0.041 -2.62 0.01 African or African American 0.062 2.79 0.01 0.062 2.76 0.01 Asian or Asian American 0.033 1.16 0.25 0.033 1.16 0.25 Hispanic 0.028 0.78 0.44 0.025 0.72 0.47 Spring 04 0.057 3.22 0.00 0.053 3.01 0.00 Fall 04 0.029 1.56 0.12 0.028 1.52 0.13 Spring 05 0.052 2.87 0.00 0.045 2.56 0.01 Fall 05 0.031 1.74 0.08 0.033 1.80 0.07 Spring 06 0.067 3.68 0.00 0.060 3.41 0.00 Fall 06 0.021 1.19 0.23 0.021 1.15 0.25 Spring 07 0.014 0.74 0.46 0.013 0.70 0.48 Fall 07 0.020 1.12 0.26 0.022 1.20 0.23 Spring 08 0.046 2.56 0.01 0.044 2.40 0.02 Freshmen -0.016 -1.28 0.20 Junior 0.016 1.48 0.14 Senior 0.013 0.66 0.51 Issues in Economics 0.043 1.29 0.20 Repeating Principles of Microeconomics 0.016 0.72 0.47 Required 0.009 0.87 0.38 Adjusted R 2 60.7% 60.4% F (p-value) 28.87 (0.00) 36.76 (0.00) n 470 470 Partial F (p-value) 1.20 (0.30) 11 JOURNAL FOR ECONOMIC EDUCATORS, 13(1), 2013 Non-significant variables can inflate the apparent percent of the variation in the dependent variable explained by the regression. We remove from the full model those variables with p-values more than 0.05. Groups of indicator variables must be removed or kept together in reducing the model, because they function together to describe multilevel categories. So Freshman, Junior and Senior were removed together, because they jointly describe class level, and none of them was significant at the 0.05 level. Although Asian or Asian American and Hispanic were not significant predictors, they remain in the model because African or African American was significant. Similarly, all of the indicator variables for semester were kept in the reduced model, because some of them were significant predictors. A partial F-test shows that the reduced model is not significantly worse at predicting assessment percent than the full model. The more streamlined model explains almost as much of the variation in the dependent variable as the full model does. Self-appraisal percent is more significant than any of the other explanatory variables except for cumulative GPA and gender. The fact that men performed better than women may be because all of the test questions and most of the quiz questions were multiple choice. ACT composite, completing Principles of Macroeconomics prior to Principles of Microeconomics, completing a calculus course, Ballard and Johnson’s (2004) basic math quiz, and needing to take our University’s remedial math course are also very significant predictors in the reduced model. If the self-appraisal percent were increased by ten percentage points, assessment percent would be predicted to increase by about 0.8 percentage points, according to the reduced model for the complete data set. The “African or African American” variable is positive and very significant, however only 3% of the students are in this category. Other data categories also represent very small percentages of the total observations. Less than five percent of the students are in each of the following categories: African or African American, Asian or Asian American, Hispanic, Seniors, Issues in Economics completers, Principles of Microeconomics repeaters and Principles of Macroeconomics completers. In the spirit of sensitivity testing, we homogenize the data by removing observations in these low frequency categories. The results for the homogenized data are shown in Table 3. The coefficients for the most significant variables in the full and reduced models for the homogenized data (shown in Table 3) are very similar to those for the full and reduced models for the complete data set (shown in Table 2). Since these coefficients appear stable, the low frequency categories do not appreciably distort the reduced model for the complete data set. Our self-appraisal variable remains a very significant predictor of student achievement. Conclusion Students who complete online one-minute papers more regularly also perform better in Principles of Microeconomics. Self-appraisal percent is the most significant explanatory variable after cumulative GPA and gender. These results persist even after small categories are removed. 12 JOURNAL FOR ECONOMIC EDUCATORS, 13(1), 2013 Table 3: Full and Reduced Models for the Homogenized Data Set This does not tell us whether frequent completion of one-minute papers improves performance or whether doing the assignment is associated with an otherwise omitted student characteristic. It could measure prompt, consistent study time, that is, a lack of procrastination. To complete one-minute papers more frequently, students need to remember the task in time to do it, and they are more likely to do this if they are studying further ahead of tests and quizzes. Active student engagement outside of class may increase self-appraisal percent. The one-minute paper assignment, however, may also encourage timely study. So even if frequent completion of one-minute papers is associated with a lack of procrastination, we do not know the direction of causality. Also doing the one-minute papers involves both reflection and writing. Further research is needed to distinguish the independent effects on student performance of procrastination, metacognition and writing. Full Model Reduced Model Coefficient t-statistic p-value Coefficient t-statistic p-value Constant -0.035 -0.85 0.40 -0.024 -0.59 0.55 GPA 0.140 12.31 0.00 0.139 12.26 0.00 Male 0.046 4.76 0.00 0.046 4.73 0.00 Self-appraisal percent 0.082 4.13 0.00 0.083 4.18 0.00 Calculus course 0.037 3.74 0.00 0.037 3.71 0.00 ACT Composite 0.006 3.63 0.00 0.006 3.64 0.00 Remedial math course -0.042 -2.57 0.01 -0.042 -2.54 0.01 Basic math quiz 0.006 1.94 0.05 0.007 2.05 0.04 Spring 04 0.076 3.96 0.00 0.077 4.00 0.00 Fall 04 0.032 1.59 0.11 0.032 1.62 0.11 Spring 05 0.075 3.78 0.00 0.075 3.79 0.00 Fall 05 0.050 2.58 0.01 0.050 2.59 0.01 Spring 06 0.085 4.18 0.00 0.084 4.14 0.00 Fall 06 0.028 1.46 0.14 0.028 1.46 0.15 Spring 07 0.031 1.46 0.15 0.029 1.37 0.17 Fall 07 0.029 1.48 0.14 0.029 1.48 0.14 Spring 08 0.050 2.57 0.01 0.049 2.54 0.01 Freshmen -0.026 -1.93 0.05 -0.027 -2.01 0.05 Junior 0.017 1.41 0.16 0.017 1.49 0.14 Required 0.013 1.09 0.28 Adjusted R 2 63.5% 63.5% F (p-value) 36.78 (0.00) 38.73 (0.00) n 392 392 Partial F (p-value) 1.18 (0.28) 13 JOURNAL FOR ECONOMIC EDUCATORS, 13(1), 2013 References ACT, Inc. 1998. “Concordance Between SAT I Verbal Score and ACT English Score.” laregentsarchive.com. . Accessed 27 April 2013. Anderson, B., H. Benjamin and M.A. Fuss. 1994. “The Determinants of Success in University Introductory Economics Courses.” Journal of Economic Education, 25(2): 99-119. Angelo, T.A. and K.P. Cross. 1993. Classroom Assessment Techniques: A Handbook for College Teachers, 2 nd Edition. Hoboken: Jossey-Bass Publishers. Ballard, C.L. and M.F. Johnson. 2004. “Basic Math Skills and Performance in an Introductory Economics Class.” Journal of Economic Education, 35(1): 3-23. Chizmar, J.F. and A.L. Ostrosky. 1998. “The One-minute Paper: Some Empirical Findings.” Journal of Economic Education, 29(1): 1-8. Das, Amaresh. 2010. “Econometric Assessment of ‘One Minute’ Paper as a Pedagogic Tool.” International Education Studies, 3(1): 17-22. Didia, D. and B. Hasnat. 1998. “The Determinants of Performance in the University Introductory Finance Course.” Financial Practice and Education 8(1): 102-107. Dorans, N.J. 1999. “Correspondences between ACT and SAT I scores. College Board Report No. 99-1, ETS RR No. 99-2.” ets.org. . Accessed 27 April 2013. Durden, G.C. and L.V. Ellis. 1995. “The Effects of Attendance on Student Learning in Principles of Economics.” American Economic Review, 85(2): 343-346. Grove, W.A., T. Wasserman and A. Grodner. 2006. “Choosing a Proxy for Academic Aptitude.” Journal of Economic Education, 37(2): 131-147. Krohn, G. and C. O’Connor. 2005. “Student Effort and Performance Over the Semester.” Journal of Economic Education, 36(1): 3-28. Lawson, L. 1994. “The Role of Attitude in Learning Economics: Race and Gender Differences.” Journal of Economics and Finance, 18(2): 139-151. Lumsden, K. and A. Scott. 1987. “The Economics Student Reexamined: Male-Female Differences in Comprehension.” Journal of Economic Education, 18(4): 365-375. Mosteller, F. 1989. “The ‘Muddiest Point in the Lecture’ as a Feedback Device.” On Teaching and Learning, 3: 10-21. Park, K.H. and P.M. Kerr. 1990. “Determinants of Academic Performance: A Multinomial Logit Approach.” Journal of Economic Education, 21(2): 101-111. Pozo, S. and C.A. Stull. 2006. “Requiring a Math Skills Unit: Results of a Randomized Experiment.” American Economic Review, 96(2): 437-441. Stowe, K., 2010. “A Quick Argument For Active Learning: The Effectiveness of One-minute Papers.” Journal for Economic Educators, 10(1): 33-39. Swope, K.J. and P.M. Schmitt. 2006. “The Performance of Economics Graduates Over the Entire Curriculum: The Determinants of Success.” Journal of Economic Education, 37(4): 387- 394. Vredenburg, D. 2004. “Using On-line Discussion Forums for Minute Papers.” The Teaching Professor, 18(10): 6. Williams, M., C. Waldauer, and V. Duggal. 1992. “Gender Differences in Economic Knowledge: An Extension of the Analysis.” Journal of Economic Education, 23(3): 219-231. http://www.laregentsarchive.com/pdfs/Planning/MP%20SAT%20to%20ACT%20Concordance%20Table%20for%20English.pdf http://www.laregentsarchive.com/pdfs/Planning/MP%20SAT%20to%20ACT%20Concordance%20Table%20for%20English.pdf http://www.ets.org/Media/Research/pdf/RR-99-02-Dorans.pdf http://www.ets.org/Media/Research/pdf/RR-99-02-Dorans.pdf