19 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 THE IMPACT OF QUESTION ORDER ON MULTIPLE CHOICE EXAMS ON STUDENT PERFORMANCE IN AN UNCONVENTIONAL INTRODUCTORY ECONOMICS COURSE Paul Kagundu and Glenwood Ross1 Abstract We investigate the effect of question order on multiple-choice exams on students’ performance in an unconventional introductory economics course. The course is an introduction to the global economy and comprises elements of principles of economics, introductory international trade and introductory international finance. The tests in two sections of the course were administered in four versions. On one of the versions, multiple-choice questions are ordered according to the order in which course material was offered, while questions on the other versions are randomly scrambled. Our empirical analysis reveals no statistically significant effect of question order on students’ grades. Key Words: question order, multiple-choice exams, students’ performance, unconventional economics course JEL Classification: A22 Introduction In an effort to reduce cheating on multiple-choice tests, instructors often use several different versions of the same test. In many cases, a sequenced version of the test is accompanied by several randomly scrambled versions of the test. The questions on the sequenced test are presented in a logical fashion, based on the order in which the course material was offered. As a result, students may glean cues and prompts from a prior question or set of questions in a logical sequence to lead them to a correct response. If this is the case, an unintended consequence of this effort to minimize cheating could be the introduction of a bias in favor of those students who receive the sequenced version of the test versus those who receive the scrambled versions. A number of studies have taken a look at the importance of question order on multiple- choice tests in introductory economics courses. The courses examined in these studies have typically been principles of economics, principles of macroeconomics, and principles of microeconomics courses in which material is generally presented in a logical building block manner. This study attempts to extend the work of previous studies by examining the impact of 1 P. Kagundu, Senior Lecturer of Economics, Department of Economics, Penn State University; G. Ross, Clinical Associate Professor of Economics, Department of Economics, Georgia State University. 19 20 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 question order on multiple-choice tests in a Global Economy course, an unconventional introductory economics class.2 The class is unconventional in the sense that (1) it is a hybrid course comprised of roughly equal elements of economic principles, introductory international trade, and introductory international finance; and (2) not all the material is presented in a typical building block sequential manner of most introductory economics courses. The content in the first third of the course consists of standalone topics whose understanding is not dependent on the mastery of any other topic in the section. This is where a select group of basic economic concepts that are needed in the remainder of the course are introduced. These concepts include definitions of economics, marginal analysis, opportunity cost, supply and demand, gross domestic product (GDP), inflation, and the production possibilities frontier (PPF). Each of these topics is independent of the others and can be presented in any order. The remainder of the course is much more structured in that the topics covered in international trade and international finance build upon each other. One can conceive of the first part of the course as a collection of random topics, while the last two-thirds of the course is more logically sequenced. This dichotomy allows us to relate the importance of question order to the nature of course content. A priori, it would seem that question order on multiple-choice tests should matter more for logically ordered and related course content than for course content that is randomly ordered and unconnected. The next section presents a brief review of the literature on question order and student performance on multiple-choice tests, highlighting our contribution to this literature. This is followed by the empirical framework, the data and a discussion of simple inferences. Then we present and discuss the regression results for the impact of question order on student performance, followed by a conclusion. Literature Review Several studies examine the impact of the question order of multiple-choice exams on student performance in introductory economics courses. To date, however, no general consensus has emerged. Some studies conclude that question order may indeed matter. In one of the first studies to investigate this topic, Taub & Bell (1975) developed a regression model that included a dummy variable to indicate whether a multiple-choice test was randomly ordered or sequentially ordered. This variable proved to be significant, indicating that students who completed randomly ordered tests scored about 1.4 points lower than students who completed tests on which the questions followed the order of topics in the textbook and lectures. Carlson & Ostrosky (1992) also concluded that question order might matter. They analyzed four exams in a large microeconomics principles class in which each exam was administered in two versions — one in sequential content-order and one in random order. When each exam was analyzed individually, Carlson and Ostrosky (1992) were unable to reject the null hypothesis that the means and the variances on the sequenced and random versions of the exam were equal. When the data were pooled, however, the null was rejected. This, combined with 2 The Global Economy Course is part of the core curriculum at the university. It is designed for the economic novice and, as such, has no prerequisites. Most students who elect to take the Global Economy course are not economics majors. Since this class serves as a “gateway course” to the discipline, however, a number of students subsequently decide to become economics majors. 20 21 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 the fact that the mean score of the content-ordered exam exceeded the mean score of the scrambled exam in three of the four cases, led them to conclude that the mean level of performance may be higher on the content-ordered exam than the scrambled exam. Doerner & Calhoun (2009) found mixed results. They conducted an experiment on three large introductory economics classes—one principles of microeconomics class and two principles of macroeconomics classes. The data were stratified between male and female students and three versions of a multiple-choice final exam were administered. One version was randomly ordered, a second was sequentially ordered, and the third version had questions that were ordered in a reverse sequential format. Their results indicated that females benefited from both sequentially ordered and reverse sequentially ordered exams. Question order did not matter for males. Still other studies claim that question order doesn’t matter. Bresnock, Graves & White (1989) examined the results of three multiple-choice tests given to a large section of undergraduate principles of economics class. They concluded that question order didn’t matter, but that the pattern or distribution of correct answer responses on the multiple-choice tests did impact the degree of test difficulty. Gohmann & Spector (1989) randomly distributed sequenced and randomly ordered multiple-choice final exams to a large principles of macroeconomics class. In several specifications of linear regressions to determine whether exam performance could be attributed to the scrambling of exam questions, they found that question order had no significant effect on exam scores. More recently, Sue (2009) focused her analysis on a small class setting. Heretofore, most of the other studies that investigated the role of question order on multiple-choice exams in economics courses focused on large classes. She analyzed data for three sections of principles of microeconomics and three sections of principles of macroeconomics. The average class size was less than 30 students. In regression analysis she was unable to reject the null hypothesis that “scrambling the content order of questions in a multiple choice test does not affect student performance on the test.” We contribute to this literature by examining the impact of question order on student’s performance in an unconventional introductory economics course on the global economy. Empirical Framework To estimate the effect of question order on multiple-choice tests on students’ grades we compare the average performance of a random group of students who took a version of the test with questions ordered in the order in which the course content was covered in class to that of a control group with scrambled versions of the test. We refer to the version of the test that is ordered consistent with the lecture coverage of the course content as the “sequenced” version. The other versions of the test are, together, referred to as the “scrambled” version. Students taking the “sequenced” version of the test are our treatment group, while those taking the “scrambled” version are our control group. As such, we estimate a linear regression equation of the following form. 𝐺𝐺𝑖𝑖𝑖𝑖 = 𝛼𝛼0 + 𝛼𝛼1𝑉𝑉𝑖𝑖𝑖𝑖 + 𝛼𝛼2𝑋𝑋𝑖𝑖𝑖𝑖 + 𝜀𝜀𝑖𝑖𝑖𝑖 (1) The left-hand side variable, Gij, represents student i’s grade on the multiple-choice section of test j (j = 1, 2, 3, 4). The variable Vij represents the version of the test (sequenced or scrambled) and Xij represents a number of control variables including proxies for the student’s academic ability and/or prior knowledge of the subject. The last term in equation (1) denotes the idiosyncratic random error term. The major variable of interest is Vij, defined as a binary dummy variable 21 22 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 equal to 1 if student i took the sequenced version on test j, and 0 otherwise. Therefore, a positive and statistically significant estimate of 𝛼𝛼1 suggests a bias in favor of students taking the sequenced version. In other words, student performance benefits from ordering questions consistent with class coverage of the course material. The Data The data were collected in two sections of an introductory course on the Global Economy at a large public university in the southeastern United States. The data were collected in the Fall 2011 semester. The two sections (015 and 035) were taught by the same instructor. Table 1 presents descriptive statistics of key variables by section, test, and version. Students in each section took a total of four tests during the course of the school semester. The last of the four tests is the comprehensive final exam. Table 1: Descriptive Statistics by course section, exam, and test version Section Test Test Version Obs Mean Standard Deviation Econ2100-015 1 1 Sequenced Scrambled 16 46 77.06 69.09 11.23 12.64 2 2 Sequenced Scrambled 16 46 81.31 76.65 16.37 16.62 3 3 Sequenced Scrambled 15 44 62.20 69.50 10.88 15.09 4 (Final) 4 (Final) Sequenced Scrambled 14 46 67.21 70.54 9.96 13.07 Econ2100-035 1 1 Sequenced Scrambled 17 54 70.94 71.11 14.36 13.63 2 2 Sequenced Scrambled 17 52 79.88 77.40 13.66 14.31 3 3 Sequenced Scrambled 19 46 68.58 65.35 12.30 16.09 4 (Final) 4 (Final) Sequenced Scrambled 18 49 65.39 69.90 12.61 12.65 Tests consisted of a multiple-choice section and a problem-type question. Only students’ scores on the multiple-choice part of the test are used in our estimations. Each test was composed of four versions, one “sequenced” and three “scrambled”. On each test, about one-quarter of students taking the test had the “sequenced” version. In Table 1, there does not seem to be a statistically significant difference in average scores between those students who took the “sequenced” version and those that took the “scrambled” versions. 22 23 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 In fact, average scores are higher for the “scrambled” versions in two of the four tests. Formal tests for the difference in mean scores for the two sub-samples are presented below. Note that the distribution of test versions in the two classes was random. That is, the test version taken by student i depended entirely on where in the classroom the student sat. If students maintained their seating positions in the classroom throughout the semester, it is conceivable that the same students received the “sequenced” version of the test in all or most of the 4 tests. This does not bias our estimates if students initially randomly self-selected their seating positions. But, if students’ self-selected seating positions are correlated with ability, assignment of test versions based on seating positions biases our results. To exclude this possibility, Table 2 presents sample probabilities associated with student i receiving a “sequenced” version of the test on more than one test. These probabilities are compared with joint probabilities under a purely random assignment. For example, only 5 students (row 1, columns 1 & 2) had the “sequenced” version of the test on both the first two tests. This translates into a probability of 0.0189 (row 1, column 4) that student i had the sequenced version on both test 1 and test 2 compared to the joint probability of the same event of 0.0352 (row 1, column 5). Going down to the bottom of the table, we show the probabilities are even small that student i got the “sequenced” version on more than two exams. We are, therefore, confident that our regression estimates are not biased by the assignment of the test versions to students. Table 2: Distribution of Test Versions (Sequenced versus Scrambled) Tests Number of Students with Sequenced Version Total Number of Students Probability Joint Probability (purely random distribution of exams) 1 and 2 only 5 264 0.0189 0.0352 1 and 3 only 7 257 0.0272 0.0352 1 and 4 only 10 257 0.0389 0.0352 2 and 3 only 8 255 0.0314 0.0352 2 and 4 only 9 255 0.0353 0.0352 3 and 4 only 9 248 0.0363 0.0352 1, 2 and 3 only 1 388 0.0026 0.0117 2, 3, and 4 only 2 379 0.0053 0.0117 1, 3, and 4 only 2 381 0.0052 0.0117 1, 2, 3, and 4 1 515 0.0020 0.0039 Simple Inference Here we use sample statistics to test for possible differences in population parameters between the “sequenced” and the “scrambled” versions of the tests. In particular, we conduct an F-test for differences in variances as well as a t-test for differences in mean scores. These tests assume that the two samples (“sequenced” and “scrambled”) are independent and drawn from normally distributed populations with equal variances. We begin with a simple visual test in the 23 24 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 form of a boxplot for grades in the two samples. The boxplot provides an informal test of the independent samples assumption. Figure 1: Grade on the Multiple Choice Portions of the Tests From Figure 1, the two samples do not seem to differ much in regard to students’ grades. Further, the distributions for both groups seem symmetric enough to justify a t-test for difference in mean scores between the two populations. Nevertheless, we check for normality using the normal quantile plot of residuals of “grade” (Gij). The appropriate residuals here are computed as the difference between the observed grade on the multiple-choice portion of the tests (Gij) and the group-specific mean grade (group ≡ “sequenced”, “scrambled”). The normality assumption is satisfied if the quantiles of the residuals are linearly related to the quantiles of the normal distribution. Figure 2 below presents the normal quantile plots for the “sequenced” sub-sample, and the “scrambled” sub-sample. Figure 2: Normal Quantile Plots 20 40 60 80 10 0 0 1 G ra de o n th e m ul tip le c ho ice p or tio n of th e te st Graphs by Question Sequencing (1=sequenced, 0=Scrambled) -4 0 -2 0 0 20 40 R es id ua ls o f t -t es t f or g ra de -40 -20 0 20 40 Inverse Normal Normal q-q plot - Sequenced (non-scrambled) version -4 0 -2 0 0 2 0 4 0 R e si d u a ls o f t- te st f o r g ra d e -40 -20 0 20 40 Inverse Normal Normal q-q plot - Scrambled Versions 24 25 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 From Figure 2, we conclude that the points on the normal quantile plots (representing residuals of “grade”) are very close to the straight line. Therefore, it is very plausible that both samples (“sequenced” and “scrambled”) are from normally distributed populations. We now proceed to conduct F-tests for differences in variances and independent samples t-tests. Difference in variances We test the null hypothesis that the two samples are drawn from populations with equal variances against the alternative that population variances are not equal. That is, 𝜎𝜎12 − 𝜎𝜎22 = 0 Table 3: F-tests for difference in Variances (Sequenced versus Scrambled) by Course Section and Test (a) Section 015 Version n f DF F-critical value Pr(F>f) Test1 Scrambled 46 1.2668 45, 15 2.5650 0.6357 Sequenced 16 Test 2 Scrambled 46 1.0312 45, 15 2.5650 0.9989 Sequenced 16 Test 3 Scrambled 44 1.9253 43, 14 2.6618 0.1835 Sequenced 15 Final Scrambled 46 1.7242 45, 13 2.7601 0.2861 Sequenced 14 (b) Section 035 Version n f DF F-critical value Pr(F>f) Test 1 Scrambled 54 0.9 53, 16 2.4635 0.7386 Sequenced 17 Test 2 Scrambled 52 1.0986 51, 16 2.3995 0.8751 Sequenced 17 Test 3 Scrambled 46 1.7104 45, 18 2.3635 0.216 Sequenced 19 Final Scrambled 49 1.0067 48, 17 2.4115 0.9623 Sequenced 18 25 26 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 against the alternative that 𝜎𝜎12 − 𝜎𝜎22 ≠ 0.3 In this case, 𝜎𝜎12 and 𝜎𝜎22 are the population variances for the “scrambled” versions of the tests and the “sequenced” version of the tests respectively. In all the tests reported in Table 3, we fail to reject the null of equal population variances. Next, we proceed to test for differences in population means based on the assumption of normality and equal population variance. Difference in Means The test results presented in Table 4 show no statistical evidence of a bias in favor of students who took the “sequenced” version. The null hypothesis for the t-test is that there is no statistical difference in mean scores across versions: 𝜇𝜇1 − 𝜇𝜇2 = 0, where 𝜇𝜇1 and 𝜇𝜇2 denote population means for the “scrambled” and the “sequenced” versions respectively. On the other hand, the alternative hypothesis is that mean scores on the “sequenced” version are higher than those on the scrambled versions (𝜇𝜇1 − 𝜇𝜇2 < 0). The t-tests fail to reject the null hypothesis in all cases except one – test 1 in section 015. Overall, the t-tests suggest that students that took the scrambled versions of the tests performed at least as well as those that took the “sequenced” version. Regression Analysis Tests based on mean scores and sample variances may not fully exclude the likelihood that the order of questions on the test has an effect on students’ performance. A linear regression analysis that controls for other possible determinants of students’ performance is a more effective way to tease out the impact of any given factor on students’ grades while holding other factors constant. We estimate several regression specifications based on equation (1). Our baseline specification takes the following form: 𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝑖𝑖𝑖𝑖 = 𝛼𝛼0 + 𝛼𝛼1𝐴𝐴𝐴𝐴𝐴𝐴𝐺𝐺𝐴𝐴𝐺𝐺𝑖𝑖𝑖𝑖 + 𝛼𝛼2𝑆𝑆𝐺𝐺𝑆𝑆𝑖𝑖𝑖𝑖 + 𝛼𝛼3𝑃𝑃𝐺𝐺𝑃𝑃𝑃𝑃𝐺𝐺𝐺𝐺𝑃𝑃𝑃𝑃𝐴𝐴𝑖𝑖 + 𝛼𝛼4𝑆𝑆𝐺𝐺𝑃𝑃𝐴𝐴𝑃𝑃𝑃𝑃𝐴𝐴015𝑖𝑖 + 𝜀𝜀𝑖𝑖𝑖𝑖 (2) Again, Gradeij denotes student i’s grade on the multiple choice section of test j. Attendij denotes the proportion of class sessions attended by student i in which material covered on test j was covered in class lectures. In other words, student i’s attendance during the first weeks of the semester (before test 1) is matched with the student’s grade on the multiple-choice section of test 1.4 Seqij is a dummy variable equal to 1 if student i took the “sequenced” version of test j, and 0 elsewhere. This is our variable of interest. Equation (2) also includes a dummy variable, Prioreconi, to control for the effect of prior knowledge of the subject on performance. This dummy variable is equal to 1 if student i had taken any college-level economics courses before enrolling for the Global Economy course. 3 The test statistic is given by the ratio of the two sample variances �𝑆𝑆1 2 𝑆𝑆2 2�. S1 2 and S22 are the sample variances. We reject the null hypothesis if 𝑆𝑆1 2 𝑆𝑆2 2 ≥ 𝑓𝑓0.05,𝑛𝑛1−1,𝑛𝑛2−1. 4 To further clarify on the attendance variable, we should emphasize that this variable is constructed separately for each test. For example, if a student missed the first 2 weeks of the semester but attended the rest of the class sessions during the semester, her attendance corresponding to test 1 would be about 0.5 (50 percent) while her attendance corresponding to the other three tests would be 1 (100 percent). Please also see the variable description in the appendix. 26 27 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 Table 4: T-tests for difference in Means (Sequenced versus Scrambled) by Course Section and Test5 (a) Section 015 Version n Mean t DF T-critical value Pr(T3.5 8.524*** (1.797) High School Econ 0.421 3.027 (2.088) (2.366) Prior College Econ 5.833*** 7.191*** (1.535) (1.658) SAT<1200 2.561 (2.422) SAT>1600 7.914*** (1.804) Constant 62.794*** 60.812*** 63.039*** 60.457*** 57.353*** 52.935*** (2.092) (4.256) (4.272) (4.611) (4.273) (4.990) R-squared 0.10 0.10 0.10 0.10 0.14 0.41 No. of Observations 515 354 350 354 354 246 Notes: Standard errors in parenthesis; ***: significant at 1 percent; **: significant at 5 percent; *: significant at 10 percent. 30 31 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 The next pair of control variables captures prior exposure to economics. The first of these two variables, “High School Econ”, is a binary dummy variable equal to 1 if the student took any economics classes in high school and 0 elsewhere. The second of the two variables, “Prior College Econ” is as defined earlier in the paper. Model 4, 5, and 6 in Table 6 presents estimates of these variables’ effect on students’ performance. Exposure to economics in high school had no statistically significant effect on students’ performance. However, students with prior college- level exposure to economics performed better, on average, by 5 – 7 percentage points. Finally, we added students’ self-reported SAT scores as proxies for ability. The variable labeled “Below 1200 SAT” is a dummy variable equal to 1 if a student reported a total SAT score of 1200 or below and 0 elsewhere. Similarly, “Above 1600 SAT” is a dummy variable equal to 1 if a student reported a total SAT score of 1600 or higher. The reference group in this case includes students who reported SAT total scores between 1200 and 1600. Estimates of model 6 reported in Table 6 suggest no statistically significant difference in performance between those reporting a total SAT score of 1200 or less and the reference group (12003.5 8.717** (2.091) High School Econ 0.113 2.029 (2.592) (2.968) Prior College Econ 7.221** 8.319** (1.803) (1.931) SAT<1200 3.377 (2.791) SAT>1600 8.436** (2.097) Sequenced × Attendance 2.684 -0.197 -2.061 -2.572 -2.329 -0.189 (5.872) (8.686) (9.321) (8.770) (8.470) (8.938) Sequenced × section 015 0.329 -1.988 -2.627 -2.436 -2.575 -4.612 (2.790) (3.293) (3.288) (3.382) (3.205) (3.435) Sequenced × Test 1 8.247* 9.249* 9.444* 9.592* 9.050* 4.747 (4.053) (4.503) (4.509) (4.533) (4.384) (4.497) Sequenced × Test 2 8.218* 7.886+ 6.663 6.816 5.367 1.580 (4.039) (4.632) (4.634) (4.701) (4.552) (4.723) Sequenced × Test 3 3.068 5.179 3.834 3.962 1.936 -5.727 (3.978) (4.469) (4.486) (4.539) (4.444) (4.597) Sequenced × Freshman -15.631* -15.714* -15.53* -17.30** -18.58** (6.562) (6.562) (6.553) (6.420) (6.413) 32 33 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 Sequenced × Sophomore -8.641 -8.943 -8.654 -11.636+ -13.109* (6.520) (6.462) (6.579) (6.349) (6.410) Sequenced × Junior -13.747* -14.692* -14.71* -17.52** -17.10** (6.697) (6.637) (6.658) (6.513) (6.574) Sequenced × (GPA<2.5) 0.693 9.094 (9.553) (9.662) Sequenced × (GPA>3.5) -0.567 (4.075) Sequenced × high School econ -0.919 4.181 (4.466) (5.195) Sequenced × Prior college econ -3.558 -2.760 (3.466) (3.912) Sequenced × (SAT<1200) -0.039 (5.735) Sequenced × (SAT>1600) -1.617 (4.100) Constant 64.248** 59.762** 61.352** 61.2** 56.601** 48.647** (2.302) (4.807) (4.794) (5.315) (4.819) (6.114) Observations 515 354 350 350 350 246 R-squared 0.120 0.131 0.153 0.153 0.196 0.461 F(k’, n-k) 1.43 1.52 1.38 1.38 1.62 1.46 Prob>F 0.211 0.148 0.197 0.197 0.11 0.127 Standard errors in parentheses; ** p<0.01, * p<0.05, + p<0.1 k’ denotes the number of restrictions under the null; k denotes the number of explanatory variables in the model (including the constant); n is the number of observations. In all the regression specifications reported in Table 7, we fail to reject the null of equal coefficients at conventional confidence levels. In simple terms, the results presented in Table 7 do not suggest any differences in the behavior of the two groups in the sample (those that took the sequenced version of the tests versus those that took the scrambled versions of the tests). Conclusions We examined the impact of question order on multiple-choice tests on student performance in an unconventional introductory economics course. Our empirical estimates indicate that question order does not influence student performance. Therefore, instructors of introductory economic courses need not be concerned about introducing bias in multiple-choice exams by using scrambled and unscrambled tests. This finding reinforces the conclusions of earlier studies by Bresnock, Graves and White (1989), Gohmann and Spector (1989), and Sue (2009). We also found no evidence that the structure of the course content influences the impact of question order on student performance. No systematic bias was found either when the course content consisted of unrelated standalone topics or when the course content was presented in a building block sequential manner. 33 34 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 Not surprisingly, our results show that academic ability does matter. For example, a student with a GPA of 2.5 or lower coming into the class will typically score about 11 percentage points below the course averages for those with GPAs between 2.5 and 3.5, and roughly 19 percentage points - several letter grades - below those of students with GPAs greater than 3.5, all else constant.7 Attendance is also positively correlated with performance. However, teasing out the causal effect of attendance requires dealing with potential endogeneity, since attendance may be driven by students’ ability and motivation. Although we cannot conclusively confirm it, prior studies that focus on the role of attendance suggest a positive causal effect on academic performance. Encouraging students to attend class by providing incentives whenever possible is a likely worthwhile effort. References Bresnock, Anne E, Philip E Graves, and Nancy White. "Multiple-Choice Testing: Question and Response Position." Journal of Economic Education, 1989: 239-245. Carlson, Lon J, and Anthony L. Ostrosky. "Item Sequence and Student Performance on Multiple- Choice Exams: Further Evidence." Journal of Economic Education, 1992: 232-235. Doerner, William M., and Joseph P Calhoun. "The Impact of the Order of Test Questions in Introductory Economics." Social Science Research Network. April 2, 2009. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1321906 (accessed June 4, 2012). Gohmann, Stephen F, and Lee C Spector. "Test Scrambling and Student Performance." Journal of Economic Education, 1989: 235-238. Rodgers, J. R. A panel-data study of the effect of student attendance on university performance. Vol. 45. 3 vols. Australian Journal of Education, 2001. Romer, David. Do Students Go to Class? Should They? Vol. 7. 3 vols. The Journal of Economic Perspectives, 1993. Sue, Della L. "The Effect of Scrambling Test Questions on Student Performance in a Small Class Setting." Journal for Economic Educators, 2009: 32-41. Taub, Allan J, and Edwaed B Bell. "A Bias in Scores on Multiple-Form Exams." Journal of Economic Education, 1975: 58-59. Thatcher, Andrew, Peter Fridjhon, and Kate Cockcroft. The Relationship between lecture attendance and academic performance in an undergraduate psychology class. Vol. 37. 3 vols. South Africa Journal of Psychology, 2007. 7 The university’s grade scale is one with “pluses” and “minuses”. We consider the “pluses” and “minuses” to be different letter grades. For example, A+, A, and A- are distinct letter grades. 34 35 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 Data Appendix Table A1: Variable Description and Sources Variable Label Variable Description Source Grade (Dependent Variable) Grade (out of 100) on the multiple choice section of the tests. Instructor’s grade book Attendance The proportion of classes attended before each test. For instance, if a student missed 2 of 8 class sessions before test1, attendance corresponding to the student’s first test grade would be 0.75 (6/8). Attendance corresponding to the test 2 is based on class sessions after test1 and before test2. Attendance for test3 and test4 are similarly computed. Instructor’s attendance records. Section015 This is a binary variable equal to 1 if a student was enrolled in course section 015, 0 elsewhere Students’ enrolment records. Test 1 This is a binary variable equal to 1 for observations corresponding to test1, 0 elsewhere. Instructor’s grade book Test 2 This is a binary variable equal to 1 for observations corresponding to test2 grades, 0 elsewhere. Instructor’s grade book Test 3 This is a binary variable equal to 1 for observations corresponding to test3 grades, 0 elsewhere. Instructor’s grade book Sequenced This is a binary variable equal to 1 if student took the sequenced version of the test, 0 elsewhere. Instructor’s grade book Freshman This is a binary variable equal to 1 if the student is a freshman, 0 elsewhere. Student self- reported information Sophomore This is a binary variable equal to 1 if the student is a Sophomore, 0 elsewhere. Student self- reported information Junior This is a binary variable equal to 1 if the student is a Junior, 0 elsewhere. Student self- reported information Below 2.5 GPA This is a binary variable equal to 1 if the student’s GPA is 2.5 (out of 4) or below, 0 elsewhere. Student self- reported information 35 36 | JOURNAL FOR ECONOMIC EDUCATORS, 15(1), 2015 Above 3.5 GPA This is a binary variable equal to 1 if the student’s GPA is 3.5 (out of 4) or above, 0 elsewhere. Student self- reported information High School Econ This is a binary variable equal to 1 if the student took any economics courses in high school, 0 elsewhere. Student self- reported information Prior College Econ This is a binary variable equal to 1 if the student took college-level economics course prior to the current course, 0 elsewhere. Student self- reported information Below 1200 SAT This is a binary variable equal to 1 if the student reported SAT total score of 1200 or below. Student self- reported information Above 1600 SAT This is a binary variable equal to 1 if the student reported SAT total score of 1600 or higher. Student self- reported information Table A2: Descriptive Statistics Variable Name Obs Mean Std. Dev. Min Max Grade 515 71.369 14.516 27 100 Attendance 560 0.761 0.324 0 1 Section 015 560 0.464 0.499 0 1 Section 035 560 0.536 0.499 0 1 Sequenced Test Version 515 0.256 0.437 0 1 Freshman 360 0.311 0.464 0 1 Sophomore 360 0.367 0.483 0 1 Junior 360 0.233 0.424 0 1 Senior 360 0.089 0.285 0 1 GPA 2.5 or below 356 0.056 0.231 0 1 GPA between 2.5 & 3.5 356 0.539 0.402 0 1 GPA 3.5 or higher 356 0.236 0.425 0 1 Had Economics in HS 360 0.833 0.373 0 1 Had Economics in College 360 0.322 0.468 0 1 SAT score 1200 or below 256 0.172 0.378 0 1 SAT score between 1200 & 1600 256 0.438 0.497 0 1 SAT score 1600 or higher 256 0.391 0.489 0 1 36