25 | JOURNAL FOR ECONOMIC EDUCATORS, 14(1), SUMMER 2014 25 MEASURING UP: ASSESSMENT IN MICROECONOMICS Jill K. Hayter1 and Carolyn F. Rochelle2 Abstract Appropriate assessment is of major importance for universities today. Many faculty perceive assessment as already occurring through grade assignment. This paper investigates grades versus knowledge of learning objectives as forms of assessment. By analyzing the relationship between examination questions and post-test comprehension of learning objectives in Principles of Microeconomics, this study tests differences in proportions of correct responses from the two evaluation methods. For some learning objectives there are statistical differences between the two proportions but insignificant differences for the others. These mixed results demonstrate exam questions and learning objective post-test questions are not necessarily equal measures of student learning. Keywords: Grades versus assessment, learning objectives, economic education JEL classification: A20, A22, I21 Introduction Assessment of learning is of major importance for universities today. For example, business schools seeking AACSB accreditation or reaffirmation must implement the Assurance of Learning standards, often referred to as assessment. In the past, grades and academic performance have been used as the accepted measure of student success in various disciplines including Economics (Elzinga and Melaugh 2009), Accounting (Jones and Fields 2001), and Business Statistics (Rochelle and Dotterweich 2007). Recently, however, administrators have challenged whether a student’s course grade is an adequate measure of learning. As part of the assessment process, institutions have developed learning objectives for individual courses as well as learning objectives at the program level. While faculty members often contend that course grades are an adequate measure of learning, recent views of assessment emphasize that grades, while potentially correlated to learning objectives, are not measures of learning objectives themselves (Carnegie Mellon 2012). 1 Assistant Professor of Economics, Department of Economics and Finance, East Tennessee State University, Johnson City, TN 37614 2 Lecturer of Economics, Department of Economics and Finance, East Tennessee State University, Johnson City, TN 37614 26 | JOURNAL FOR ECONOMIC EDUCATORS, 14(1), SUMMER 2014 26 Advocates for assessment have several arguments against using grades as a measure of achievement. Gloria Rogers (2003) states that grades merely represent how well a student met the course expectations and requirements. Grades can vary by differences in course content and grading policies among professors teaching the same course. Rogers’ argument that course content is not the same across faculty members, however, fails to consider a program that has common course learning objectives already in place. While different methods professors use in calculating a student’s final grade may be a concern for studies that only use final grades as a measure of achievement, professors can tell a great deal about the level of student understanding from looking at a raw exam, quiz, or homework grades. Despite the recent emphasis on using learning assessments as measures of achievement, some faculty members have recognized the grading process as a potential assessment tool. Walvoord and Anderson refer to grading as “…the process by which a teacher assesses student learning through classroom tests and assignments, the context in which good teachers establish that process, and the dialogue that surrounds grades and defines their meaning to various audiences” (1998, 1). Walvoord and Anderson provide practical ways for teachers to evaluate learning outcomes in their classrooms through the grading process and state that when faculty use the grading process as a means of assessment, they “…maintain maximum faculty control over curricular content; over the teaching, learning, and grading process in classrooms; and over the tests, assignment, criteria, and standards by which faculty assess student learning” (1998, 154). This study contributes to the economic education and assessment literature by analyzing the relationship between performance on examination questions and knowledge of learning objectives as measured by a post-test of course learning objectives in Principles of Microeconomics at a regional university with an enrollment of approximately 15,000 students. The course learning objectives were related to specific exam questions covering a unique learning objective. To the authors’ knowledge this is the first study that seeks to provide empirical evidence on the assessment versus grades debate by examining knowledge of learning objectives. Previous Literature A number of studies have examined student performance in economics courses. Marburger (2001, 2006) used exam scores to analyze performance. Browne et. al. (1991) examined performance by utilizing student scores from a Test of Understanding College Economics (TUCE). Elzinga and Melaugh (2009) used final letter grade in the course to examine performance. These studies provide a foundation for analyzing student performance in economics and include several different ways of measuring performance. However, there is a paucity of literature on the best measurement(s) of that performance. Over recent years, faculty members have increasingly gathered assessment data in their classrooms, but have failed to see any significant improvement in student learning (Marchese 1999). Faculty are often concerned with the time it takes to conduct learning assessments. Often assessment is performed through the use of rubrics that are lengthy and wordy. Baryla et. al. (2012) examined the costs and data quality associated with complex, lengthy rubrics. Using 27 | JOURNAL FOR ECONOMIC EDUCATORS, 14(1), SUMMER 2014 27 factor analysis and data from more than 1,400 oral communications student rubrics with a total of 49 criteria, they investigated whether numerous criteria assessed the same item. Results indicated that at most seven of the 49 evaluation criteria were significant. These findings suggest that the assessment process is already more complex and time consuming for faculty members than is necessary. This paper extends the economic education and assessment literature to investigate the relationship between examination questions and comprehension of learning objectives in Principles of Microeconomics courses at a mid-sized regional university. Using data on specific exam questions and knowledge of learning objectives as measured by a post-test given at the end of the semester, this study seeks to provide empirical evidence towards resolving the grades versus assessment debate. While student grades are not directly analyzed, grading is examined in by considering a subset of examination questions related to each learning objective. By matching the learning objectives to specific exam questions, the authors compare the proportion of correct responses on examination questions for a specific learning objective to the proportion of correct responses on the learning objective post-test for the same learning objective. As significant resources are allocated each year to assessment, the findings of this study may be of particular interest to faculty members and college administrators in search of a more efficient means to conduct assessment of learning objectives in the classroom. Data The data set was constructed from two Principles of Microeconomics courses, one from Spring 2011 and one from Fall 2011, taught by the same professor. The students enrolled in these courses were primarily freshman and sophomore pre-business majors. All business majors (Accounting, Economics, Finance, Management and Marketing) at this university are required to take Principles of Microeconomics and earn a grade of a C or better. There are no prerequisites for taking the Principles of Microeconomics course although most students choose to take Principles of Macroeconomics before taking Principles of Microeconomics. The two classes met twice a week for eighty minute class meetings and were similar in size. Students enrolled in either the spring or fall Principles of Microeconomics course were evaluated on a combination of exams (75% of final grade) and quizzes (25% of final grade). Although quiz grades were used in calculating the student’s final grade, this study only includes exam and learning objective post- test questions in the analysis. The Economics department has five common learning objectives in place for Principles of Microeconomics. Learning objectives are defined as concepts that students should know upon completion of the course. The five learning objectives for Principles of Microeconomics are described below.  Learning Objective 1: Use the marginal cost-benefit model for decision-making.  Learning Objective 2: Use the supply-demand model to demonstrate the effect of various events on market price and quantity traded.  Learning Objective 3: Understand, measure, and apply the various elasticity concepts. 28 | JOURNAL FOR ECONOMIC EDUCATORS, 14(1), SUMMER 2014 28  Learning Objective 4: Define, recognize the causes of, and provide potential solutions to the market failures associated with externalities, common resources, and public goods.  Learning Objective 5: Use economic models of costs and revenues to make optimal business decisions in various market structures. At the end of each semester, students were given a ten question learning objective post- test related to the five learning objectives. According to Bloom’s Taxonomy of Cognitive Skills, each learning objective had one question that was classified as using lower-order thinking skills and one question that was classified as using higher-order thinking skills (Anderson et. al. 2001). Students were informed that they would receive extra credit, the only such opportunity for the semester, for each question answered correctly on the learning objective post-test. With the university plus/minus grading system, many students were at the margin for a specific grade, making extra credit especially attractive. Three forty question examinations were given each semester. Utilizing information from all three exams allowed the mapping of 240 exam questions (120 questions per semester) to the corresponding learning objective. In the instance that exam questions required knowledge of more than one learning objective, the exam question was mapped to the most recently covered learning objective. For this reason, very few exam questions were mapped to the first learning objective, “Use the marginal cost-benefit model for decision-making,” since this concept is incorporated into most topics in microeconomics. Exam questions and the corresponding learning objective questions were all multiple choice and were similar in level of difficulty according to Bloom’s Taxonomy (Anderson et. al. 2001). The final data set includes only those students who had taken all three exams and the learning objective post-test: 182 total students with 89 students from Spring 2011 and 93 students from Fall 2011. Table 1 provides a breakdown of the aggregated number of questions per exam that correspond to each learning objective. TABLE 1: Aggregated Number of Exam Questions Mapped to Learning Objectives Exam1 Exam 2 Exam 3 Learning Objective 1 3 Learning Objective 2 42 Learning Objective 3 32 Learning Objective 4 6 61 Learning Objective 5 16 80 N 80 80 80 Differences in student competency based on exams versus the learning objective post- test were measured by aggregating all students’ responses on exam questions for each specific learning objective and by aggregating all students’ responses on the learning objective post-test for each specific learning objective. Then, responses for all exam and learning objective post-test 29 | JOURNAL FOR ECONOMIC EDUCATORS, 14(1), SUMMER 2014 29 questions were divided into correct and incorrect answers. Finally, the proportions of correct answers for exam questions and learning objective post-test questions were calculated. Table 2 shows the aggregated responses that were obtained for each learning objective. TABLE 2: Aggregated Responses for Exam and Assessment Questions Exam Questions Assessment Questions Total responses # Correct Proportion correct Total responses # Correct Proportion correct Learning Objective 1 271 203 0.749 364 269 0.739 Learning Objective 2 3826 2833 0.740 364 229 0.629 Learning Objective 3 2920 1976 0.677 364 269 0.739 Learning Objective 4 6119 4724 0.772 364 266 0.731 Learning Objective 5 8704 6331 0.727 364 236 0.648 Methodology and Empirical Results In order to determine if significant differences exist in the proportions of correct responses from the two evaluation methods, a z test for the difference between the two proportions was performed for each learning objective. A z test was chosen because the procedure readily identifies any significant differences, including the direction of the differences. The following hypothesis was tested for each of the five learning objectives: H0: pcorrect exam questions = pcorrect assessment questions H1: pcorrect exam questions ≠ pcorrect assessment questions Table 3 presents the results from the z tests. As can be seen from Table 3, the results of the z tests are mixed. Whether a difference in proportions exists depends on the individual learning objective tested. Where significant differences are found, the direction of the differences varies with the specific learning objective. TABLE 3: Results of Z Test for the Difference Between Two Proportions Exam Proportion Correct Assessment Proportion Correct Difference in Proportions Z score Learning Objective 1 0.749 0.739 0.010 0.287 (.7739) Learning Objective 2 0.740 0.629 0.111 4.576*** (.00001) Learning Objective 3 0.677 0.739 -0.062 -2.41* (.0160) 30 | JOURNAL FOR ECONOMIC EDUCATORS, 14(1), SUMMER 2014 30 Learning Objective 4 0.772 0.731 0.041 1.816 (.0694) Learning Objective 5 0.727 0.648 0.079 3.305** (.0010) Notes. Numbers in parentheses are p values. *p < .05; **p < .01; ***p < .001 Learning Objectives with No Significant Difference At a significance level of .05, both learning objectives one and four show no evidence of a difference in proportions between correct student responses from exam questions for that learning objective and the two learning objective post-test questions pertaining to the same learning objective. Learning objective one is unique in that there were few exam questions mapped to this objective. As mentioned previously, this is because of the nature of learning objective one. The learning objective post-test questions and exam questions used to evaluate learning objective one were very similar, and the conclusion of no significant difference is quite possibly indicative of this and not a surprising finding. Alternatively, the much larger number of exam questions for learning objective four allows for a more thorough evaluation of the students’ knowledge of the course material related to this learning objective. Some questions tested very basic understanding while others required more advanced knowledge as well as higher order application. It is encouraging that there is no significant difference in proportions of correct responses with the two learning objective post-test questions and the more numerous exam questions for learning objective four. Learning Objectives with Significant Differences At a significance level of .01, both learning objectives two and five find that students have a significantly higher proportion of correct exam questions than learning objective post-test questions. The remaining objective, learning objective three, shows the reverse sign for the difference in proportions. At a significance level of .05, students have a significantly lower proportion of correct exam questions than correct learning objective post-test questions. These mixed results suggest exam questions and the learning objective post-test questions are not equal measures of knowledge on course content. One potential explanation for the difference in proportion of correct exam questions and proportion of correct learning objective post-test questions is the deterioration of learning. If deterioration of learning is occurring, then we should see no difference in proportions between correct student responses from exam questions and the two learning objective post-test questions pertaining to learning objective five. This is because exam three consisted only of questions related to learning objective five, and the learning objective post-test was given on the same day. Additionally, one would expect differences in the proportions of correct exam questions and the proportion of correct learning objective post-test questions for all other learning objectives since students were tested over those learning objectives in exam one and exam two. Our findings show significant differences with respect to learning objective five; although for two of the learning objectives where we would expect differences to exist there are none. These findings do not support the deterioration of learning hypothesis. 31 | JOURNAL FOR ECONOMIC EDUCATORS, 14(1), SUMMER 2014 31 Conclusion This study seeks to provide empirical evidence for the assessment versus grades debate by analyzing the relationship between performance on examinations and knowledge of learning objectives as measured by a learning objective post-test in Principles of Microeconomics courses. Results from the statistical analysis are mixed. For learning objectives one and four, there is no evidence of a difference in proportion of correct answers of learning objective versus exam questions. Additionally, for learning objectives two and five, the proportion of correct answers is higher for exam questions than the corresponding learning objective post-test questions. Finally, for learning objective three, the proportion of correct answers is higher on the learning objective post-test questions than the exam questions. Further empirical research is needed in the grades versus assessment arena. Had results consistently shown no significant differences in the proportion of correct exam questions and the proportion of correct learning objective post-test questions, it would appear that assessment could be taking place through examinations. However, since there are significant differences for three of the five learning objectives, there is no evidence that exam questions and learning objective post-test questions are equal measures of student learning of course content. These mixed results suggest a need for caution when developing assessment instruments and suggest that additional research is needed. Additional research includes tracking individual student responses on exam questions and post-test questions. Future research also includes additional analysis of deterioration of learning and analyzing student paired data to examine if students who had not grasped some of the more difficult concepts in the beginning stages of the course had gained further understanding of them by the end of the course. References Anderson, L. W., Krathwohl, D. R., Airasian, P.W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J. and Wittrock, M.C. (Eds.). 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Abridged ed. Addison Wesley Longman, Inc.: 31. Baryla, Ed, Gary Shelley, and William Trainor. 2012. “Transforming Rubrics Using Factor Analysis.” Practical Assessment, Research & Evaluation 17(4). http://pareonline.net/getvn.asp?v=17&n=4. Browne, Neil M., John Hoag, Mark Wheeler, and Nancy Boudreau. 1991. “The Impact of Teachers in Economic Classrooms.” Journal of Economics 17: 25-30. Carnegie Mellon Faculty. 2012. “Assessment Examples and Tools.” Enhancing Education. http://www.cmu.edu/teaching/assessment/index.html. Elzinga, Kenneth, and Daniel O. Melaugh. 2009. “35,000 Principles of Economics Students: Some Lessons Learned.” Southern Economics Journal 76(1): 32-46. Jones, Jefferson P. and Kent T. Fields. 2001. “The role of Supplemental Instruction in the first Accounting Course.” Issues in Accounting Education 16(4): 531-547. Marburger, Daniel R. 2001. “Absenteeism and Undergraduate Exam Performance.” Journal of Economic Education 32(Spring): 99-109. http://pareonline.net/getvn.asp?v=17&n=4 http://www.cmu.edu/teaching/assessment/index.html 32 | JOURNAL FOR ECONOMIC EDUCATORS, 14(1), SUMMER 2014 32 ———. 2006. “Does Mandatory Attendance Improve Student Performance?” Journal of Economic Education 37(Spring): 148-155. Marchese, T. 1999. “Assessment Today – and Tomorrow.” Change 31.5 (September): 4. Rochelle, Carolyn F., and Douglas Dotterweich. 2007. “Student Success in Business Statistics.” Journal of Economics and Finance Education 6(1), Summer. Rogers, Gloria. 2003. “Do Grades Make the Grade for Program Assessment? Assessment Tips with Gloria Rogers.” Communications Link Fall/Winter: 8-9. Walvoord, Barbara E., and Virginia J. Anderson. (1998) 2010. Effective Grading: A Tool for Learning and Assessment. San Francisco: Jossey-Bass.