Geetha Rajaram 1 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 TEACHING ECONOMETRICS USING FORMATIVE ASSESSMENT Geetha Rajaram 1 Abstract This article studies the effectiveness of formative assessment techniques for an econometrics course. A large scale project with extensive formative assessment was included in the course, incorporating both summative and formative assessment. The specific assignment is for students to learn all steps to turn raw data obtained from the Interuniversity Consortium for Political and Social Research (ICPSR) using the statistical software SPSS into a viable thesis that is worthy of undergraduate conference presentations and publications. Learning gains from implementation of this project using extensive formative assessment are measured by changes in student course grades. Key Words: Undergraduate Economics Education, Teaching of Economics JEL Classifications: A22 Introduction For a course as challenging for students as econometrics, how do instructors know that students with varying math and analytical skill levels are learning the material before it is too late to affect their grade? Instructors need continuous information on student learning to address the problem. There are two types of assessments that can provide useful information on how students’ are learning throughout an introductory econometrics course: summative and formative assessments. As defined by Bloom, Hastings and Madaus (1971), summative evaluation is the “evaluation used at the end of a term, course or program for purposes of grading, certification, and evaluation of progress, or research on the effectiveness of a curriculum, course of study or educational plan”, for example homework, mid-terms, and final exams (p 155). Formative evaluation was first used by Scriven (1969) to describe an assessment form to improve curriculum. Bloom, Hastings, and Madaus (1971) expanded the definition from purposes of curriculum improvement to include improvement in the “process of teaching and learning, and since formative evaluation takes place at the formation stage, every effort should be used to improve the process” (p 155). Summative evaluation is the approach that is currently used by most econometrics instructors. Becker and Watts (1996) found that problem sets are used more extensively in econometrics courses than any other courses in economics. Although problem sets and exams are helpful and essential, they may not give the instructor continuous information on student learning throughout the course. Furthermore, as articulated by Becker and Greene (2001), problem sets and exams in econometrics are rarely based on actual data and events. Actual data and events are 1 Assistant Professor of Economics, Department of Business and Economics, Whittier College, 13406 Philadelphia Ave., Whittier, CA 90608 2 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 helpful in teaching students the theoretical concepts in econometrics. Moreover, with the availability and relatively easy accessibility of actual data and events, it is very possible to use real world examples instead of contrived ones. With the availability of such resources, designing a project that incorporates extensive formative assessment is feasible. Formative assessment should include two fundamental characteristics: continuity and feedback. As eloquently articulated by Walstad (2005), summative assessment helps instructors judge students “mastery of content,” and formative assessment is “continuous and can help shape instruction and learning throughout the course” (p 193). The purpose of the first element of formative assessment, continuity, is for instructors to identify early on what students are learning and modify their instruction to help students learn before it is too late. The results of the project discussed in this paper show that continuity in assessment throughout the project helped students develop a deeper understanding and learning of theoretical material. The second element defining formative assessments is feedback. In an article written by Black and Wiliam (1998), formative assessment was defined as “those activities undertaken by teachers and/or by their students, which provide information to be used as feedback to modify the teaching and learning activities in which they are engaged” (p 7). As further summarized by Black and Wiliam (1996) the goal of feedback mirrors the definition presented by Ramaprasad (1983), where the purpose of feedback is stated to be to close the gap between the actual level and the referenced level of information and learning. The actual level is the learning level where a student is at, and the referenced level is where the student should be. In 2009, a large-scale project with extensive formative assessment was included in my upper-division econometrics course. In this project, both elements of continuity and feedback were included. The assignment was designed to incorporate summative as well as formative assessment. The specific assignment is for students to learn all the necessary steps to turn raw data obtained from the Interuniversity Consortium for Political and Social Research (ICPSR) using the statistical software SPSS into a viable thesis that is worthy of undergraduate conference presentations and publications. Learning gains from implementation of this project using extensive formative assessment were measured by changes in student course grades. The motivation to alter the course in 2009 initially was based on my personal dissatisfaction with how the course seemed to be handled by students previous to 2009. Students were not bridging the gap between theory and practical applications. The course evaluations from these earlier classes yielded some interesting results. Students felt they were not fully engaged in the application of the concepts, thus making it difficult for them to understand the concepts. Many felt the course was just too difficult. Two-thirds of the students, who took the class in 2008, for example, felt they were not able to apply the concepts and properly learn the material. Given the diversity of students’ mathematical and analytical skill levels in the course, overcoming this difficulty presented a very real challenge. Teaching introductory econometrics the “traditional way” (mostly theoretically) to students with diverse levels of skills may not be the most effective method. The smaller class sizes at a liberal arts college gave me the opportunity to bridge the gap between theory and practical applications by implementing extensive use of formative assessment in addition to summative assessment in the course. This paper proceeds as follows: Section II briefly introduces the literature on the effectiveness of formative assessment. Section III discusses the stages of the econometric project and the formative assessment tools used. Section IV presents the results with respect to course grades before and after the changes in the course. Other benefits and costs to implementation of the project are summarized in section V, followed by concluding remarks. 3 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 Effectiveness of Formative Assessment Blaming students for lack of skill is a common explanation of poor student performance (Wilson and Scalise 2006). But as noted by Wilson and Scalise the problem can be summarized not entirely as a problem of “low ability” but also as a “lack of engaged students” (p 636). This project will show through formative assessment that students can become more engaged and raise their level of ability and understanding of the materials in the course. Black and Wiliam in a seminal article “Inside the Black Box” (1998) were able to identify over twenty studies that show significant and substantial learning gains following the enhancement of formative assessment. The studies range from assessment of kindergarten to university students. Many of the studies also concluded that the improved formative assessment helps low achievers more than other students. The learning gains were measured by an “effect size” in the range between 0.4 and 0.7. Learning gains were typically measured by a meta-analysis procedure using the difference between an experimental and control group divided by the standard deviation of the control (Glass, McGraw, and Smith 1981). This ratio is called the effect size. The effect size is a method of quantifying the comparison between a control and experimental group. An effect size of 0.4 translated into an average student who was a part of formative assessment classroom methods performing as well as the top third of students who did not participate (Coe 2002). Black and Wiliam (2003) also conducted a study involving twenty-four teachers in six schools in England called the KMPFAP project (King’s Medway Oxforshire Formative Assessment Project) in mathematics and science. They showed quantitative evidence that formative assessment raised the standard of achievement on the National Curriculum Test and the General Certificate in Secondary Education Test with a point estimate of the mean effect size of 0.32. In addition to the work done in math and science classes by Black and Wiliam, a study from a principle of economics course shows similar results. Applying feedback action as a form of formative assessment using applied exercises in a principles of economics course, Faulk’s (2007) regression results show that formative assessment had a positive and significant effect on the difference between pre-test and post-test scores. Faulk showed that minority students made the most improvement. However, his results also show that the formative assessment dummy in the regression was not a significant determinant of targeted assessment scores in supply-and- demand analysis, price controls, and monetary policy. He concludes that applied exercises and feedback as a formative assessment tool “may serve to motivate students to study more and thereby improve their general understanding of economics rather than lead to significant improvement in targeted topics” (p 13). Faulk’s studies do show that formative assessment had a measurable success in raising overall student test scores. The Project The project that was added to the introductory econometrics course in 2009 was designed as a mini-dissertation for students with the extensive use of formative assessment. During the course, students learn how to work with a large survey from beginning to end. They download and clean up data, recode and transform data, create descriptive statistics, run regressions, write a literature review, explain the data, theory, model and hypothesis, write an analysis explaining the regression output, and make policy proposals. Simultaneously, students learn the standard topics in econometrics (with a reduced number of topics to accommodate a large-scale project), thus allowing for more depth but less breadth in the course. This type of change could be controversial on some levels. First, students are not taught all the topics in econometrics that are 4 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 usually covered in many traditional courses. Second, given the loss of breadth in materials, some faculty may determine that students are not as prepared for graduate level study in econometrics based on this course. However, including a large-scale project allows students to thoroughly apply concepts, thus improving their abilities to perform better in their writing and analytical skills in graduate school. Moreover, the project gives students who usually struggle with the material continuous feedback for them to improve their work. I briefly discuss a possible remedy for the loss of breadth in the course at the end of this paper. Both formative and summative assessments are used throughout the project. The formative assessment of this project is achieved by requiring students to revisit their paper (project) in many stages. Steps in Implementing the Project The course is taught during the fall semester. The first step begins before students depart for the summer. I meet with students who are registered for the fall course to show them how to log on to the Interuniversity Consortium for Political and Social Research website (ICPSR) and look for surveys. They are instructed how to download a survey using SPSS. This is done so that students have the summer months to look for a survey on a topic they are interested in pursuing. At the beginning of the course in September, each student meets with the professor for a formative assessment of step one. Students are required to show the instructor two surveys, their first and second choice. The instructor meets with each student to go through the survey on the computer and talk to them about their interests. Students who do not have their surveys are penalized with a loss of points. They are given a chance to turn in credible surveys within a week into the semester for points to be added back on. In step two, students “clean up their survey.” This process is taught to them through a project done jointly in class using a topic and survey the professor has chosen from ICPSR. Students are shown step-by-step how to recode data into binary variables, transform variables, assess missing observations, and other data-related functions. At the end of step two, students turn in the original survey and the “cleaned up” data from the survey. They also turn in a step-by-step log of how they transformed their survey from its original form to its current form. The instructor should be able to replicate the student’s data and end up with similar results. At this point students meet with the professor individually to assess all their work and are given both positive and critical feedback on their progress. More specifically, students who need to redo parts of step two will have the chance to turn in a second draft, reducing the possibility of data-related problems in the final version of the paper. The formative assessment in step two requires the professor to assign a grade on how well the student “cleaned up their survey” and the student to assign themselves a grade as well. Students are told if the gap in the two grades is far apart, the student will be penalized through a loss of points. During the meeting with the instructor, students’ will reveal the grade they assigned to themselves, and the instructor will have the grade they assigned to the student written on their paper. The instructor shows the grade at that point. If the grades are more than half a grade apart, points will be deducted for that section of the draft. This form of assessment is repeated throughout the other steps of the project. In steps three, four, and five, students write (a) their theoretical analysis, econometric model, and literature review sections of their paper, (b) their descriptive statistics section, and (c) their regression results and policy analysis section. Each section is turned in as a separate step so that students can be given extensive assessment of each section. 5 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 For formative assessment to be helpful, both positive and critical comments should be specific. For example, the comment is not just “excellent” as a positive comment, but “your definition and calculation of total cost of health care is excellent”, or “the human capital model should test for the diminishing marginal qualities of your experience variables, thus you should square that independent variable.” By the final draft students have been given extensive formative assessment that is continuous with frequent feedback. At each step, the instructor and students both grade their work, and students are penalized if there is more than half a grade gap between the two grades. If the grades are unsatisfactory, students are given the choice of turning in multiple drafts on each section. They finally turn in a completed draft a few days before the final exam. Given the extensive formative assessment, students are no longer surprised at their grade in the project and in the course. Results Evidence on the effectiveness of formative assessment is presented by an analysis of the effect size measurement, analysis of descriptive statistics, and an analysis of ordinary least square regression of overall course grades. Observations are from classes taught in 2007, 2008, and 2009. 2 Every effort was made by the instructor to be consistent with the grading of students for all three years. Homework, midterm, computer, and final exams were similar for all three years. Analysis of Effect Size Learning gains in educational research are typically measured by using a meta-analysis procedure called the effect size (ES). A population ES measure using the standardized means between two known populations is called the Glass Delta (Glass, McGraw and Smith, 1981): This is the difference in means of the experimental group and control group divided by the standard deviation of the control group. Effect sizes generally assume that both groups (control and experimental) are normally distributed and have equal variances. 3 The control group is comprised of students who took the course in 2007 and in 2008 whereas the experimental group is the group of students from 2009. The result in Table 1 indicates the ES of a comparison of the final course grade between the control and experimental group is 0.6. The finding in this study is evaluated by a two-tailed T- test of pair of independent group for mean differences resulting in a p-value of 0.08 with a corresponding confidence interval at the 90% confidence level of [0.2, 9.6]. The ES is an equivalent measure to a Z-score of a standard normal distribution. The ES size can be interpreted as the percentile of the average experimental group student relative to the 2 One student was dropped, because he did not attend a single class and received a zero score on all assignments. Data on students are actual data available to the instructor or obtained from the Registrar’s Office. 3 The Levene’s Test in this study shows that there are no differences between variances of the population. The graphical P-P plot and Q-Q plot show normality. In addition, the Shapiro-Wilk test, valid for sample sizes below 50, indicates a normal distribution. 6 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 average control group student. 4 Effect sizes can be also interpreted as the percent of “non- overlap” of experimental group scores to those of control group scores (Cohen, 1988; p 21). An ES of 0.6 suggests a non-overlap of 73% in the two distributions. Table 1: Mean and Standard Deviation of Course Grades Mean Std Dev Formative Assessment dummy = 1 79.06 (n = 12) 6.77 Formative Assessment dummy = 0 74.23 (n = 25) 8.29 Analysis of Descriptive Statistics The data represent three semesters of introductory econometrics course. Years 2007 and 2008 represent the control group (dummy formative assessment = 0) and the year 2009 represents the experimental group (dummy formative assessment = 1). The mean characteristics of the pooled group, control, and experimental group are displayed in Table 2. The average Table 2: Mean and Standard Deviation (S.D.) for Pooled, Control, and Experimental Group Pooled Control Experimental Mean S.D. Mean S.D. Mean S.D. Formative Assessment dummy 0.32 0.47 --- --- --- --- Final Course Grade* 75.8 8.06 74.23 8.29 79.06 6.77 Race dummy (White = 1) 0.62 0.49 0.56 0.51 0.75 0.45 Gender dummy (female = 1) 0.68 0.47 0.64 0.49 0.75 0.45 Age 20.19 0.84 20.2 0.91 20.16 0.72 Grade Point Average 3.27 0.38 3.31 0.38 3.17 0.38 Economic Major dummy 0.68 0.47 0.76 0.44 0.5 0.52 Grade in Microeconomics Principles (A/A- = 1)** 0.38 0.49 0.28 0.46 0.58 0.51 Grade in Macroeconomics Principles (A/A- = 1) 0.35 0.48 0.4 0.5 0.25 0.45 Calculus 1 dummy 0.11 0.31 0.52 0.51 0.50 0.52 Calculus 2 dummy 0.43 0.5 0.40 0.5 0.50 0.52 Introduction to Elementary Statistics dummy 0.7 0.46 0.84 0.37 0.42 0.51 Absence dummy*** 0.19 0.39 0.2 0.41 0.17 0.39 N = 37 25 12 *See appendix for grading plan. ** All courses listed are taken before econometrics. All students took both introductory economics. ***Students who missed more than the two permissible absences. 4 Further interpretation of effect sizes can be found in Rosenthal and Rubin (1982), Glass, McGraw and Smith (1981), Rosenthal (1984), Coe (2002), Cohen (1969), and Cohen, Kulik and Kulik (1988). 7 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 course grade is 75.8% which is a C grade level. The average student is a 20-year- old white female, a junior with an average grade point average of 3.27, an economics major who did not get an A in her introductory courses, who took introductory statistics but not calculus, and did not miss more than two classes during the semester. Both introductory economics courses are required for econometrics. About one third of the students got an A or A- in the introductory economics courses. A comparison of the students between the two groups indicates, on average in both groups, students are of junior standing with a larger percentage of white and female students in the experimental group but lower percentages of economics majors (76% for the control group compared to 50% for the experimental group). With respect to prior coursework taken, a lower percentage of students from the control group got A’s in introductory microeconomics (28% compared to 58%), but fared better in introductory macroeconomics (40% compared to 25% A grades). About the same percentage of students took first semester calculus. Twice as many students took introductory statistics in the control group compared to the experimental group, 84% versus 42%. Higher percentages of students had taken Calculus 2 in the experimental group compared to the control group (50% versus 40%). The average grade point average for the control group is only slightly higher than that of the experimental group, a difference of 0.14. As indicated previously, students from the experimental group on average scored 4.83 percentage points higher on their overall econometric course grade compared to control group students. A further review of the effectiveness of formative assessment measured by the differences in mean course grade between the control (without formative assessment) and experimental groups (with formative assessment) is presented in Table 3. Formative assessment was especially significant in raising course grades for nonwhites and females, a significant increase of 5.71 percentage points in overall grade for nonwhites, and 6.6 percentage points for females. For whites and males, the assessment had no significant effect in raising overall course grade. Mean differences are significantly positive for students of all majors who took the econometrics course, especially for non-economics majors. Course grades improved by 5.3 percentage points for economics majors and 8.8 percentage points for non- majors, almost a full letter grade. The steeper the climb in terms of learning economic theoretical concepts, the more the student benefited from formative assessment. Changing assessment techniques did not significantly affect students who do not show up to class, but was significantly effective for those who attended, 4.7 percentage points, approximately half a letter grade. This is not surprising. The more motivated a student is, the more likely to attend class, thus benefitting from the assessment. Moreover, if a student misses class, he or she is not participating in the formative assessment process. Analyzing the effectiveness of formative assessment for students based on their previous coursework does not follow a consistent pattern. Formative assessment was significantly helpful for students who did not take introductory statistics, an increase of a substantial 10.38 percentage points, a full letter grade. Yet, it was also helpful to those on the other end of math skills, students who took second semester calculus, approximately 5 percentage points, while not significant for students who took only the first semester calculus. This may suggest the continuous assessment and feedback helped students with the least and the most amount of math skills, more than those with middle-level math levels. 8 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 Table 3: Course Grade Descriptives and Mean Differences for Control and Experimental groups Control Experimental Mean Diff. p-value Mean S. D. Mean S. D. Formative Assessment 74.23 8.29 79.06 6.77 4.83 0.08 Race 1 = White 0 = nonwhites 72.83 76.02 7.74 8.97 78.17 81.73 7.39 4.37 5.34 5.71 0.07 0.05 Gender 1 = Female Gender 0 = Male 72.36 77.58 8.54 7.05 79.01 79.20 7.85 2.40 6.65 1.62 0.03 0.45 Economics Major dummy = 1 Economics Major = 0 76.36 67.51 7.12 8.68 81.75 76.36 6.29 6.62 5.39 8.85 0.03 0.01 Absence dummy = 1 = 0 65.50 76.42 7.44 7.07 68.65 81.14 3.67 5.07 3.15 4.72 0.18 0.04 (A/A- = 1) (A/A- = 0) Grade in Microeconomics Principles dummy 79.31 72.26 5.74 8.49 82.70 73.93 5.31 5.28 3.39 1.67 0.09 0.53 (A/A- = 1) (A/A- = 0) Grade in Macroeconomics Principles dummy 77.42 72.11 6.18 9.00 84.66 77.19 1.17 6.85 7.24 5.08 0.01 0.09 Calculus 1 dummy = 1 Calculus 1 = 0 75.55 72.81 8.82 7.79 80.84 77.27 5.78 7.73 5.29 4.46 0.15 0.08 Calculus 2 dummy = 1 Calculus 2 = 0 77.34 72.17 4.87 9.54 82.31 75.80 5.71 6.53 4.97 3.63 0.01 0.25 Intro. to Elementary = 1 Statistics dummy = 0 74.77 71.43 8.71 5.54 75.21 81.81 7.08 5.42 0.44 10.38 0.80 0.00 N 25 12 * p-values conducted from t-test to compare means of two independent groups. Numbers in bold are significant at the 0.05 significance level. Regression Analysis Results from an ordinary least square regression are presented in Table 4. Characteristics such as gender, race, and a student’s class standing had no significant effect on the econometrics course grade. The coefficient for formative assessment is positive and highly significant, improving student course grades by approximately 7.7%, all else equal. This supports Black and Wiliam’s (1998) observation from a review of twenty-one studies that there were substantial learning gains following the enhancement of formative assessment. On the other hand, a regression model with the final exam grade as the dependent variable did not yield significant results for the formative assessment coefficient. This suggests that the assessment was helpful in raising overall performance (combination of homework, exams, paper and computer exam), but did not necessarily affect one particular exam. 9 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 Table 4: Regression Results on Course Grades Course Grade Variable Coefficient p-Value Intercept 31.26 0.24 Formative Assessment dummy = 1 7.72 0.001*** Race dummy (nonwhite = 1) -3.01 0.117 Gender dummy (female = 1) -0.85 0.671 Age -0.03 0.977 Grade Point Average (GPA) 13.32 0.000*** Economic Major dummy=1 5.03 0.027** Absence dummy = 1 -5.39 0.050** Grade in Principal Microeconomics dummy (A/A- = 1) -1.65 0.659 Grade in Principal Macroeconomics dummy (A/A- = 1) -1.64 0.526 Calculus 1 dummy = 1 -2.59 0.352 Calculus 2 dummy = 1 3.65 0.079* Introduction to Elementary Statistics dummy = 1 -1.36 0.560 Number of Observations 37 R-sq 0.79 Note:*** 0.01 significance level, ** 0.05 significant level, and * 0.10 significance level The regression results also show that a letter grade increase in GPA increases the econometrics course grade by more than a letter grade, approximately 13.32 percentage points. Given that the average grades for most econometrics courses are in the low C range, the effect of a change in GPA on the econometrics grade is substantial. Not surprisingly, student motivation measured by absences is an important determinant of course grade. Students who missed more than two classes show a decrease in course grade of about 5.3 percentage points, dropping a student half a letter grade. Previous introductory economics courses or lower level math courses had no significant effect on the econometrics course grade. 5 A student who took higher levels of calculus, however, realized an increase of more than 3 percentage points in overall course grade (p < 0.10). Other Benefits and Costs of the Project The most obvious benefit of formative assessment is the significant improvement in student course grades. In addition to improvement in course grades, students are also encouraged to send their papers to undergraduate conferences and publications. In the instructors judgment, six papers were worthy of conference presentations out of a class of twelve students in the fall of 2009. Only one of twelve student papers in 2007 and three of thirteen papers in 2008 were judged as good enough for submission. In order to make a conference presentation possible, students are encouraged to take the course during their junior year, thus increasing the likelihood 5 A student receiving an overall course grade of A/A- was the dummy variable one, and not introductory course taken since all students were required to take both introductory courses for econometrics. 10 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 of conference presentations and submissions for publication in their senior year. In the fall of 2010, four of seven junior-standing students who took the course in 2009 sent their papers to the Southern California Conference for Undergraduate Research (SCCUR). Prior to 2009, only one student had presented at SCCUR. Students are also encouraged to expand their econometrics papers into their senior project. Two of the twelve students from 2007 and seven of the thirteen from 2008 choose to turn their econometrics paper into their thesis, whereas eleven out of the twelve in 2009 choose or intend to turn their econometric papers into their senior thesis. From faculty evaluations, some students have indicated they view the work as worthy of an eight-unit course (econometrics and senior project), instead of a four-unit course, in order to justify the demanding work required of them. In addition, students who do not show up to the periodic meetings could be denied expansion of their project into a senior project. This incentive is especially effective if the same instructor teaches both courses. The incentive to expand the econometrics project into a senior project course is a powerful motivator that allows for successful periodic monitoring and helps justify students’ work load. The costs associated with this project are the loss of breadth of material, the time- constraint for the instructor, and structural problems with the implementation of such a large- scale project. The implementation of a multi-step project with continuous feedback comes at a cost of loss of breadth in econometric topics. This problem can be remedied by expanding the curriculum to two semester courses in econometrics. The first semester would cover probability distribution, introduction to regression, classical two-variable regression model; properties of estimators; hypothesis testing, multiple regression, estimation, hypothesis testing, multicollinearity; specification error, heteroscedasticity, and dummy variables. The second semester would cover topics such as alternate functional forms, autocorrelation, qualitative choice and limited dependent variable models, simultaneous equations, and topics in time-series analysis and forecasting. This enables the instructor to include extensive formative assessment as well as to cover all relevant topics. A second possibility is to institute an applied econometrics course and a theoretical econometrics course, with the latter requiring more stringent mathematics. Secondly, there are time constraints for the instructor. The implementing the projects and meeting with students after each stage is very time consuming. The benefit of student learning from this process outweighs the cost of time to faculty. In addition, specialization will diminish the marginal cost of implementation to the instructor over time. Other than meeting with students individually, the project will require less time from the instructor each additional year the course is taught. Moreover, if the same instructor teaches the senior project course in the following term, the work of advising students on a project for that course is significantly reduced. Structurally, the project requires smaller class sizes, teaching the course in the fall term, a statistical computer program for student to access such as SPSS, and institutional membership in a data archive organization such as ICPSR. Other than smaller class sizes, most colleges can implement the other structural changes at a very reasonable cost. Teaching the course in the fall allows the instructor to meet with the students at the end of the spring term. This in turn enables students to use the summer to look for surveys on topics that are of interest to them. Smaller class sizes can be a challenge, although not necessarily a structural impediment for most liberal arts colleges. Other colleges could restrict the course to juniors and seniors and only economics majors as a possible solution. 11 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 Concluding Remarks Classroom experimentation with teaching the course with and without formative assessment will continue for the next several years. In addition to classroom experiments in econometrics, future research may result as student outcomes from formative assessment experiments in the introductory microeconomics course are collected. Incorporating the project into the econometrics course yielded useful results. First, the instructor included extensive formative assessment throughout the course. Second, students more fully engaged the material and raised their level of performance in the course. Given these results, I argue that formative assessment was the principle means of raising student performance in my introductory econometrics course from 2007 and 2008 to 2009. The success of incorporating layers of formative assessment is evident in the improvement in overall course grade and the increased quality of papers written by students. The unmeasured benefit to this instructor is the ability to engage students throughout the semester. Moreover, the ability of students with diverse math and analytical skill levels to master the theoretical concepts in econometrics and to apply them to real world situations has substantially reduced this instructor’s frustration in teaching econometrics. References Becker, William, and Watts, Michael. 1996. “Chalk and Talk: A National Survey of Teaching Undergraduate Economics.” American Economic Review. 86(2): 448-54. Becker, William and Greene, William. 2001. “Teaching Statistics and Econometrics to Undergraduates”. The Journal of Economic Perspective, 15(4): 169-182. Black and Wiliam 1996. “Meanings and Consequences: A Basis for Distinguishing Formative and Summative Functions of Assessment?” British Educational Research Journal. 22(5): 537-548. Black, Paul and Wiliam, Dylan. 1998. “Assessment and Classroom Learning,” Assessment in Education. 5(1): 7-74. Black, Paul and Wiliam, Dylan. 1998. “Inside the Black Box: Raising Standards though Classroom Assessment.”Phi-Delta-Kappa., 80(2): 139-148. Black, Paul and Wiliam, Dylan. 2003. “In Praise of Educational Research: Formative Assessment,” British Educational Research Journal. 29(5): 623-637. Bloom, Madaus, and Hastings.1971. Evaluation to Improve Learning. McGraw-Hill Book Company. Coe, Robert. 2002. Paper from Conference of British Educational Research Association. University of Exeter, England. September 12 – 14. Cohen, J. 1969. “Educational outcomes of tutoring: A meta-analysis of findings.”American Educational Research Journal, 19: 237-248. Cohen, J., Kulik, J.A. and Kulik, C.C. 1988. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, HJ: Lawrence Earlbaum Associates. Faulk, Dagney. 2007. “Formative and Summative Assessment in Economics Principles Courses: Are Applied Exercises Effective? Prepared for 2008 Annual Meeting of the American Economic Association, January 4-6, New Orleans, LA. Glass, McGraw and Smith, 1981. Meta-Analysis in Social Research. Sage Publications Inc. Newbury Park, CA, London, England, and New Delhi. Ramaprasad, A. 1983. “On the definition of feedback.” Behavioral Sciences, 28: 4-13. 12 | JOURNAL FOR ECONOMIC EDUCATORS, 11(1), SUMMER 2011 Rosenthal, Robert and Rubin D.B. 1982. “A simple, general purpose display of magnitude of experimental effect.” Journal of Educational Psychology, 74: 66-169. Rosenthal, Robert. 1984. Meta-Analysis Procedures for Social Research. Sage Publications, Beverly Hills, London, New Delhi. Scriven, M. 1967. The methodology of evaluation. In R. W.Tyler, R. M. Gagne, and M Scriven (Eds.), Perspectives of curriculum evaluation. Vol. I: 39-83. Chicago, IL: Rand McNally. Walstad, William. 2005. Teaching Economics: More Alternatives to Chalk and Talk. Becker, William, coedited with Michael Watts and Suzanne Becker. Cheltenham UK: Edward Elgar Publishing, LTD, chapter 11. Wilson, Mark and Scalise, Kathleen. 2006. “Assessment to Improve Learning in Higher Education: The BEAR Assessment System.” Higher Education. 52(4): 635-663. Yorke, Mantz. (2003). “Formative Assessment in Higher Education: Moves towards Theory and the Enhancement of Pedagogic Practice.” Higher Education. 45(4): 477-501. Appendix Grade Requirements in 2007 and 2008 Grade Requirements in 2009 Midterm Exam 25% Midterm Exam 20% Comprehensive Final Exam 35% Comprehensive Final Exam 35% Quizzes /Homework 10% Quizzes/Homework 10% Paper 25% Paper 30% Attendance 5% Attendance 5% .