Microsoft Word - wilsonzietz11.29.04d.doc JOURNAL FOR ECONOMIC EDUCATORS • Volume 4 • Number 4 • Fall 2004 13 SYSTEMATIC BIAS IN STUDENT SELF-REPORTED DATA By Mark L. Wilson and Joachim Zietz* Abstract The study analyzes the extent to which student self-reported data are biased and what variables can predict the degree of the bias. A variable that students feel more sensitive about is compared in terms of reporting bias to other less sensitive variables. The reporting bias is significant only for the sensitive variable. The study explains the reporting bias for the sensitive variable on the basis of student characteristics. The study uses a data set that consists of about 450 individual college student records. The study’s results have implications for the analysis of survey data outside of economic education. (JEL A22, C81) Introduction Data collected by questionnaire used to be rather infrequently employed in economics. Over the last decade or two, however, the use of survey data has risen significantly in economics (Boulier and Goldfarb 1998). One possible explanation for this trend is the many advances in econometric technique related to survey data, such as qualitative and limited dependent variable models (Maddala 1986) and panel data models (Hsiao 2003). Unfortunately, survey data raise issues beyond those that can be resolved by the appropriate choice of econometric technique, such as how to convert qualitative responses into numerical form (Nardo 2003) or how to assess the accuracy of survey responses (Bound et al. 2001; Bertrand and Mullainathan 2001). This paper is concerned with data validity—in particular, systematic differences between true and self-reported data for a key variable, grade point average (GPA), commonly used to estimate educational production functions. Several studies in (economic) education have noted problems with self-reported grades (e.g., Sawyer et al. 1989). Some have examined this issue further. For example, Maxwell and Lopus (1994) suggest that the survey respondent’s type of school, by Carnegie classification, is a statistically significant predictor of inaccurate self-reports. In the related educational psychology literature, Cassady (2001) finds that low-performing students overreport performance measures, such as GPA, more than high-performing students. This result supports earlier evidence uncovered by Dobbins et al. (1993) that students tend to inflate their past performance scores to a level that they consider socially acceptable or desirable. 1 While informative, the previous work in this area does not typically use regression analysis to identify the key determinants of the reporting bias or use variables that are commonly available in studies on economic education. Hence, there is little practical guidance researchers in economic education can derive from the results on reporting bias in applied work. * Mark L. Wilson, West Virginia Insurance Commission, mark.wilson@wvinsurance.gov; Joachim Zietz, Middle Tennessee State University, jzietz@mtsu.edu. 1 According to prior work in applied psychology (e.g., Greenwald 1980), such behavior is motivated by the desire of low performers to protect their self-esteem. By contrast, the newly developing literature within economics on social interaction models would see in such behavior an example of a local norm of behavior that can evolve in optimizing models of choice behavior with direct social interactions among agents (Glaeser and Scheinkman 2002). JOURNAL FOR ECONOMIC EDUCATORS • Volume 4 • Number 4 • Fall 2004 14 This paper extends the earlier work on bias in the self-reporting of GPA in a number of directions. First, the self-reporting bias in GPA is compared to that in other variables that students may feel less sensitive about, such as age. Second, the paper identifies with the help of regression analysis some key determinants of the systematic reporting bias in GPA in terms of student characteristics that are typically available in studies in economic education. Knowledge of these determinants could provide researchers in economic education with valuable information on adjusting the data to obtain more reliable estimation results. Third, the paper predicts the reporting bias and thereby provides some simple guidance on when to expect a significant self-reporting bias. The paper is organized as follows. The following section briefly discusses the data. The next section reports the estimation results. The paper ends with a brief summary of the results and some conclusions. Data The data are derived from an end-of-semester student evaluation instrument given to a group of principles of economics students at a large, public, comprehensive university. The survey solicited information about the students’ evaluation of the instructor and several pieces of biographical information. Although students identified themselves, they were assured that the instrument was “to be used for research purposes only.” A total of about 450 students completed the survey. Because the data were collected in the context of a student evaluation of the teacher, there was no reason for students to expect their biographical information would be tested for credibility. The survey results were matched with the administrative records of each student. For the purpose of this study it is assumed that the administrative records are accurate. Grade point average is the sensitive variable that is being analyzed. This variable is widely used to identify student aptitude (e.g., Lopus and Maxwell 1995) and effort (e.g., Stratton et al. 1994). Age and gender are two presumably less sensitive variables with which GPA is compared. The age variable has been shown to influence student learning (e.g., Marlin and Niss 1980; Seiver 1983). The gender variable has been widely employed in the estimation of student performance (e.g., Watts and Bosshardt 1991). Empirical Analysis The first stage of the empirical analysis consists of simple correlations of actual and self-reported data for the three variables Age, gender, and GPA. The results are provided in Table 1. The small discrepancy in the Age data is consistent with the widespread tendency to understate one’s age. But the understatement is very slight, and the self-reports are almost perfectly correlated with the actual values. With regard to gender, every student answered correctly, which suggests that all participants took the survey seriously. One can conclude that there does not appear to be a noteworthy reporting bias for the two less sensitive variables. However, the results for the more sensitive grade point data are materially different. GPA values are overstated by a substantial margin, and the correlation between reported and actual values is much less than unity. The results of Table 1 confirm earlier studies that the reporting bias for the more sensitive variable warrants further analysis. First, matched pairs are used to test the hypothesis that the self-reported GPA is equal to the administrative value. The result of this test confirms that there is a systematic overstatement of self-reported grades (t-value = 11.3; p-value = 0.000). Second, in a regression of self- reported GPA on both a constant and actual GPA, the joint null hypothesis of the constant being equal to zero and the slope coefficient being equal to unity, which is what we would expect under the null hypothesis of no bias, is rejected at very high levels of statistical significance (F-value (2,447) = 109.4, p-value = 0.000). Hence, one may conclude that there is a systematic tendency to overstate JOURNAL FOR ECONOMIC EDUCATORS • Volume 4 • Number 4 • Fall 2004 15 GPA values, which is consistent with the results of previous studies (Goldman et al. 1990; Dobbins et al. 1993; Maxwell and Lopus 1994; Cassady 2001). Table 1. Means and Correlation Coefficients (449 observations) Self-Report Administrative Record Correlation Coefficient Age 22.3 22.5 0.99 Male = 1 0.55 0.55 1.0 GPA 2.92 2.73 0.85 The follow-up question, which appears not to have been addressed in the literature, is whether one can predict the reporting bias for such sensitive data as GPA with variables that are commonly available in economic education studies. If one could indeed predict the reporting bias, it may be possible to reduce the measurement problem in quantitative studies that utilize these kinds of data. Based on earlier work (Dobbins et al. 1993; Cassady 2001), it appears that the GPA reporting bias is the highest for students with the lowest grades. This would suggest that a student’s actual GPA, or some other similar measure of academic achievement, is a key explanatory variable for GPA overstatement. Figure 1 illustrates the negative relationship between GPA overstatement and GPA. It is apparent from this scatter plot that GPA overstatements far outnumber GPA understatements. There is also an obvious negative relationship between GPA overstatement and GPA. Hence, the difference between self-reported GPA and actual GPA does not appear to be the result of a random process, but it looks to be systematic in nature. JOURNAL FOR ECONOMIC EDUCATORS • Volume 4 • Number 4 • Fall 2004 16 Table 2: Regression Results Explaining the Difference between Self-Reported and Actual GPA (overreport) Variables Model 1 Model 2 Model 3 Model 4 Model 5 Constant 1.044 3.334 3.213 3.322 3.537 (12.8) (10.4) (10.5) (10.5) (10.9) GPA -.313 -2.818 -2.824 -2.915 -3.269 (-11.6) (-6.6) (-7.1) (-7.2) (-7.8) GPA 2 .844 .857 .889 1.047 (4.7) (5.1) (5.2) (5.9) GPA 3 -.089 -.092 -.095 -.116 (-3.7) (-4.1) (-4.2) (-4.9) Age .002 .002 .001 (0.8) (0.6) (0.4) Male -.056 -.054 -.065 (-2.3) (-2.3) (-2.9) Expected .074 .069 .056 Grade (4.1) (3.8) (3.4) Actual -.038 -.039 -.024 Grade (-1.8) (-1.8) (-1.2) Freshman -.074 -.071 (-2.1) (-2.1) R 2 .3474 .4441 .4698 .4742 .4715 P-Values: LM-Het .000 .137 .163 .156 White-Het .000 .000 .000 .001 JB .000 .000 .000 .000 Reset .000 .533 .301 .120 Notes: T-values are in parentheses. For Models 1 to 4, they are based on White’s heteroskedasticity consistent variance- covariance matrix. Probability values (p-values) below 0.05 indicate a statistical problem. LM-Het and White-Het refer to a Lagrange Multiplier and White’s test for heteroskedasticity, respectively. JB is Jarque-Bera’s test for normality, and Reset is Ramsey’s test for functional form. Regression analysis provides a more comprehensive assessment of the determinants of GPA overstatement because it allows one to look at more than just one determining variable of GPA overstatement. In addition, statistical significance can be established. A first regression model specifies the difference between self-reported and actual GPA (overreport) as a linear function of GPA. The results of this regression are presented as Model 1 of Table 2. The estimated slope JOURNAL FOR ECONOMIC EDUCATORS • Volume 4 • Number 4 • Fall 2004 17 coefficient verifies the negative relationship between GPA overstatement and GPA. From a statistical point of view, however, the equation is unsatisfactory. There is evidence of significant heteroskedasticity and non-normal residuals, and the Reset test identifies a problem with functional form. The functional form problem is resolved if GPA is allowed to enter the regression with higher- order terms. This is evident from the results of Model 2 in Table 2. The Reset test is no longer significant. However, heteroskedasticity is still a significant problem based on White’s heteroskedasticity test. 2 Next, a number of determining variables in addition to GPA are added to the regression model (Model 3). These include the age of the student (Age), the student’s gender (Male), 3 the student’s expected course grade (Expected Grade), and the student’s actual end-of-semester course grade (Actual Grade). If low-performing students inflate their reported levels of past performance to what they consider socially acceptable levels, they are likely to do the same with expected levels of future performance. If this reasoning is correct, the coefficient of the variable Expected Grade should be statistically significant and positive. In the same vein, one would expect the end-of-semester grade to be negatively related to inflated self-reports on GPA because good students do not need to inflate their self-reports to protect their self-esteem or to achieve a level of academic achievement that they deem socially desirable or acceptable. The estimation results for this expanded set of variables are collected in the column labeled Model 3 in Table 2. The results confirm that the degree of GPA overstatement is not only dependent on GPA but also on other student characteristics. In particular, male students tend to overstate their GPA slightly less on average than female students, a result that contrasts with some earlier work by Goldman et al. (1990), which identifies males as inflating their GPA more than females. As suggested above, a higher expected grade induces a more significant GPA overstatement, whereas a higher actual course grade is associated with a less pronounced GPA overstatement on average. A student’s age, by contrast, appears to have no statistically significant influence on GPA overstatement. Model 4 adds one more explanatory variable to Model 3, the student’s semester standing in terms of being freshman or not. 4 Goldman et al. (1990) report that freshmen tend to inflate their grades less than students with more advanced standing. This result is replicated in Model 4 as the coefficient of the variable Freshman is negative and statistically significant. Model 5 addresses the problem of heteroskedasticity that appears to affect Models 1 through 4. To accomplish this, Model 4 is re-estimated using a weighted least squares technique based on maximum likelihood principles. The resulting model allows the variance of the error term [var (ut)] to be a function of an arbitrary value. After some experimentation, the best-fitting model turned out to make the variance of the error term a linear function of the dependent variable (overreport): var (ui) = .057 + .029 overreport, (9.1) (4.4) where the numbers in parentheses are t-values. The estimated regression coefficients for the weighted least squares specification are given as Model 5 of Table 2. They differ only slightly from the ordinary least squares estimates of Model 4, which suggests that the estimates are rather robust not only with respect to the inclusion/exclusion of variables but also with respect to the estimation method. 2 The t-values reported in Table 2 are based on White’s heteroskedasticity-consistent variance-covariance matrix. 3 Male students are coded as 1, females as 0. 4 The variable is coded 1 for freshman and 0 for all other students. JOURNAL FOR ECONOMIC EDUCATORS • Volume 4 • Number 4 • Fall 2004 18 Since actual GPA is by far the most important determinant of GPA overstatement, the relationship between GPA and GPA overstatement is depicted in Figure 2. The graph is constructed from the estimates of Model 4 by setting the variables Age, Expected Grade, and Actual Grade equal to their sample averages. The indicator variables Male and Freshman are set at unity, which means that the graph depicts the relationship between GPA overstatement and GPA for male freshman students. For female students past their freshman year, GPA tends to be overstated more, and the corresponding curve would be above that depicted in Figure 2. Figure 2 not only confirms the nonlinear and negative relationship between GPA overstatement and GPA but also suggests that, on average, students misrepresent their GPA enough to ensure that their self- reported GPAs do not fall much below 2.25. This would suggest that self-reported GPA data below 2.25 are significantly biased and possibly not very useful for statistical work that assumes the absence of measurement error, such as ordinary least squares regression. In the given sample, about a third of the data points fall below this threshold. Summary and Conclusions This paper has verified with data collected for the purpose of students’ evaluations of teachers that more sensitive data, such as grade point average, are self-reported with a greater bias than less sensitive data, such as age and gender. Also, an attempt has been made to predict the reporting bias for the sensitive variable. Regression analysis reveals a number of variables that can explain the reporting bias for grade point average, among them a polynomial in grade point average, gender, and expected grade. The results confirm earlier work by other authors that grade point average overstatement is inversely related to actual grade point average. Students with grade point averages below about 2.3 appear on average to inflate their GPA in self-reports to at least this level. This result should be useful for empirical studies in economic education that utilize self-reported grade point average as a variable. JOURNAL FOR ECONOMIC EDUCATORS • Volume 4 • Number 4 • Fall 2004 19 In particular, it suggests that self-reports of grade point averages for students below about 2.3 are likely to be significantly biased and should probably not be utilized without modification. The methodology employed in this paper to identify the determinants of self-reporting bias and the point at which the reporting bias may become a major concern for statistical analyses should also be of interest to studies in other areas of economics where survey data are utilized and where there is a need to reduce measurement error in essential variables. References Bertrand, Marianne, and Sendhil Mullainathan. 2001. “Do People Mean What They Say? Implications for Subjective Survey Data.” American Economic Review (May): 67-72. Boulier, Bryan L., and Robert S. Goldfarb. 1998. “On the Use and Nonuse of Surveys in Economics.” Journal of Economic Methodology (June): 1-21. Bound, John, Charles Brown, and Nancy Mathiowetz. 2001. “Measurement Error in Survey Data.” In Handbook of Econometrics, Vol. 5, edited by James J. Heckman and Edward Leamer. North- Holland: 3705-3843. Cassady, Jerrell C. 2001. “Self-Reported GPA and SAT: A Methodological Note,” Practical Assessment, Research & Evaluation 7(12). Available online at: http://pareonline.net/ getvn.asp?v=7&n=12. Dobbins, Gregory H., Jiing-Lih Farh, and James D. Werbel. 1993. “The Influence of Self-Monitoring and Inflation of Grade-Point Averages for Research and Selection Purposes.” Journal of Applied Social Psychology: 321-334. Glaeser, Edward L., and Jose A. Scheinkman. 2002. “Non-market Interactions.” In: Advances in Economics and Econometrics: Theory and Applications: Eighth World Congress, edited by M. Dewatripont, L.P. Hansen, and S. Turnovsky. Cambridge: Cambridge University Press. Goldman, Bert A., Wesley L. Flake, and M.B. Matheson. 1990. “Accuracy of College Students’ Perceptions of Their SAT Scores, High School and College Grade Point Averages Relative to Their Ability.” Perceptual and Motor Skills: 514. Greenwald, Anthony G. 1980. “The Totalitarian Ego: Fabrication and Revision of Personal History.” American Psychologist: 603-618. Hsiao, Cheng. 2003. Analysis of Panel Data. Cambridge University Press. Lopus, Jane S., and Nan L. Maxwell. 1995. “Teaching Tools: Should We Teach Microeconomic Principles before Macroeconomic Principles?” Economic Inquiry (April): 336-350. Marlin, James W., and James F. Niss. 1980. “End of Course Evaluations as Indicators of Student Learning and Instructor Effectiveness.” Journal of Economic Education (Spring): 16-27. Maddala, G.S. 1986. Limited-Dependent and Qualitative Variables in Econometrics. Cambridge University Press. Maxwell, Nan L., and Jane S. Lopus. 1994. “The Lake Wobegon Effect in Student Self-Reported Data.” American Economic Review (May): 201-206. Nardo, Michela. 2003. “The Quantification of Qualitative Survey Data: A Critical Assessment,” Journal of Economic Surveys (December): 645-668. Sawyer, Richard, Joan Laing, and Walter Houston. 1989. “Accuracy of Self-Reported High School Courses and Grades of College-Bound Students.” College and University (Spring): 288-299. Seiver, Daniel. 1983. “Evaluations and Grades: A Simultaneous Framework.” Journal of Economic Education.” (Summer): 32-38. Stratton, Richard W., Stephen C. Myers, and Randall H. King. 1994. “Faculty Behavior, Grades, and Student Evaluations.” Journal of Economic Education (Winter): 5-15. Watts, Michael, and William Bosshardt. 1991. “How Instructors Make a Difference: Panel Data Estimates from Principles of Economics Courses.” Review of Economics and Statistics (May): 336-340.