Graduate Students Journal of Psychology Copyr Graduate Student Journal of Psychology Copyright 2004 by the Department of Counseling & Clinical Psychology 2004, Vol. 6 Teachers College, Columbia University ISSN 1088-4661 A Taxometric Analysis of Experimenter-Induced Response Bias on Self-Report Data Jessica Neubauer and Robert E. McGrath Fairleigh Dickinson University Taxometric analysis was developed to help determine whether a latent variable best conforms to a dimensional or categorical (taxonic) model. One study (Beauchaine & Waters, 2003) found that the taxonic structure of ratings could be influenced through instructional set. Their findings raise questions about whether taxometric analysis is necessarily identifying the organic structure of rat- ings-based latent variables. However, the study was limited to ratings of target individuals un- known to the rater. The present study was conducted to determine whether self-ratings are equally susceptible to manipulation. Undergraduate students were asked to complete a battery of four self- report personality measures which previous research indicated may tap a common latent variable. Participants randomly received an instructional set that implied either a taxonic or dimensional structure for self-ratings. The results suggest that self-ratings may be more resistant to instructional set than ratings of others, as data from both groups conformed to a dimensional model rather than a taxonic structure. cut, and the means are then plotted. The resulting graph will vary in shape depending on whether the underlying construct is continuous or discontinuous. Specifically, a Psychologists have long recognized that many psycho- logical measures offer, at best, a rough approximation of the latent variables they are intended to represent (Cronbach & Meehl, 1955). The very structure of psychological con- structs is often not understood, making it impossible, for example, to determine whether a dimensional or categorical model provides a more accurate representation of these constructs. One consequence of such disparities is that psy- chologists develop measurement devices without a clear conceptualization of their underlying construct. Determin- ing the underlying structure of a psychological construct is one step toward making it a more tangible entity. Taxomet- ric procedures (Meehl & Yonce, 1994, 1996; Waller & Meehl, 1998) are a set of statistical analyses developed for the purpose of determining whether a latent variable of in- terest demonstrates an inherent dimensional or categorical structure when multiple indicators of that latent variable are available. Several taxometric methods are currently avail- able, including Mean Above Minus Below a Cut (MAM- BAC), Maximum Eigenvalue (MAXEIG), and Latent Mode (Lmode). latent taxonic structure will be represented in a graph by an inverted U, whereas a dimensional structure will manifest as U-shaped. As a consistency test for the results, the pro- cedure can be repeated reversing the input and output vari- ables. When more than two indicators are available, it is possible to conduct a MAMBAC analysis twice for each possible pair. The MAXEIG method also requires that one indicator be designated the input variable, but it uses two or more other indicators as output variables. Again, the input vari- able is sorted. Subsamples are created based on equally spaced intervals along the input variable, and the eigen- value, based on a factor analysis of the output variables, is calculated within each of these subsamples. In this case, the plot of eigenvalues as a function of interval should re- sult in an inverted U shape for data that conforms to a taxonic model, whereas data that conforms to a dimensional model will be represented by a U shape. The procedure can be repeated using each available indicator as the input vari- able. MAMBAC involves designating one indicator as the input variable and a second indicator as the output variable. The input variable is sorted, and a series of “cuts” are then made at each value along the input variable. The mean of the output variable is calculated, both above and below each Latent dimensional or categorical structures can also be identified graphically when research data are subjected to a third taxometric procedure known as Lmode. This proce- dure is based on a factor analysis of all available indicators, with the purpose of generating essentially error-free factor scores. A unimodal frequency distribution of these scores is indicative of an underlying dimensional structure, whereas two or more peaks suggest a taxonic structure. If enough indicators are available to allow selection of sub- Jessica Neubauer and Robert E. McGrath, School of Psychology, Fairleigh Dickinson University. Correspondence concerning this article should be addressed to Jessica Neubauer, School of Psy- chology T-WH1-01, Fairleigh Dickinson University, 1000 River Road, Teaneck, NJ 07666-1914. E-mail: JessNeubauer@aol.com 32 TAXOMETRICS AND RESPONSE STYLE sets, multiple Lmode graphs can be generated from subsets of indicators as a consistency check. Studies have found taxometric analysis superior to al- ternative procedures, such as cluster analysis, as an indica- tor of taxonic structure (Cleland, Rothschild, & Haslam, 2000; Waller & Meehl, 1998). The method has been used to evaluate the latent structure of various psychological constructs, including eating disorders (Tylka & Subich, 2003), sexual orientation (Gangestad, Bailey, & Martin, 2000; Haslam, 1997), antisocial behavior (Skilling, Quinsey, & Craig, 2001), and depression (Ruscio & Ruscio, 2000). For example, using five non-behavioral indicators, Tylka and Subich (2003) found consistent evidence of a dimensional structure across clinical eating disorders. Such results highlight the value of a statistical method that incor- porates procedures for evaluating the consistency of con- clusions about taxonicity. Though numerous studies have demonstrated that the application of taxometric analyses may enhance under- standing of psychological constructs, questions still remain as to how viable the results of such analyses are when they are based on data that has been collected via rating scales. A recent article by Beauchaine and Waters (2003) raises concerns about the degree to which the outcomes of taxometric analyses can be influenced by response set. Here, undergraduates were asked to evaluate the quality of graduate school admission essays based on six criteria. Raters were informed either that the “essays reflected a full range of graduate school performance,” or that they were “written by students who either struggled or excelled in graduate school” (Beauchaine & Waters, 2003, p. 6). The authors found consistent evidence that the struc- ture of the ratings differed as hypothesized based on in- structional set. Analyses of ratings by individuals who were instructed that the essays’ authors reflected a full range of performance produced dimensional results, whereas those of raters instructed that authors either strug- gled or excelled produced categorical results. As a result, Beauchaine and Waters questioned whether taxonic struc- ture can be manipulated through pre-existing expectancies. They concluded that “these results have potential implica- tions for future taxometrics research and suggest that a sole reliance on rating-scale data may be ill-advised when test- ing taxonic hypotheses” (Beauchaine & Waters, 2003, p.10). In fact, though, their results are only relevant to ratings of others who are unknown to the raters. Therefore, it is uncertain whether their conclusions apply when the rater has already developed an impression of the target prior to receiving the instructions. Specifically, the application of their conclusions remains unclear when both the rater and target are the same individual, as is the case in self-report. Since the studies listed above all used self-report as the basis for drawing conclusions about taxonicity, the present study was conducted to determine if the response bias de- tected by Beauchaine and Waters (2003) extends to self- report rating scales as well. The results have important implications concerning the degree to which taxometric studies that involve self-report measures are reflecting in- herent structure or rater perceptions. Method Participants A total of 115 undergraduate students at Fairleigh Dickinson University completed a battery of five self-report personality inventories. Among those who provided demo- graphic data, 70.8% (75) were females and 29.2% (31) were males. The sample was 49% Caucasian, 25% African American, 9.6% Hispanic, 9.6% Asian, 1% Native Ameri- can, and 5.8% Other. The mean age of the respondents was 20.47 years of age (SD = 4.39), with a range from 17 to 40. Procedure Participants were recruited from introductory psychol- ogy and freshman seminar courses and asked to complete four personality measures. These consisted of Rosenberg’s (1965) Self-Esteem Scale, the neuroticism scale of the Ey- senck Personality Inventory (Eysenck & Eysenck, 1968), the Core Self-Evaluations Scale (Judge, Erez, Bono, & Thoresen, 2003), and a generalized self-efficacy scale (Judge, Erez, Bono, & Thoresen, 2002). These scales were chosen because Judge and colleagues (2002) have provided evidence that these measures all tap a common construct, despite the differences in their names. Participants were randomly assigned to two groups. All participants were given standard instructions for the completion of self-report measures, with one exception. The Taxonic Instructions group consisted of 54 participants (47%) who were told, “Previous studies suggest that most people either produce very high or very low scores on these types of scales.” The Dimensional Instructions group con- sisted of 61 participants (53%) who were told, “Previous studies suggest that people produce a wide range of scores on these types of scales, from low to medium to high.” Results When analyzed, Pearson correlation coefficients sug- gest that the measures utilized correlate relatively well with one another, suggesting they are tapping a shared construct. The correlations between measures ranged from 0.32 to 0.71; each was significant at p < .001 (see Table 1). The data from the Taxonic Instructions group and the Dimensional Instructions group were subjected to three taxometric procedures: MAMBAC, MAXEIG, and Lmode. Without exception, each procedure yielded results indicat- ing that the responses given conformed to a dimensional structure regardless of instructional set. Graphs generated from the MAMBAC and MAXEIG analyses failed to depict the inverted U shape characteristic of an underlying taxonic structure in either group. Similarly, the Lmode graphs for 33 NEUBAUER AND MCGRATH Discussion Discussion both samples were unimodal, again failing to detect a taxon (see Figures 1 and 2). Each taxometric method allows for estimation of the taxonic base rate if a taxon exists. A high degree of vari- ability in these estimates across analyses provides a secon- dary piece of evidence supporting a dimensional conclu- sion. In the Taxonic Instructions group, the mean estimated base rate from the MAMBAC analyses was 0.50, while the mean based on the MAXEIG analyses was 0.77. Lmode allows two independent estimates of taxon base rate. In the taxonic instructions group, these estimates were .06 and 1.00. These results suggest substantial variability in esti- mates of the taxon base rate, a finding that supports the conclusion that the distribution of scores in the Taxonic Instructions group conform to a dimensional structure. The results of the current study suggest that responses to self-report rating scales may not be as subject to manipu- lation via instructional set as Beauchaine and Waters (2003) suggest. Even when given instructions suggesting a taxonic style of responding, students responded in a dimensional manner. The similarity in the style of responding across the two conditions supports the continued use of self-report rating scales for the purpose of assessment. The results of the current study suggest that responses to self-report rating scales may not be as subject to manipu- lation via instructional set as Beauchaine and Waters (2003) suggest. Even when given instructions suggesting a taxonic style of responding, students responded in a dimensional manner. The similarity in the style of responding across the two conditions supports the continued use of self-report rating scales for the purpose of assessment. It is important to note, however, that the sample size available for this pilot study is insufficient to justify firm conclusions about the absence of taxonic structure. Data continue to be gathered, and more extensive results will be made available at a future date. It is important to note, however, that the sample size available for this pilot study is insufficient to justify firm conclusions about the absence of taxonic structure. Data continue to be gathered, and more extensive results will be made available at a future date. References References Table 1 Beauchaine, T. P., & Waters, E. (2003). Pseudotaxonicity in MAMBAC and MAXCOV analyses of rating-scale data: Turning continua into classes by manipulating ob- server’s expectations. Psychological Methods, 8, 3-15. Beauchaine, T. P., & Waters, E. (2003). Pseudotaxonicity in MAMBAC and MAXCOV analyses of rating-scale data: Turning continua into classes by manipulating ob- server’s expectations. Psychological Methods, 8, 3-15. Correlations between Pairs of Scales. RSE EPI EPI CSE CSE GSE GSE RSE RSE 1.00 1.00 EPI EPI 0.32 0.32 1.00 1.00 CSE CSE 0.71 0.71 0.51 0.51 1.00 1.00 GSE GSE 0.71 0.71 0.48 0.48 0.70 0.70 1.00 1.00 Note. RSE = Rosenberg Self-Esteem Scale; EPI = Neuroticism Scale of the Eysenck Personality Inventory; CSE = Core Self- Evaluations Scale; GSE = General Self-Efficacy Scale. All corre- lations are significant, p < .001. Note. RSE = Rosenberg Self-Esteem Scale; EPI = Neuroticism Scale of the Eysenck Personality Inventory; CSE = Core Self- Evaluations Scale; GSE = General Self-Efficacy Scale. All corre- lations are significant, p < .001. Cleland, C. M., Rothschild, L., & Haslam, N. (2000). De- tecting latent taxa: Monte Carlo comparison of taxomet- ric, mixture model, and clustering procedures. Psycho- logical Reports, 87(1), 37-47. Cleland, C. M., Rothschild, L., & Haslam, N. (2000). De- tecting latent taxa: Monte Carlo comparison of taxomet- ric, mixture model, and clustering procedures. Psycho- logical Reports, 87(1), 37-47. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281- 302. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281- 302. Eysenck, H. J., & Eysenck, S. B. G. (1968). Manual of the Eysenck Personality Inventory. San Diego, CA: Educa- tional and Industrial Testing Service. Eysenck, H. J., & Eysenck, S. B. G. (1968). Manual of the Eysenck Personality Inventory. San Diego, CA: Educa- tional and Industrial Testing Service. -3 -1 0 1 2 3 4 0. 0 0. 2 0. 4 Factor Scores R el at iv e Fr eq ue nc y LMODE CURVE Gangestad, S. W., Bailey, J. M., & Martin, N. G. (2000). Taxometric analyses of sexual orientation and gender identity. Journal of Personality and Social Psychology, 78, 1109-1121. Gangestad, S. W., Bailey, J. M., & Martin, N. G. (2000). Taxometric analyses of sexual orientation and gender identity. Journal of Personality and Social Psychology, 78, 1109-1121. Haslam, N. (1997). Evidence that male sexual orientation is a matter of degree. Journal of Personality and Social Psychology, 73, 862-870. Haslam, N. (1997). Evidence that male sexual orientation is a matter of degree. Journal of Personality and Social Psychology, 73, 862-870. Judge, T. A., Erez, A., Bono, J. E., & Thoresen, C. J. (2002). Are measures of self-esteem, neuroticism, locus of control, and generalized self-efficacy indicators of a common core construct? Journal of Personality and So- cial Psychology, 83, 693-710. Judge, T. A., Erez, A., Bono, J. E., & Thoresen, C. J. (2002). Are measures of self-esteem, neuroticism, locus of control, and generalized self-efficacy indicators of a common core construct? Journal of Personality and So- cial Psychology, 83, 693-710. Figure 1. L-mode results for Taxonic Instructions group. Figure 1. L-mode results for Taxonic Instructions group. Judge, T. A., Erez, A., Bono, J. E., & Thoresen, C. J. (2003). The Core Self-Evaluations Scale: Development of a measure. Unpublished manuscript, University of Florida. Judge, T. A., Erez, A., Bono, J. E., & Thoresen, C. J. (2003). The Core Self-Evaluations Scale: Development of a measure. Unpublished manuscript, University of Florida. Figure 2. L-mode results for Dimensional Instructions group. Figure 2. L-mode results for Dimensional Instructions group. -3 -1 0 1 2 3 4 0. 0 0. 2 0. 4 Factor Scores R el at iv e Fr eq ue nc y LMODE CURVE Meehl, P. E., & Yonce, L. J. (1994). Taxometric analysis: I. Detecting taxonicity with two quantitative indicators using means above and below a sliding cut (MAMBAC procedure). Psychological Reports, 74, 1059-1274. Meehl, P. E., & Yonce, L. J. (1994). Taxometric analysis: I. Detecting taxonicity with two quantitative indicators using means above and below a sliding cut (MAMBAC procedure). Psychological Reports, 74, 1059-1274. Meehl, P. E., & Yonce, L. J. (1996). Taxometric analysis: II. Detecting taxonicity using covariance of two quantita- tive indicators in successive intervals of a third indicator Meehl, P. E., & Yonce, L. J. (1996). Taxometric analysis: II. Detecting taxonicity using covariance of two quantita- tive indicators in successive intervals of a third indicator 34 TAXOMETRICS AND RESPONSE STYLE 35 (MAXCOV procedure). Psychological Reports, 78, 1091-1227. Rosenberg, M. (1965). Society and the adolescent self- image. Princeton, NJ: Princeton University Press. Ruscio, J. & Ruscio, A. M. (2000). Informing the continu- ity controversy: A taxometric analysis of depression. Journal of Abnormal Psychology, 109, 473-487. Skilling, T. A., Quinsey, V. L., & Craig, W. M. (2001). Evidence of a taxon underlying serious antisocial behav- ior in boys. Criminal Justice & Behavior, 28, 450-470. Tylka, T. L., & Subich, L. M. (2003). Revisiting the latent structure of eating disorders: Taxometric analyses with nonbehavioral indicators. Journal of Counseling Psy- chology, 50, 276-286. Waller, N. G., & Meehl, P. E. (1998). Multivariate taxometric procedures: Distinguishing types from con- tinua. Thousand Oaks, CA: Sage Publications, Inc. A Taxometric Analysis of Experimenter-Induced Response Bias on Self-Report Data Participants Procedure