NODA Journal 22_2.indd SPRING 2015 • VOLUME 22, NUMBER 2 5 Non-Cognitive Assessment in Higher Education: Social Desirability and the Prediction of College Outcomes A.J. Metz, Qin Hu, Alexandra R. Kelly, Andrew R. Fox, David Shirley, and Laken Shirley Non-cognitive assessment is used to identify at-risk college students and leverage limited resources to promote academic performance and persistence. Instruments that measure these psychosocial attitudes and skills require self-reported responses and, thus, may be subject to distortion. This study examined the social desirability response bias in a specific non-cognitive assessment tool, the Student Strengths Inventory (SSI), including gender and ethnic differences. Results show that college students did not respond to the SSI in a socially desirable way. Additionally, the SSI subscales contributed to significant variance in the prediction of academic performance and persistence. This study empirically supports the use of non-cognitive assessment in higher education and suggests interventions for using non-cognitive assessment data at the individual, group, and aggregate level. Introduction Institutions of higher education continue to struggle with retaining and graduating their students. Recognizing a need to connect with students early in the first term, colleges and universities have begun to implement non-cognitive assessment to measure students’ academic skills, attitudes, behaviors, and commitment to staying in school. Research has demonstrated the significance of these factors in identifying at-risk students and predicting their college outcomes (Casillas, et al., 2012; Robbins, et al., 2006). In fact, non-cognitive assessment may serve as a foundation for proactive outreach and targeted interventions. However, because non-cognitive assessment tools typically require students to self-report their responses, the results may be subject to response distortion (Davis, Thake, FEATURED ARTICLE A.J. Metz (AJ.Metz@utah.edu) is an Assistant Professor in Counseling Psychology Program at the University of Utah Qin Hu is a Doctoral Student in Counseling Psychology at the University of Utah Alexandra R. Kelly is a Doctoral Student in Counseling Psychology at the University of Utah Andrew R. Fox is a Doctoral Student in Clinical Psychology at Seattle Pacific University David Shirley is a Doctoral Student in Counseling Psychology at the University of Utah Laken Shirley is a Doctoral Student in Counseling Psychology at the University of Utah 6 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION & Vilhena, 2010). If students respond in a socially desirable manner, it may undermine the predictive validity of these measures, decrease the effectiveness of interventions built around these measures, and more importantly, fail to identify at risk-students. Thus, it is important to determine if non-cognitive assessment tools are subject to socially desirable responding. The following introduction provides a background for this study by highlighting retention statistics, describing non-cognitive assessment in higher education, summarizing the empirical support for these measures, and explaining the nature and impact of socially desirable responding. Retention and Graduation Rates National retention rates continue to demonstrate the need to identify students at risk for stopping out, dropping out, or otherwise leaving. The ACT Institutional Data File (2013) reports national first-to-second-year retention rates based on degree offerings and institutional selectivity. With respect to degree offerings, two-year public institutions have the lowest first-to-second-year retention rates (55%) compared to two-year private institutions (58%), four-year public BA/BS degree-granting institutions (64.9%), and four-year private BA/BS degree granting institutions (67.3%). Public and private institutions that offer graduate degrees have slightly and moderately higher first-to-second-year retention rates (68.9% to 77.7% and 69.5% to 81.3% respectively). Institutional selectivity continues to impact retention rates such that those that admit students with higher ACT/SAT scores and higher class rank demonstrate greater first–to-second-year retention rates. For example, four-year public institutions with “highly selective” admissions standards (the middle 50% have ACT scores of 25-30, and the majority are admitted from the top 10% of their high school class) had a 90% first-to-second- year retention rate, compared to a 71% retention rate for those with “traditional” admissions standards (the middle 50% have ACT scores of 18-24, and the majority are admitted from the top 50% of their high school class) and a 57% retention rate for those with open enrollment (ACT, 2013). National graduation rates also demonstrate the need to intervene to promote academic success and persistence. Aud, et al., (2013) reported that only 57% of first-time, full-time students at four-year public institutions completed their bachelor’s degrees within six years. The six-year graduation rates were slightly higher at four-year private nonprofit institutions (65%) but lower at four-year private for-profit institutions (42%). The graduation rates at two-year degree- granting institutions are even more disturbing. Only 20% of full-time, first-time undergraduate students who pursued a certificate or associate’s degree at a two- year public institution completed their degree within 150 percent of the normal time required. With respect to differences based on gender and culture, women complete their degrees at higher rates than men (Aud et al., 2013), and Asian and White students complete their four-year and two-year degrees at higher rates than Black, Hispanic, or Pacific Islander students (U.S. Department of Education, NCES, 2013). SPRING 2013 • VOLUME 20, NUMBER 2 7 Effects of Student Attrition The effects of student attrition are significant and far reaching. Schneider and Yin (2011) calculated that between 2003 and 2008, the federal government provided over $1.5 billion in grants to students who dropped out or stopped out of college after their first year. During this period of time, individual states provided almost $6.2 billion to fund college students who never returned for their second year (Schneider & Yin, 2011). In one year, colleges and universities across the US spent more than $263 billion on education and related expenses (Johnson, 2012). Johnson provided a breakdown of this cost by degree. He calculated that for students in the Beginning Postsecondary Students (BPS) Longitudinal Study cohort from 2003-2009, the average (mean) education and related costs per student were $16,127 for a certificate program, $33,856 for an Associate’s degree, and $53,781 for a Bachelor’s degree. The counterpoint is to look at the fiscal benefits of retaining students through graduation. Based on the average cost of tuition and fees charged to full-time, in-state undergraduate students, the average public 4-year university will experience a net gain of $35,572 for each student retained for four years (not including tuition inflation). This does not include room and board, which averages approximately $9,598 per year (College Board, 2014). Gardner (1981) anticipated the need for a “college completion agenda” over 20 years ago: “the student has become a precious commodity. Institutions must now concern themselves with retaining students so that, if nothing else, budgets can be preserved” (p. 79). His observations are even more salient today, given the national agenda of college and career readiness and the increased competition among postsecondary institutions. Although the institutional costs associated with student attrition are easily calculated using standard accounting practices, the costs for an individual are more difficult to quantify but equally, if not more, devastating. Economically, it is hard to argue against the benefits of a college education. Human capital theory is often used to describe the relations between education and income and suggests that employers will pay a higher wage to more educated and skilled workers. The Bureau of Labor Statistics (BLS) confirms this theory. For example, the 2013 median weekly earnings for those with a bachelor’s degree were 55% higher than those with a high school diploma (BLS, 2014). College graduates can expect to earn approximately $1 million dollars more over a lifetime than a high school graduate (Julian, 2012). In 2013, the unemployment rate for those with a Bachelor’s degree was almost half of the unemployment rate for high school graduates (BLS, 2014). Higher levels of education are related to lower levels of incarceration and use of social safety-net programs and higher levels of civic participation including voting, volunteerism, and blood donation (Baum & Payea, 2004). Those with more schooling are also more likely to exercise, obtain preventive care, and exhibit lower health risk factors associated with smoking, drinking, diet/exercise, illegal drug use, hypertension, and diabetes (Cutler & Lleras-Muney, 2006). It is clear, education benefits the individual and society. It is not surprising, therefore, that colleges and universities are investing 8 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION considerable resources in developing retention strategies. These strategies include extended orientation programs, supplemental instruction, first-year experience seminars, peer mentoring, and a host of enhancements to student development, student life, and advising programs. Further, institutions are purchasing enrollment management services such as those provided by the Noel Levitz Group and utilizing enrollment management assessment instruments such as the National Survey of Student Engagement (NSSE; Indiana University, 2002) and the Cooperative Institutional Research Program Freshman Survey (Higher Education Research Institute, 2013). A relatively recent strategy is to adopt psychometrically sound instruments that assess non-cognitive psychosocial and risk factors among incoming students, administer those instruments during orientation or early in the first semester, and then use data provided by those instruments to align campus resources with the specific weaknesses and strengths of incoming students. Non-Cognitive Assessment in Higher Education Prior to discussing non-cognitive assessment, it is important to operationalize the term. Traditionally, institutions of higher education have used “cognitive” or academic factors such as high school GPA, standardized test scores, and high school rank to predict student success. The term “non-cognitive factor” has more recently been used to refer to the motivational, psychological, and social variables that are also related to student academic functioning. To clarify the distinction between cognitive and non-cognitive factors, Sommerfeld (2011) suggested replacing the term “cognitive” with “academic” to refer to factors based on formal education (e.g., grades) and replacing the term “non-cognitive” with “non- academic” to refer to factors that help a student adapt, cope, and follow through with academic activities (e.g., personality, attitudes, study habits, goal setting, and commitment). Over the years, theories and models have been proffered to explain how non- cognitive or non-academic factors influence college success and retention (e.g., Astin, 1993; Bean, 1985; Pascarella, 1985; Terenzini & Reason, 2005; Tinto, 1993;). More recently, researchers have begun to empirically distill the most important psychosocial and skill factors from this conceptual base in order to develop psychometrically strong assessment tools that would identify students at risk for academic failure and drop out. For example, Robbins et al. (2004) conducted a meta-analysis of 109 studies and found that academic-related skills, academic self- efficacy, and academic goals were the best predictors of college student retention. Other significant predictors of retention included institutional commitment, social support, institutional selectivity, and social involvement. Results also supported the predicted relationships between non-cognitive factors and college grades. The best predictor of GPA was academic self-efficacy. Other significant predictors of grade point average included achievement motivation, financial support, academic goals, academic-related skills, social involvement, institutional commitment, and social support. In addition to investigating the individual effects of non-cognitive factors on SPRING 2015 • VOLUME 22, NUMBER 2 9 retention and performance, Robbins et al. (2004) were interested in determining whether the combined effects of non-cognitive factors could account for variance in retention and performance over and above that accounted for by traditional predictors (cognitive factors) such as high school GPA and ACT/SAT score. They found that cognitive predictors accounted for approximately 9% of the variance in retention, non-cognitive factors accounted for 21% of the variance, and when combined, 23% of the variance in retention could be explained. With respect to GPA, cognitive factors accounted for 25% of the variance, non-cognitive factors accounted for 27% of the variance, and when combined, 34% of the variance in GPA could be explained (Robbins et al., 2004). These results confirm the contribution non-cognitive factors play in college student retention and performance above and beyond that of traditional cognitive predictors. There are many examples of non-cognitive assessment measures used in higher education: the College-Readiness Performance Assessment System (C-PAS; Conley, Lombardi, Seburn, & McGaughy, 2009), the Student Readiness Inventory (SRI; Le, Casillas, Robbins, & Langley, 2005), the College Student Inventory (CSI; Stratil, 2009), the Student Strengths Inventory (SSI; Gore, Leuwerke, Brown, & Metz, 2009), and the Non-Cognitive Questionnaire (NCQ; Sedlacek & Brooks, 1976). Although each assesses a different combination of non-cognitive factors, the data can be used in similar ways. For example, postsecondary institutions can use non-cognitive assessment information to create precise at-risk prediction equations to identify the students most likely to fail academically and most likely to drop out. Non-cognitive assessment information can be further used to direct students to interventions and programming that fit their unique needs. In addition to identifying at-risk students, data can be used to discover student strengths that could be capitalized on through participation in student leadership positions or peer mentorship opportunities. Finally, non-cognitive assessment information can be used to predict budget models and inform enrollment management objectives and strategies. Socially Desirable Responding It is beneficial not only for the individual, but for the institution to accurately identify at-risk students and connect them with appropriate resources and support. A well-known validity threat inherent in all self-report measures, but particularly troublesome in the assessment of non-cognitive factors, is social desirability. The social desirability bias refers to the tendency of an individual to answer questions in a way that presents the individual in a favorable manner to secure the approval of others (Davis, Thake, & Vilhena, 2010; Mesmer-Magnus, Viswesvaran, Deshpande, & Joseph, 2006). For example, there is research to show that individuals may overestimate desirable traits and behaviors like donating to charities and underestimate socially undesirable traits and behaviors such as alcohol and drug use (Groves, 1989). If students respond to non-cognitive assessment items in a socially desirable manner (reflecting social expectations rather than actual behavior), it may undermine the predictive validity of the 10 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION instrument and the effectiveness of interventions built around it (Davis et al., 2010; Mesmer-Magnus et al., 2006; Thornton & Gupta, 2004). By responding in a socially desirable manner, students might not be identified as at-risk and might not be targeted for intervention. Without intervention, they might slip through the cracks and fail academically or drop out. Thus, the social desirability response bias is of particular concern when attempting to develop and use non-cognitive assessment tools. There is a large body of work documenting the social desirability response bias in the social and behavioral sciences (Crowne & Marlowe, 1960; Edwards, 1957; Paulhus, 1991). More specifically, socially desirable responding has been evidenced in self-report measures assessing attitudes (Fisher, 1993), behaviors (Mensch & Kandel, 1988), goal orientation (Tan & Hall, 2005), and personality variables (Mick, 1996). Mesmer-Magnus et al. (2006) found that individuals with high emotional intelligence and high self-esteem were more likely to engage in socially desirable responding. This is perhaps due to the ability of individuals with high emotional intelligence to read social cues and adjust to “socially appropriate” responses. An alternative explanation is that individuals with high self-esteem possess positive self-conceptions, and they may consciously or unconsciously generalize some of these positive attributions to all aspects of their lives. Several studies also found that women engaged in higher levels of socially desirable responding than men (Bernardi, 2006; Bernardi & Guptill, 2008). For example, in one study, socially desirable responding caused women to report more ethical responses to decision-making scenarios than men (Dalton & Ortegren, 2011). At the same time, another study found that the effect of gender on social desirability was attenuated by religiousness (Chung & Monroe, 2003). Women who reported greater religiosity had higher social desirability scores than less religious women or men in general (regardless of religiousness). Finally, women were more likely to display socially desirable responding when self-report measures were administered face-to-face with a same sex interviewer (Thornton & Gupta, 2004). Dalton and Ortegren (2011) point to gender role socialization as an explanation for why women may be more susceptible to the social desirability response bias than men. Other studies, however, have failed to find socially desirable responding in self-report measures. In four separate studies, Erdle and Rushton (2011) found that neither self-esteem nor social desirability impacted a general factor of personality as measured by the Big Five Inventory (BFI). This finding suggests that personality is not an artifact of self-evaluation (as would be reflected by the presence of socially desirable responding) but rather an accurate and substantive representation of an individual’s true characteristics. Similarly, Peterson, Casillas, and Robbins (2006) found little to no evidence of socially desirable responding in either the Big Five Inventory or the Student Readiness Inventory, a measure of non-cognitive strengths and weaknesses. At the same time, it would be difficult to generalize this result, given the sample lacked diversity with respect to gender (73.3% female) and race and ethnicity (83.8% Caucasian). Thus, although there is some evidence to suggest a lack of socially desirable responding in non-cognitive assessment, more research SPRING 2015 • VOLUME 22, NUMBER 2 11 is needed to understand gender and cultural differences. The purpose of the present study was to answer three research questions: 1) Do college students respond to the Student Strengths Inventory (SSI), a measure of non-cognitive strengths and weaknesses, in a socially desirable manner? 2) Are there gender or ethnic differences in socially desirable responding? 3) Can the SSI predict academic performance and persistence? We hypothesized that college students would not respond to the SSI in a socially desirable manner. However, given previous research showing that women are more prone to socially desirable responding than men, we expected gender differences. We made no hypotheses regarding racial and ethnic group differences as research in this area was minimal and typically addressed racially charged topics. We further predicted that the subscales of the SSI would predict academic performance as measured by cumulative GPA and persistence as measured by students’ intentions to continue their undergraduate education. Method Participants Participants were 645 undergraduate students enrolled in a large research university in the Mountain West. The majority were underclassmen (60%), enrolled full-time (89.7%) and female (67.2%). Of those self-reporting ethnicity, 69.1% identified as European American, 16% as Latina/o, 7% as Asian American, 1.9% as Black or African American, 1.1% as Native Hawaiian or Pacific Islander, and 0.8% as American Indian or Alaskan Native; 4% endorsed more than one ethnic background. This sample was representative of the institution’s undergraduate population with respect to ethnic diversity but included a slightly higher percentage of women and full-time students. Measures Student Strengths Inventory. The Student Strengths Inventory (SSI; Gore, Leuwerke, Brown, & Metz, 2009) is a 48-item face-valid self-report measure that assesses six non-cognitive, or psychosocial, factors found to have the greatest impact on student success outcome variables. Using a six-point Likert scale ranging from 1 (“strongly disagree”) to 6 (“strongly agree”), participants respond to items describing behaviors or characteristics related to the following six domains: Academic Engagement (e.g. “I turn homework in on time”), Academic Self-Efficacy (e.g. “I will succeed in my chosen major”), Campus Engagement (e.g. “Being active in extra-curricular activities is important to me”), Educational Commitment (e.g. “I see value in completing a college education), Resiliency (e.g. “I manage stress well”), and Social Comfort (e.g. “I am comfortable in groups”). Respondents’ raw scores, which range from 8 to 48 on each scale, were used for analyses. Internal consistency reliability estimates for the six scales ranged from .77 to .90 in college 12 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION samples of over 8,000 students (Gore, Leuwerke, Brown, & Metz, 2009). Moreover, there is evidence of construct validity for the SSI as there were strong correlations between SSI scale scores and similar scale scores on the SRI (Le, Casillas, Robbins, & Langley, 2005). Social desirability. The Marlowe-Crowne Social Desirability Scale (MCSDS; Crowne & Marlowe, 1960) is a 33-item questionnaire comprised of a series of too-good-to-be-true self-statements that individuals endorse as true or false (e.g. “I have never intensely disliked anyone”). Scores are interpreted such that higher scores are indicative of a greater desire to present oneself in a socially desirable light. Crowne and Marlowe (1960) reported an internal reliability estimate of .88 (KR-20) and a test-retest correlation of .89. Persistence intentions. Participants’ commitment to their education and plans for continuing in their undergraduate programs were assessed through six rationally developed, first-person items that were rated on a five-point Likert-type scale ranging from 1 (“strongly agree”) to 5 (“strongly disagree”). Three items reflecting a high likelihood of persistence (e.g. “It is likely that I will register at this university next Fall”) were reverse scored, while three were not (e.g. “I have no idea what I want to major in”). The result was a total score for each respondent ranging from 6 to 30, with higher values reflecting a greater commitment to persisting in college. Procedure Participants were recruited through a research subject pool (58%) or campus flyers (42%). The subject pool is advertised to students in undergraduate courses in a department of educational psychology (e.g., all First-Year Experience courses) as a way to participate in research and earn research credit. In some cases, research participation is a mandatory component of the class; in other cases, it is extra credit. Students access the subject pool online, read brief descriptions about research studies, and accept or decline participation. Anyone who agreed to participate in our study was directed to an online survey; once finished, they were assigned 30 minutes of research credit (corresponding to the average length of time it took pilot participants to complete the survey.) In addition to the subject pool, flyers posted around the campus advertised a web link for our study. Students could type the link into their web browser, review the informed consent document, and agree or decline to participate. Those who agreed to participate were directed to an online survey; once finished, they had the opportunity to be entered into a drawing to win one of four $50 gift cards. Analyses Descriptive statistics, Pearson’s zero-order correlations, and partial correlations were calculated for all study variables. Separate univariate analyses of variance (ANOVA) were conducted to examine gender and cultural differences in study SPRING 2015 • VOLUME 22, NUMBER 2 13 variables. Multiple linear regression was used to explore the relationship between participants’ GPA, persistence intentions, SSI subscales, and social desirability scores. Treating GPA and persistence intentions as the dependent variables in separate hierarchical regression analyses, social desirability was entered as the first independent variable, followed by the six SSI variables. Results Descriptive statistics. Table 1 provides the score range, mean, and standard deviation for each study variable. It is important to note that the mean GPA was 3.2, and the mean score on a measure of persistence intentions was 26.47, indicating that the students in our sample performed well academically and expected to graduate from the college they were enrolled in at the time. The mean score for the social desirability measure was somewhat lower in this sample than scores obtained in recent research (Peterson, Casillas, & Robbins, 2006). Gender and cultural differences in major study variables were assessed by conducting a series of univariate analyses of variance (ANOVA). Significant gender differences were found in GPA, persistence intentions, and three SSI subscales. Specifically, women had higher GPAs, persistence intentions, and scores on Educational Commitment and Academic Engagement. Men scored higher on the Resiliency subscale than women. To examine cultural differences, individual racial and ethnic categories were dichotomized into a minority (N = 152) and non-minority (N = 351) group. Combining diverse racial and ethnic groups into one “minority” group is not preferred, but it did allow for additional analyses. Specifically, subsequent univariate ANOVAs revealed significant cultural differences in three SSI subscales. Non-minority participants had higher scores on Academic Self-Efficacy, Social Comfort, and Educational Commitment than their ethnic minority peers. Correlations. Table 2 presents zero-order correlations of the six SSI subscales and self-reported cumulative GPA. Academic Engagement, Academic Self-Efficacy, and Educational Commitment were significantly related to GPA (r = .22, .26, and .11, respectively). Table 3 presents zero-order correlations of the six SSI subscales and persistence intentions. Academic Engagement (.20), Academic Self-Efficacy (.26), Social Comfort (.22), Resiliency (.12), and Educational Commitment (.30) were all significantly related to students’ intentions to persist in college. Partial correlations. To assess the effects of social desirability, partial correlations were calculated. Tables 2 and 3 provide adjusted correlations between the SSI subscales and the outcome variables of GPA and persistence intentions. Zero-order correlations are displayed on the left, and correlations with social desirability partialed out are in the middle. To determine the effect of social desirability, the partial correlations were subtracted from the zero-order correlations. For GPA, partial correlations ranged from 0 to .01. For persistence intentions, partial correlations ranged from 0 to .02. Overall, the partial correlations demonstrate that social desirability has almost no impact on the relationship between the SSI and academic performance and persistence. These 14 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION results were replicated for men and women and minority and non-minority groups. Hierarchical regression. Two hierarchical linear regression analyses were performed to confirm results of the partial correlations and further demonstrate the predictive power of the SSI scales with two college outcomes – academic performance, as measured by cumulative GPA, and retention, as measured by persistence intentions. A two-step hierarchical regression model was used, with social desirability entered in step one as a control variable and the SSI subscales entered in the second step. Social desirability accounted for 0% of the variance in GPA, and the SSI subscales accounted for an additional 13% of the variance [F (7, 526) = 10.63, p<.001]. In the regression predicting retention, social desirability accounted for 0% of the variance, and the SSI subscales accounted for an additional 12% of the variance [F (7, 526) = 9.7, p<.001]. As there were significant gender differences in some of the major study variables, additional hierarchical regression analyses were performed separately for women and men. For women, a combination of SSI subscales accounted for 17% of the variance in GPA [F (6, 352) = 11.48, p<.001]. Examination of the standardized regression coefficients reveals that Academic Engagement, Academic Self-Efficacy, Social Comfort, Resiliency, and Educational Commitment were significant predictors of GPA for this group. For men, the SSI subscales accounted for 10% of the variance in GPA [F (6, 172) = 3.17, p<.01]. Academic Self- Efficacy was the only significant predictor of GPA for men. Similar analyses were conducted using persistence intentions as the dependent variable. For women, 15% of the variance in persistence intentions was accounted for by SSI subscales [F (6, 354) = 10.01, p<.001]. Social Comfort and Educational Commitment were significant individual predictors of persistence for this group. For men, 9% of the variance in persistence intentions was accounted for by SSI subscales [F (6, 170) = 2.56, p<.05], but none of the SSI subscales were individually predictive. Additional hierarchical regression analyses were also performed to examine cultural differences in the relationship between SSI subscales and college persistence and performance. For ethnic minority participants, 13% of the variance in GPA was accounted for by SSI subscales [F (6, 127) = 2.94, p<.01]. Review of the standardized regression coefficients showed that Academic Engagement and Academic Self-Efficacy were significant predictors of GPA for minority participants. For non-minority participants, 15% of the variance in GPA could be accounted for by SSI subscales [F (6, 318) = 8.86, p<.001]. Campus Engagement, Academic Engagement, Academic Self-Efficacy, Resiliency, and Educational Commitment were all significant predictors of GPA for non-minority participants. These analyses were then replicated using persistence intentions as the dependent variable. For ethnic minority participants, 17% of the variance in persistence intentions was accounted for by SSI subscales [F (6, 126) = 4.02, p<.001]. Review of the standardized regression coefficients demonstrates that Academic Engagement was a significant predictor of persistence for minority participants. For non-minority participants, 14% of the variance in persistence intentions could be accounted for by SSI subscales [F (6, 318) = 8.32, p<.001]. Social Comfort and Educational SPRING 2015 • VOLUME 22, NUMBER 2 15 Commitment were significant predictors of persistence for non-minority participants. Discussion Overall, this study demonstrated that the SSI, a non-cognitive assessment instrument, is not vulnerable to socially desirable responding. Researchers used both partial correlations and hierarchical regression to examine this potential validity threat in the entire study sample and then in subsamples to explore differences based on gender and culture. Although women scored slightly higher than men on a measure of social desirability, the difference was not statistically significant. With respect to cultural differences, it was the non-minority participants who had slightly higher scores on social desirability than their ethnic minority counterparts, although this difference was not statistically significant either. Our study findings are not consistent with research demonstrating that women engage in higher levels of socially desirable responding than men (Bernardi, 2006; Bernardi & Guptill, 2008; Chung & Monroe, 2003; Dalton & Ortegren, 2011). However, the topics under investigation in these studies (e.g., ethical decision- making) and the methods by which social desirability was measured (e.g., face-to- face with interviewer) lacked consistency and differed dramatically from the topic of non-cognitive assessment. At the same time, our findings are consistent with research using another non-cognitive measure to predict academic performance in college students (Peterson, Casillas, & Robbins, 2006). Thus, our finding that the SSI is not subject to response distortion provides another key piece of support for the ability of non-cognitive assessment tools to be used in higher education. Non-Cognitive Factors Related to College Outcomes Similar to previous research examining non-cognitive assessment and college student outcomes (Robbins et al., 2004; Robbins, et al., 2006), a combination of non-cognitive factors contributed to the prediction of academic performance and persistence. However, the relative influence of the non-cognitive factors varied by gender and race and ethnicity. For example, the SSI subscales were better able to predict GPA and persistence intentions for women in our sample (17% and 15% of the variance respectively) versus men (10% and 9% of the variance respectively). Additionally, most of the SSI subscales individually contributed to the prediction of women’s GPA, while for men, only academic self-efficacy was significant. The results for women are consistent with previous research; the results for men are not. With respect to gender differences in persistence intentions, social comfort and educational commitment were significant predictors of women’s persistence intentions, while none of the SSI subscales significantly predicted men’s persistence intentions. Again, the results for women are consistent with previous research; the results for men are not. It is interesting that no individual non-cognitive factors 16 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION contributed to the prediction of men’s persistence intentions, when a combination of these factors contributed to 9% of the overall variance in persistence intentions. It may be that there is something about the combination of these factors or that there are additional factors more influential to men’s persistence intentions than the six non-cognitive factors captured in the SSI. Thus, more research is required to better understand these findings. Given the small number of ethnic minority students in our sample and the way in which we dichotomized culture into ethnic minority and non-minority groups, generalization of this set of results is limited but still warrants discussion. First, there were fewer cultural differences in the prediction of GPA and persistence intentions for ethnic minority (13% and 17%) and non-minority participants (15% and 14%). However, differences were apparent in specific non-cognitive factors that contributed to the prediction of college outcomes. For example, while most of the SSI subscales individually contributed to the prediction of non-minority participants’ GPAs (as is consistent with previous research), only Academic Engagement and Academic Self-Efficacy were significant predictors of ethnic minority students’ GPAs. In the prediction of persistence intentions, Social Comfort and Educational Commitment were significant individual predictors for non-minority participants (similar to the results of our female sample), while only Academic Engagement was significant for ethnic minority participants. Academic Engagement seems to be a significant predictor for ethnic minority participants, which is not surprising as it has been found to be the top predictor of academic performance and persistence in previous studies (Robbins et al., 2006). Implications for Practice Non-cognitive assessment can be used at the individual, group, and aggregate levels to promote academic success and retention in college. At the individual level, results can be used to provide feedback on student strengths and weaknesses. For example, an academic advisor could use non-cognitive assessment results to help a student establish academic goals, develop an action plan, and identify relevant campus resources. Additionally, an advisor could use non-cognitive assessment results to help students take advantage of their strengths by engaging in campus leadership opportunities, serving as a peer mentor, or getting involved in service learning programs. Broader conversations might include an examination of how an individual’s strengths and weaknesses are likely to impact course performance and college persistence intentions. Individual feedback and action planning can take place in any context in which the student has a meaningful relationship with an adult mentor or advisor (e.g., athletics, TRIO, and the Multicultural Student Center). Non-cognitive assessment results can also be used to engage students in conversations about career planning and transition. University career counselors can use the results to help students identify their strengths and translate those strengths into resume elements or help students learn how to talk about those strengths during a job interview. Non-cognitive assessment results can also be used in group settings on the SPRING 2015 • VOLUME 22, NUMBER 2 17 college campus. First-year experience courses that focus on helping students develop study strategies and better understand the requirements for college success seem to be an excellent platform for integrating non-cognitive assessment. In fact, most of the textbooks used in courses such as these have chapter titles that align well with non-cognitive factors (e.g., motivation, goal setting, action planning, and getting involved on campus). Of course, there are several additional contexts in which college personnel can utilize non-cognitive assessment results. Orientation leaders or residential advisors can be trained to help students interpret their non- cognitive assessment results and develop action plans for growth. Finally, given the applicability of non-cognitive factors to success in the workplace, career centers that offer workshops or group career counseling can use results from non-cognitive assessment to help students discuss their strengths and weaknesses in an interview setting. The group environment provides valuable opportunities for mock interview practice and peer feedback. Finally, non-cognitive assessment results can be aggregated to provide administrators with data to understand the overall strengths and weaknesses of an incoming cohort and make strategic institutional decisions about where additional resources might be directed and additional support programs developed. Some non-cognitive assessment platforms can further provide administrators with success probability metrics to inform their early warning systems. Limited in resources, institutions are often forced to choose which group of at-risk students to target with outreach and intervention efforts. With non-cognitive assessment results, institutions might have an easier time making these decisions. For example, an institution might elect to target students in the middle of the risk-distribution where it believes it has the potential to have the biggest impact on retention. Limitations and Directions for Future Research This study provided support for the use of non-cognitive assessment in predicting academic success and retention in higher education. Given this sample was collected at a large, public four-year institution with little ethnic diversity, the findings might not generalize to other types of colleges (e.g., community colleges or small liberal arts schools) or students of ethnically diverse backgrounds. Given the intersection of race and ethnicity and social class, further research should include ethnically diverse populations and measures of socioeconomic status. Additionally, future research on the predictive utility of the SSI (or other non- cognitive assessment tools) should be studied longitudinally to assess cumulative GPA in subsequent semesters or years and retention as measured by actual enrollment in subsequent terms versus intentions to return to school. Finally, additional research should employ the use of interventions designed to strengthen the understanding of the directional influence between non-cognitive variables and academic outcomes. 18 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION References ACT Institutional Data File. (2013). ACT national collegiate retention and persistence to degree rates. Iowa City, IA: ACT, Inc. Retrieved fromhttp://www.act.org/ research/policymakers/reports/graduation.html Astin, A. W. (1993). What matters in college: Four critical years revisited. San Francisco, CA: Jossey-Bass. Aud, S., Wilkinson-Flicker, S., Kristapovich, P., Rathburn, A., Wang, X., &Zhang, J. (2013). The condition of education 2013 (NCES 2013-037). U.S. Department of Education, National Center for Education Statistics. Washington, DC. Retrieved from http://nces.ed.gov/programs/coe/. Baum, S., & Payea, K. (2005). Education pays: The benefits of higher education for individuals and society. New York, NY: College Board. Bean, J. P. (1985). Interaction effects based on class level in an explanatory model of college student dropout syndrome. American Educational Research Journal, 22, 35-64. Bernardi, R. A. (2006). Associations between Hofstede’s cultural constructs and social desirability response bias. Journal of Business Ethics, 65, 45-53. Bernardi, R. A., & Guptill, S. T. (2008). Social desirability response bias, gender, and factors influencing organizational commitment: An international study. Journal of Business Ethics, 81, 797-809. Bureau of Labor Statistics, U.S. Department of Labor. (2014). Usual weekly earnings of wage and salary workers: First quarter 2014. Retrieved from http://www.bls. gov/news.release/pdf/wkyeng.pdf Casillas, A., Robbins, S., Allen, J., Kuo, Y., Hanson, M. & Schmeiser, C. (2012). Predicting early academic failure in high school from prior academic achievement, psychosocial characteristics, and behavior. Journal of Educational Psychology, 104, 407-420. Chung, J., & Monroe, G. S. (2003). Exploring social desirability bias. Journal of Business Ethics, 44, 291-302. College Board. (2014). Trends in higher education: Average published undergraduate charges by sector, 2013-2014. Retrieved from http://trends.collegeboard.org/ college-pricing/figures-tables/average-published-undergraduate-charges- sector-2013-14 Conley, D., Lombardi, A., Seburn, M., & McGaughy, C. (2009). Formative assessment for college readiness: Measuring skill and growth in five key cognitive strategies associated with postsecondary success. Paper presented at the annual conference of the American Educational Research Association, San Diego, CA. Crowne, D. P. & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349-354. Cutler, D. M., & Lleras-Muney, A. (2006, June). Education and health: Evaluating theories and evidence. National Bureau of Economic Research Working Paper 12352. Retrieved from: http://www.nber.org/papers/w12352 Dalton, D., & Ortegren, M. (2011). Gender differences in ethics research: The importance of controlling for the social desirability response bias. Journal of Business Ethics, 103, 73-93. SPRING 2015 • VOLUME 22, NUMBER 2 19 Davis, C. G., Thake, J., & Vilhena, N. (2010). Social desirability biases in self- reported alcohol consumption and harms. Addictive Behaviors, 35(4), 302-311. Erdle, S., & Rushton, J. (2011). Does self-esteem or social desirability account for a general factor of personality (GFP) in the Big Five? Personality and Individual Differences, 50, 1152-1154. Fisher, R. J. (1993). Social desirability bias and the validity of indirect questioning. Journal of Consumer Research, 20(2), 303-315. Gardner, J. N. (1981). Developing faculty as facilitators and mentors. New Directions for Student Services, 14, 67-80. doi: 10.1002/ss.37119811407 Gore, P. A., Leuwerke, W., Brown, S. D., & Metz, A. J. (2009). Development and initial validation of a measure of non-cognitive/psychosocial factors in a college sample. (Unpublished manuscript). University of Utah. Salt Lake City, UT Groves, G. M. (1989). Survey errors and survey costs. New York, NY: Wiley. Higher Education Research Institute. (2013). Cooperative Institutional Research Program (CIRP) freshman survey. Los Angeles, CA: HERI. Retrieved from http:// www.heri.ucla.edu/researchers/instruments/CIRP/2014-CIRP-Freshman-Survey. pdf Johnson, N. (2012). The institutional costs of student attrition (Research Paper). American Institutes for Research. Retrieved from http://www.deltacostproject. org/sites/default/files/products/Delta-Cost-Attrition-Research-Paper.pdf Julian, T. (2012, October). Work-life earnings by field of degree and occupation for people with a bachelor’s degree: 2011. American Community Survey, US Census Bureau. Retrieved from http://www.census.gov/prod/2012pubs/acsbr11-04.pdf Le, H., Casillas, A., Robbins, S. B., & Langley, R. (2005). Motivational and skills, social, and self-management predictors of college outcomes: Constructing the Student Readiness Inventory. Educational and Psychological Measurement, 62, 1-28. Mensch, B. S., & Kandel, D. B. (1988). Underreporting of substance use in a national longitudinal youth cohort. Public Opinion Quarterly, 25, 100-124. Mesmer-Magnus, J., Viswesvaran, C., Deshpande, S., & Joseph, J. (2006). Social desirability: The role of over-claiming, self-esteem, and emotional intelligence. Psychology Science, 48, 336-356. Mick, D. G. (1996). Are studies of dark side variables confounded by socially desirable responding? The case of materialism. Journal of Consumer Research, 23(2), 106-119. National Survey of Student Engagement (2013). A fresh look at student engagement: Annual results 2013.University Center for Postsecondary Research and Planning. Retrieved from http://www.nsse.iub.edu/NSSE_2013_Results Pascarella, E. T. (1985). College environmental influences on learning and cognitive development: A critical review and synthesis. In J. Smart (Ed.), Handbook of theory and research in higher education (Vol. 1; pp. 1-61). New York, NY: Agathon. Paulhus, D. L. (1991). Measurement and control of response bias. In J.P. Robinson, P.R. Shaver, & L.S. Wrightsman (Eds). Measures of personality and social psychological attitudes, Vol. 1 (pp. 17-59). San Diego, CA: Academic Press. 20 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION Peterson, C. H., Casillas, A., & Robbins, S. B. (2006). The Student Readiness Inventory and the Big Five: Examining social desirability and college academic performance. Personality and Individual Differences, 41, 663-673. Robbins, S. B., Allen, J., Casillas, A., Peterson, C., & Le, H. (2006). Unraveling the differential effects of motivational and skills, social, and self-management measures from traditional predictors of college outcomes. Journal of Educational Psychology, 98, 598-616. Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288. Schneider, M. & Yin, L. M. (2010). The high cost of low graduation rates: How much does dropping out of college really cost? Washington, DC: American Institutes for Research. Retrieved from http://files.eric.ed.gov/fulltext/ED523102.pdf Sedlacek, W. E., & Brooks, G. C., Jr. (1976). Racism in American education: A model for change. Chicago, IL: Nelson-Hall. Sommerfeld, A. (2011). Recasting non-cognitive factors in college readiness as what they truly are: Non-academic factors. Journal of College Admission, 213, 18- 22. Retrieved from www.nacacnet.org Stratil, M. (2009). College student inventory form A. Coralville, IA: Noel-Levitz, Inc. Tan, J. A., & Hall, R. J. (2005). The effects of social desirability bias on applied measures of goal orientation. Personality and Individual Differences, 38, 1891- 1902. Terenzini, P. T., & Reason, R. D. (2005, November). Parsing the first year of college: Rethinking the effects of college on students. Paper presented at the Annual Conference of the Association for the Study of Higher Education, Philadelphia, PA. Thornton, B., & Gupta, S. (2004). Comparative validation of a partial (versus full) randomized response technique: Attempting to control for social desirability response bias to sensitive questions. Individual Differences Research, 2, 214-224. Tinto, V. (1993). Leaving college: Rethinking the cause and cures of student attrition (2nd ed.). Chicago, IL: University of Chicago. US Department of Education, National Center for Education Statistics. (2013). The condition of education 2013 (NCES 2013-037), Institutional Retention and Graduation Rates for Undergraduate Students. SPRING 2015 • VOLUME 22, NUMBER 2 21 TABLE 1 Descriptive Statistics of Study Variables Measure/Scale n Minimum Maximum Mean SD MCSDS 645 0 29 15.52 6.2 SSI-CE 582 8 40 26.13 6.64 SSI-AE 582 10 40 27.41 5.14 SSI-ASE 584 8 40 32.93 5.96 SSI-SC 581 9 40 29.07 5.78 SSI-R 574 9 40 25.11 5.33 SSI-EC 581 8 40 34.53 5.69 GPA 612 0 4 3.2 1.21 Persistence 608 6 30 26.47 3.64 Intentions Note. MCSDS = Marlowe-Crowne Social Desirability Scale; SSI-CE = Campus Engagement; SSI-AE = Academic Engagement; SSI-ASE = Academic Self-Efficacy; SSI-SC = Social Comfort; SSI-R = Resiliency; SSI-EC = Educational Commitment; SD = standard deviation. TABLE 2 Correlations of the SSI and Cumulative GPA with and without Social Desirability (MCSDS) SSI Subscale Zero-order MCSDS Partial Correlation Partialed Out Correlations CE .08 .08 0 AE .22** .23 0 ASE .26** .26 0 SC .00 .01 -.01 R -.01 -.00 .01 EC .11** .12 -.01 Note. SSI = Student Strength Inventory; MCSDS = Marlow-Crown Social Desirability Scale; CE = Campus Engagement; AE = Academic Engagement; ASE = Academic Self-Efficacy; SC = Social Comfort; R = Resiliency; EC = Educational Commitment. *indicates that the correlation is significant at p < .05. **indicates that the correlation is significant at p < .01. 22 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION TABLE 3 Correlations of the SSI and Persistence Intentions with and without Social Desirability (MCSDS) SSI Subscale Zero-order MCSDS Partial Correlation Partialed Out Correlations CE .06 .05 .01 AE .20** .18 .02 ASE .26** .26 0 SC .22** .21 .01 R .12** .10 .02 EC .30** .30 0 Note. SSI = Student Strength Inventory; MCSDS = Marlow-Crown Social Desirability Scale; CE = Campus Engagement; AE = Academic Engagement; ASE = Academic Self-Efficacy; SC = Social Comfort; R = Resiliency; EC = Educational Commitment. *indicates that the correlation is significant at p < .05. **indicates that the correlation is significant at p < .01.