NODA Journal 17_1.indd FALL 2009 • VOLUME 17, NUMBER 1 57 Is Eighty Percent of Success Just Showing Up? Student Compliance and Refusal to Complete an Advisement Form as an Indicator of First-Term GPA Michael J. Roszkowski and Raymond A. Ricci Motivation and, inferentially, commitment are critical, non-cognitive factors in college success. One needs to detect and measure these attributes prior to a student’s actual enrollment in classes since early detection of at-risk students can lead to the most productive intervention initiatives. Freshmen entering into La Salle University were required to complete a form used as a basis for advising. Students complying (n = 427) and not complying (n = 291) with the request were compared on high school grade point average (GPA), SAT scores, and first-term college GPA. The noncompliant students had lower credentials on the admissions criteria (high school GPA, SAT) as well as on the outcome measure (first-term college GPA), although the effect sizes were small. The findings support the contention that compliance with requirements is a proxy for academic motivation and can serve as a cue to how well a student will perform. In most universities, the admission decision is based on consideration of the applicant’s academic performance in high school (e.g., high school GPA and/or class rank) and scores on aptitude tests (e.g., SAT or ACT). Generally, taken by itself, the high school record is a better predictor of college performance than the aptitude test, especially when the criterion is the first-term or first-year GPA. This is true in the case of both the ACT (Lotkowski, Robbins, & Noeth, 2004) and the SAT (Kobrin, Patterson, Shaw, Mattern, & Barbuti, 2008). When considered along with the high school record, however, aptitude tests improve the predictability of GPA. For example, in a sample of college-bound seniors in 2006 (n = 151,316) studied by Korbin et al. (2008), the correlation between high school GPA and first-year college GPA was .36, while the correlation between the GPA after the first year of college and the weighted composite of SAT Critical Reading and SAT Math scores equaled .32. When entered into a multiple regression, the three predictors (high school GPA, SAT Critical Reading, SAT Math) produced a multiple correlation of .44. If corrected for restriction in range and other statistical artifacts, then the correlation became .61. Although the adjusted correlation explains slightly over 37% of the variance in first-year college GPA in terms of high school GPA and SAT scores, there is FEATURED ARTICLE Michael J. Roszkowski (roszkows@lasalle.edu) is the Director of Institutional Research at La Salle University. Raymond A. Ricci (ricci@Lasalle.edu) is Assistant to the President for Mission Integration at La Salle University. 58 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION substantial variance that cannot be accounted for by these traditional measures. Such findings have led to a search for non-cognitive predictors of college outcomes (Côté & Levine, 2000; Sedlacek, 2004; Tross, Harper, Osher, & Kneidinger, 2000; Pritchard & Wilson, 2003; Wolfe & Johnson, 1995), especially among minorities (Fuertes & Sedlacek, 1995; Tracey & Sedlacek, 1989) and unique segments of the student population (Boyer & Sedlacek, 1988; Petrie & Stoever, 1997; Ting, 2000). Various general purpose inventories have been used to assess non-cognitive factors critical to academic success, including the Myers-Briggs Type Indicator (Hengstler, 1981) and measures of cognitive style (Nortridge, Mayeux, Anderson, & Bell, 1992). In addition, measures intended specifically for this purpose have been developed, such as the Noncognitive Questionnaire (Sedlacek, 2004; Tracey & Sedlacek, 1989) and a biographical inventory (Oswald, Schmitt, Kim, Ramsay, & Gillespie, 2004; Schmitt et al., 2007). However, the value of various noncognitive variables continues to be debated (Lotkowski, Robbins, & Noeth, 2004; Ransdell, 2001; Thomas, Kuncel, & Crede, 2007; Wolf & Johnson, 1995). According to a meta-analysis conducted by Robbins et al. (2004), the best non-cognitive predictors of college GPA are academic self-efficacy (r = .50) and achievement motivation (r = .30). Different definitions can be found, but most indicate that motivation is an internal state that activates behavior in a given direction (Kleinginna & Kleinginna, 1981). It is not surprising that motivation plays a major role in college performance. The importance of motivation to learning has long been recognized (Deci, Vallerand, Pelletier, & Ryan, 1991), but the problem has been how best to measure it (Annen, Kamer, & Bellwald, 2005; Entwisle, 1972; Holmes & Tyler, 1968; Ray, 1980, 1981; Ross & Doyle, 1978) and how to increase it (McClellan, 2006; Stipek, 1988; Wolters, 2003; Yoshida et al., 2008). While it is possible to assess achievement motivation using a number of standardized tests of this construct (Baker & Siryk, 1984; Byrne et al., 2004; French & Oakes, 2003; Le, Casillas, Robbins, & Langley, 2005; McCann & Garcia, 1999), one problem is that a student can easily fake the answers. Moreover, test information of this type is generally unavailable to faculty members and student services staff; according to Siebert et al. (2006, p. 750), “professors often rely on shared anecdotes and their own personal experiences as a means to understand students’ academic motivation.” However, it is also possible to determine motivation on the basis of biographical data (biodata) and observation of behaviors exhibited during pre- and post-enrollment (Moore, 2005, 2006; Moore et al., 2003). For instance, post-enrollment actions such as missing classes are frequently the consequence of a lack of motivation (Moore, 2006). Compared to surveys, naturally occurring behaviors are less likely to provide information that may be compromised because the student gave socially desirable answers. “Eighty percent of success is showing up” is a quote attributed to the actor and director, Woody Allen. While the percentage may be a bit high, this statement does carry some truth when it comes to academic achievement in higher education. Moore (2005) reported that at the University of Minnesota, first-year developmental education students are required to attend a summer orientation FALL 2009 • VOLUME 17, NUMBER 1 59 program. Some students voluntarily attend this program, whereas others have to be coerced to participate. At the end of the first semester, students who attended voluntarily achieved an average GPA of 2.9, compared to 2.1 for the students who attended involuntarily. Moreover, the latter attended 34% fewer classes and were almost three times more likely to be placed on academic probation after their first semester in college (32% versus 11%). Moore concluded that “voluntary compliance with a pre-enrollment admission requirement (in this case, voluntary attendance at a summer orientation program) is a strong predictor of subsequent academic success” (p. 325). Moore (2007) viewed compliance as an expression of students’ academic motivation and maintained that it was the most important determinant of developmental students’ failure or success at college. Our data further support Moore’s premise that noncompliance with requirements prior to the start of classes forebodes lower levels of academic performance in the first semester of college. Moreover, since our sample consisted mainly of traditionally admitted students, Moore’s point applies to college students in general. Method Participants The sample consisted of new full-time freshmen enrolled for the Fall 2008 semester in a Catholic, master’s level university located in an urban area in the mid-Atlantic region of the United States. The full-time undergraduate population of the institution is slightly over 3,000 students. Freshmen were required to complete an online form in order to inform advisors about students’ attitudes and needs. Non-respondents were contacted and followed up through e-mail, and an announcement was posted on Web pages intended for new students. Despite the stated necessity to complete this form, noncompliance was high. The freshman class consisted of 847 students. However, only 718 students were the basis of this analysis because 52 were subjected to an intervention that could have raised the GPA, and 79 did not have the opportunity to complete the survey because they enrolled late. The 718 cases used in this analysis consisted of 427 (59.47%) students who completed the advising form and 291 (40.53%) who refused to complete it. Advising Form The 7-page advising form consisted of 23 questions (see Appendix). Results The respondents and non-respondents were compared on traditional measures 60 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION used in the admission process, namely high school GPA and SAT scores (Critical Reading and Math). They were also compared on the outcome measure of first-term GPA. Mean Differences in Predictors and Criterion by Cooperation Status The differences between freshmen who complied and those who refused, which are reported in Table1, were statistically significant on the two predictors of college performance, as well as on the performance itself. The students who complied with the request to complete the form had better high school GPAs [F(1,716) = 12.23, p =.000], Critical Reading SAT scores [F(1,716) = 3.76, p = .053], Math SAT scores [F(1,716) = 4.22, p = .040], and the combined Critical Reading and Math SAT scores [F(1,716) = 5.14, p = .024). On first-term college GPA, the difference of 2.97 versus 2.82 was also statistically significant [F(1, 716) = 6.40, p = .009). However, all the effect sizes were small. The corresponding eta coefficients were .13 (high school GPA), .07 (Critical Reading SAT), .08 (Math SAT), .08 (Combined SAT), and .10 (first-term college GPA). When squared, eta reflects the proportion of the total variance that can be attributed to an effect (Olejnik & Algina, 2000). Simple Correlations between the Variables Table 2 shows that the best single predictor of first-term college GPA was high school GPA (r = .42) followed by the combined SAT score (r = .31). When considered together, the high school GPA and SAT score (combined Critical Reading and Math) produced a multiple correlation of .47. These values are consistent with the most recent uncorrected validity coefficients reported by the College Board (Kobrin et al., 2008). Value of Compliance in the Mix of Variables Predicting First-Term GPA High school GPA, combined SAT score, and survey participation were used as predictors of first-term GPA in the following hierarchical regression models: Model 1: high school GPA + SAT + compliance; Model 2: high school GPA + compliance; Model 3: SAT + compliance. The first model indicated that if both high school GPA and SAT scores are available, then the information about compliance is of limited practical value in predicting first-term GPA within the context of a multiple regression because this behavior has little unique variance to contribute to the equation. Table 3 shows that if compliance (dummy coded: 0 = no, 1 = yes) is entered last in a hierarchical multiple regression that already contains high school GPA and SAT scores, it does not add much to the prediction of first-term GPA. The change in the amount of explained variance is less than 1%. In the equation with the three predictors (high school GPA, combined SAT, and form completion), the standardized regression FALL 2009 • VOLUME 17, NUMBER 1 61 coefficients (beta) are, respectively, .363, .214, and .033. In other words, the independent contribution of compliance to the prediction equation is about one-eleventh that of high school GPA and nearly one-seventh that of the SAT score. Similarly, Model 2 indicates that information about compliance does not contribute to the prediction of first-term GPA in a hierarchical regression, with high school GPA entered on the first step and compliance on the second step. The multiple correlation increases from .424 to .426, which constitutes a small change that fails to reach statistical significance [ΔF(1,715) = 1.65, p = .200]. However, if high school GPA is unavailable (Model 3), compliance adds significantly to the degree of predictability possible on the basis of SAT alone. That is, the multiple correlation (R) increases from .311 to .319 [ΔF(1,715) = 41.13, p = .043], and the standardized regression coefficients are .305 for SAT (p = .000) and .072 (p = .043) for compliance. Discussion Motivation and, inferentially, commitment are critical, non-cognitive factors in college success. One needs to detect and measure these attributes prior to a student’s enrollment in classes since early detection of at-risk students can lead to the most productive intervention initiatives. Refusal to complete an advisement form prior to the start of the first semester can serve as an early warning mechanism. Requiring new students to complete an advising form prior to actual enrollment should have elicited the highest level of cooperation. Compliance necessitated minimal investment by the student; the form was available on the Internet and generally took less than 30 minutes to complete. Furthermore, the request emphasized the benefits of completion: helping advisors assist students in their adjustment to college. The act of noncompliance provides valuable information about academic motivation. Even without seeing the answers on the advising form, one can speculate how well the student will perform during the first semester. Not surprisingly, compliance was also correlated to the same extent with the high school record, suggesting that grades in high school may have also suffered somewhat due to noncompliance issues. Because it is related to academic performance in high school, noncompliance adds little to predicting first-term college grades beyond what is possible on the basis of high school record. Therefore, the compliance information may be most beneficial in situations where faculty and staff involved in student development do not have access to the student’s academic record and aptitude test scores. This indicator can also serve as a supplement to any standardized tests of academic motivation, providing a way of validating the answers. Conclusion Our results support Moore’s (2005, 2006, 2007) contention that certain 62 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION behaviors evident prior to the start of the semester can be indicative of the student’s performance in college, probably because they are proxies for academic motivation. Compliance, whether “showing up” for an orientation program or completing a required form, is a crucial first step on the road to success. Limitations The magnitude of the observed relationship between compliance in filling out the advising form and first-term GPA was small. Perhaps this is because other reasons could explain why the form was not completed (for instance, it could be viewed as an invasion of privacy because it requires self-disclosure). It would be insightful to study whether timely compliance in completing other types of required pre-enrollment forms also helps to predict a student’s first-term GPA. Moreover, since this study was based on one institution, the extent to which the findings can be generalized needs to be researched. References Annen, H., Kamer, B., & Bellwald, M. E. (2005, November 8–10). How to measure achievement motivation? An explorative study to measure achievement motivation in a selective assessment center. Paper presented at the 47th IMTA Annual Conference, Singapore. Baker, R. W., & Siryk, B. (1984). Measuring academic motivation of matriculating college freshmen. Journal of College Student Personnel, 25(5), 459–464. Boyer, S. P., & Sedlacek, W. E. (1988). Noncognitive predictors of academic success for international students: A longitudinal study. Journal of College Student Development, 29(3), 218–223. Byrne, Z. S., Mueller-Hanson, R. A., Cardador, J. M., Thornton, G. C. III, Schuler, H., Frintrup, A., & Fox, S. (2004). Measuring achievement motivation: Tests of equivalency for English, German, and Israeli versions of the Achievement Motivation Inventory. Personality and Individual Differences, 37(1), 203–217. Côté, J. E., & Levine, C. G. (2000). Attitude versus aptitude: Is intelligence or motivation more important for positive higher-educational outcomes? Journal of Adolescent Research, 15(1), 58–80. Deci, E. L., Vallerand, R. J., Pelletier, L. G., & Ryan, R. M. (1991). Motivation and education: The self-determination perspective. Educational Psychologist, 26(3), 325–346. Entwisle, D. R. (1972). To dispel fantasies about fantasy-based measures of achievement motivation. Psychological Bulletin, 77(6), 377–391. French, B. F., & Oakes, W. (2003). Measuring academic intrinsic motivation in the first year of college: Reliability and validity evidence for a new instrument. Journal of the First-Year Experience & Students in Transition, 15(2), 83–102. Fuertes, J. N., & Sedlacek, W. E. (1995). Using noncognitive variables to predict the grades and retention of Hispanic students. College Student Affairs Journal, 14(2), 30–36. FALL 2009 • VOLUME 17, NUMBER 1 63 Hengstler D. D. (1981, May). Prediction of academic success with the Myers-Briggs type indicator (MBTI). Paper presented at the Annual Forum of the Association for Institutional Research, Minneapolis, MN. Holmes, D. S., & Tyler, J. D. (1968). Direct versus projective measurement of achievement motivation. Journal of Consulting and Clinical Psychology, 32(6), 712–717. Kleinginna, P., Jr., & Kleinginna, A. (1981). A categorized list of motivation definitions, with suggestions for a consensual definition. Motivation and Emotion, 5(3), 263–291. Kobrin, J. L., Patterson, B. F., Shaw, E. J., Mattern, K. D., & Barbuti, S. M. (2008). Validity of the SAT® for predicting first-year college grade point average (Research Report No. 2008–5). NY: The College Board. Le, H., Casillas, A., Robbins, S. B., & Langley, R. (2005). Motivational and skills, social, and self-management predictors of college outcomes: Constructing the student readiness inventory. Educational and Psychological Measurement, 65(3), 482–508. Lotkowski, V. A., Robbins, S. B., & Noeth, R. J. (2004). Non-academic factors in improving college retention: ACT policy report. Iowa City, Iowa: American College Testing. McClellan, J. L. (2006, Nov. 26). Student motivation: How much can we really do? The Mentor: An Academic Advising Journal. Retrieved from http://www.psu.edu/dus/mentor/061129jm.htm McCann, E. J., & Garcia, T. (1999). Maintaining motivation and regulating emotion: Measuring individual differences in academic volitional strategies. Learning and Individual Differences, 11(3), 259–279. Moore, R. (2005). Pre-enrollment and post-enrollment predictors of the academic success of developmental education students. Journal of College Student Retention Research Theory and Practice, 6(3), 325–335. Moore, R. (2006). The importance of admissions scores and attendance to first-year performance. Journal of the First-Year Experience & Students in Transition, 18(1), 105–125. Moore, R. (2007). Diverse behaviors, diverse results: A motivation-based model for students’ academic outcomes. In J. L. Higbee, D. B. Lundell, I. M. Duranczyk, & R. Copeland (Eds.). Diversity and the Postsecondary Experience (pp. 129–144). Minneapolis: Center for Research on Developmental Education and Urban Literacy, University of Minnesota. Moore, R., Jensen, M., Hatch, J., Duranczyk, I., Staats, S., & Koch, L. (2003). Showing up: The importance of class attendance for academic success in introductory science courses. The American Biology Teacher, 65(5), 325–329. Nortridge J. A., Mayeux, V., Anderson S. J., & Bell, M. L. (1992). The use of cognitive style mapping as a predictor for academic success of first-semester diploma nursing students. Journal of Nursing Education, 31(8), 352–356. Olejnik, S., & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25(3), 241–286. 64 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89(2), 187–207. Petrie, T. A., & Stoever, S. (1997). Academic and nonacademic predictors of female student-athletes’ academic performance. Journal of College Student Development, 38(6), 599–608. Pritchard , M. E., & Wilson, G. S. (2003). Using emotional and social factors to predict student success. Journal of College Student Development, 44(1), 18–28. Ransdell, S. (2001). Predicting college success: the importance of ability and non-cognitive variables. International Journal of Educational Research, 35(4), 357–364. Ray, J. J. (1980). The comparative validity of Likert, projective, and forced-choice indices of achievement motivation. The Journal of Social Psychology, 111(1), 63–72. Ray, J. J. (1981). Measuring achievement motivation by immediate emotional reactions. The Journal of Social Psychology, 113(1), 85–93. Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta- analysis. Psychological Bulletin, 130(2), 261–288. Ross, M., & Doyle, K. O., Jr. (1978). Measures of academic motivation: A conceptual review. Research in Higher Education, 8(1), 1–23. Schmitt, N., Oswald, F. L., Kim, B. H., Imus, A., Merritt, S., Friede, A., & Shivpuri, S. (2007). The use of background and ability profiles to predict college student outcomes. Journal of Applied Psychology, 92(1), 165–179. Sedlacek, W. E. (2004) Beyond the big test: Noncognitive assessment in higher education. San Francisco: Jossey-Bass. Siebert, J. W., Litzenberg, K. K., Gallagher, R., Wilson, C. A., Dooley, F., & Wysocki, A. (2006). Factors associated with students’ academic motivation in agricultural economics classes. American Journal of Agricultural Economics, 88(3), 750–762. Stipek, D. (1988). Motivation to learn: From theory to practice. Englewood Cliffs, NJ: Prentice Hall. Thomas, L. L., Kuncel, N. R., & Crede, M. (2007). Noncognitive variables in college admissions: The case of the Non-Cognitive Questionnaire. Educational and Psychological Measurement, 69(4), 635–657. Ting, S. R. (2000). Predicting Asian Americans’ academic performance in the first year of college: An approach combining SAT scores and noncognitive variables. Journal of College Student Development, 41(4), 442–449. Tracey, T. J., & Sedlacek, W. E. (1989). Factor structure of the Non-Cognitive Questionnaire-Revised across samples of Black and White college students. Educational and Psychological Measurement, 49(3), 637–648. Tross, S. A., Harper, J. P., Osher, L. W., & Kneidinger, L. M. (2000). Not just the usual cast of characteristics: Using personality to predict college performance and retention. Journal of College Student Development, 41(3), 323–334. FALL 2009 • VOLUME 17, NUMBER 1 65 Wolfe, R. N., & Johnson, S. D. (1995). Personality as a predictor of college performance. Educational and Psychological Measurement, 55, 177–185. Wolters, C. A. (2003). Regulation of motivation: Evaluating an underemphasized aspect of self-regulated learning. Educational Psychologist, 38(4), 189–205. Yoshida, M., Tanaka, M., Mizuno, K., Ishii, A., Nozaki, K., Urakawa, A., Cho, U., Kataoka, Y., & Watanabe, Y. (2008). Factors influencing the academic motivation of individual college students. International Journal of Neuroscience, 118(10), 1400–1411. 66 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION TABLE 1 Mean Scores on Predictors and the Outcome Measure as a Function of Compliance with the Request to Complete the Advising Form Form Completion High School SAT Critical SAT Math SAT First-Term Status GPA Reading (CR) (M) (Combined College CR and M) GPA Completed M 3.39 518.13 516.39 1034.52 2.97 SD 0.48 79.39 80.47 142.39 0.70 Refused M 3.26 506.49 503.78 1010.27 2.82 SD 0.51 78.12 81.24 138.04 0.80 Total M 3.34 513.41 511.28 1024.69 2.91 SD 0.50 79.03 80.97 141.05 0.74 TABLE 2 Product Moment Correlations between the Predictors and the Criterion SAT SAT High First Term Form Critical Math (M) School College Completion Reading (CR) GPA GPA Status SAT Critical Reading .55*** .21*** .29*** .07* (CR) SAT Math (M) .25*** .26*** .08* SAT Combined CR and M .26*** .31*** .08* High School GPA .42*** .13*** First Term College GPA .10** *** p < .001 ** p < .01 * p < .05 FALL 2009 • VOLUME 17, NUMBER 1 67 TABLE 3 Changes in the Multiple Correlation in an Hierarchical Multiple Regression Predicting First-Term GPA on the Basis of High School GPA, SAT, and Compliance in Completing the Advising Form Model R R Square Adjusted R R Square Square Change (Δ) High School (HS) GPA .424 .179 .178 .179a HS GPA+SAT .472 .223 .221 .044b HS GPA+SAT+Compliance .473 .224 .221 .001c a ΔF(1,716) = 156.55, p =.000 b Δ F(1,715) = 40.21, p =.000 c Δ F(1,714) = 0.96, p =.327 68 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION APPENDIx Questions on the Advising Form • Why did you choose to come to [university name]? Please check all that apply. • Please finish the following sentence: In my time at [name of university] I hope to… Check all that apply. • Why did you pick your major? Check all that apply. • How certain are you that you want to major in this subject? • What other major or minors are you considering? • What is the highest academic degree you intend to obtain? • Are there any special circumstances that we need to consider in helping you plan your educational program? Check all that apply. • Do you have any specific concerns about adjusting to college life? • Where would you turn to for advice/information should the need arise? • How do you think your advisor can help you most? Select all that apply). • What is the best way for you to receive information? Check only one. • How committed are you to earning a college degree? • When you were making a choice between [university name] and the other college or university you were considering most strongly, how difficult was it to make the choice? • How sure are you that [university name] is the right choice for you? • Compared to the other school you were considering, how would you rate the cost of attending [university name]? • How important was the condition of the school’s neighborhood in your college selection process? • How easily do you make friends with people you don’t know? • How would you describe your ability to cope with new situations? • Based on other experiences you had being away from home, how likely are you to get homesick? • When you experience problems, what are you most likely to do? • How would you describe your time management skills?