NODA Journal 21_1.indd 46 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION Exploring the Predictive Value of the ACT Among First-Year College Students on Academic Probation Tabitha Young Gast and Isaac Burt This study explored the relationship between ACT scores and academic achievement among first-year college students on academic probation (n = 192) at a large southern university. Results suggest participants’ ACT scores did not correlate with second semester grade point averages (GPA). When GPAs were compared with ACT scores of the larger student body admitted during the same academic year, results indicated students on academic probation averaged significantly lower. However, comparisons with national averages suggest participants’ ACT scores demonstrated a regression to the mean. The implications this study has for retention, academic improvement, and academic counseling in higher education are discussed. Research indicates that a disturbing trend exists among incoming college freshman. Specifically, an estimated one-fourth of incoming freshmen are failing to graduate once enrolled in academic programs (Brooks, Jones, & Burt, 2012). Deemed student retention and attrition, this phenomenon is both problematic and controversial (National Survey of Student Engagement, 2011). Operationally defined, student retention is the act of sustaining students at an institution of higher learning until they graduate. Conversely, student attrition is when students drop out of an institution of higher learning before graduating (Brooks, et al., 2012). According to the National Center for Higher Education Management Systems (NCHEMS), (2007), an alarming number of universities and colleges nationwide have attrition rates over 25 percent. Clearly a problem exists that institutions of higher education must resolve. Thus, the question remains, “How do universities and colleges increase the academic success of first year students?” Currently, university admissions address this issue by requiring entrance examination scores. For example, students must obtain satisfactory scores on the ACT, formerly known as the American College Testing Program Assessment, or SAT, formerly known as the Scholastic Achievement Test scores (Bridgeman, McCamley- Jenkins, & Ervin, 2000). Further complications arise when several universities FEATURED ARTICLE Tabitha Young Gast (tlyoung1@olemiss.edu) is an Assistant Professor in Leadership and Counselor Education in the College of Education at the University of Mississippi. Isaac Burt (iburt@fiu.edu) is an Assistant Professor in Counselor Education at Florida International University. FALL 2013 • VOLUME 21, NUMBER 1 47 use the same assessment (e.g., ACT) but focus on different benchmark scores that identify potentially capable students (Maruyama, 2012). One problem stemming from the preceding phenomena is when institutions begin to receive monetary resources based partially on the benchmark scores used for admission (Dimmock, 2012). Clearly, controversy with benchmark and admission scores has a number of consequences affecting students and universities; however, universities require additional criteria for admission, such as cumulative high school grade point averages (GPAs) in making college acceptance decisions as well. The idea behind using ACT scores and high school GPAs reflects the concept that past performance and prior abilities are predictors of future college academic performance. This concept of predictive determination has received considerable attention in the research literature, both positive and negative. For instance, a number of studies affirm the predictive validity of this relationship (Astin, 1993; Wolfe & Johnson, 1995). However, a relative number of studies indicate flaws in these predictive methods and soundly criticize the validity of using them (Kern, Fagley, & Miller, 1998; Oldfield et al., 1994). Controversy in the Literature As stated previously, controversy exists among researchers on determining the legitimacy of the relationship between academic success and predictive validity of entrance examinations (Bush & Bush, 2010). To understand the development of this controversy, a detailed look at opposing views needs undertaking. Among studies confirming this predictive relationship, Wolfe and Johnson (1995) discovered college students’ combined high school GPA and composite entrance examination scores (i.e., ACT) were responsible for explaining up to 25% of the variance of students’ performance in college. Thus, Wolfe and Johnson deemed both high school GPA and entrance examination scores useful in predicting college achievement. In a report supported by the American Council on Education, Astin (1993) confirmed high school GPA and entrance examination scores were the two strongest predictors of college GPA amongst students’ admission data. Astin reported these variables yield modest predictions on a consistent basis. Therefore research suggests college admissions examination scores and high school GPA consistently predict students’ success in college. Whereas some researchers encourage the use of entrance examinations in predicting college success, others denounce its futility for making admissions decisions. For example, Oldfield et al. (1994) found the ACT did not predict college success correctly. In their work, a predictive relationship did not exist between ACT scores and first year GPA among college freshman. Due to the ACT not accurately capturing the wide array of skills necessary for college success, reliance on the examination is a dubious method at best. Similarly, Kern, Fagley, and Miller (1998) stated the ACT failed to capture important characteristics affecting students’ academic achievement. Specifically, college entrance examinations do not accurately depict students’ motivation, which can strongly 48 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION influence academic achievement or failure. Although student motivation is not the focus of this study, models of motivation such as the Transtheoretical Model of Change (Prochaska & DiClimente, 1986) and Bernaus and Gardner’s (2008) model of student motivation and perception aid in understanding the additional factors that influence student success. Therefore, research exists suggesting the ACT is not effective in predicting first-year academic success. Due to controversy surrounding the topic of academic success and predictors, this study sought to provide more clarification and expand on work already conducted. Specifically, the authors explored the question, “Do ACT scores predict academic success among first time college students?” However, unlike previous studies, this article examined the predictive value of ACT scores among students on academic probation. Hence, as opposed to other studies, this article explored the relationship between academic achievement among first-year students who experienced academic difficulty in the early college experience. To explore this pre- existing relationship, the following hypotheses were investigated: 1) The GPAs of participants (i.e., individuals on academic probation) significantly improved from the fall of 2011 to the spring of 2012. a. Null: Participants’ GPAs did not improve from the fall of 2011 to the fall of 2012. 2) ACT score was able to predict GPA among college undergraduates on academic probation. a. Null: ACT score was not able to predict GPA among participants of this study. 3) There was a correlation between college first-year college students on academic probation ACT scores and second semester GPA. a. Null: There was no correlation between first-year college students ACT score and second semester GPA. In order to examine these hypotheses, the remainder of this article contains three sections. The first section describes the methods utilized, including research design, participants, instrumentation, and procedures. Second, the results segment gives a detailed description of the three hypotheses. Third, the discussion section concludes the article specifying limitations, implications, and future research directions. Method Design The authors used a correlational research design to examine the relationship of the ACT and academic success. Two hundred and forty first-year college students on academic probation during the 2012 spring semester participated in this study. FALL 2013 • VOLUME 21, NUMBER 1 49 Participants Participants were incoming freshmen entering the second semester of their undergraduate studies at a large, southern, research-intensive university. Additionally, participants were on academic probation and mandated to attend a course designed to increase academic performance. Participants received information about the study and gave their consent. In addition, participants attained no incentives for being in the study. Likewise, participants received no consequences for declining to be in the study. For detailed explanations of this study’s demographics, please see Table 1 for a summary of participants’ defining characteristics. Measures Demographic questionnaire. Participants completed a demographic questionnaire in which they reported gender, age, marital status, ethnicity, and employment status. Procedures Data collection. The authors collected data during the first week of class. During the first day of class, students received information about the study and were sent a link to the demographic questionnaire and informed consent by way of their student accounts. Once students’ informed consent was obtained, the research team accessed archival data (i.e., GPA, ACT score), removed any identifying information from participants’ data in order to ensure confidentiality, and input data into an SPSS database for analysis. Power Analysis The authors calculated a priori power analysis using G power.com for all statistical analyses. According to Cohen (1992), in order to determine power, a number of criteria must be satisfied. First, for linear regressions with an alpha of .05 and a power equal to .95, a sample size of 128 student participants is needed to detect small (i.e., conservative) effect sizes of .6 or more (Lenth, 2001). Furthermore, a power analysis for a paired samples t-test with an alpha of .05 and a power equal to .95 needs a minimum sample size of 45 student participants to detect small, conservative effect sizes of .6 (Lenth, 2001). A power analysis for Pearson’s Product-Moment Correlation, with an alpha of .05, and a power equal to .95 (Cohen, 1992), requires a minimum sample size of 111 student participants to detect a small effect size .6 (Lenth, 2001). Thus, the sample size of 192 participants in this study was robust enough to control for Type II (i.e., failure to reject a false null hypothesis) error. 50 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION Results Sample Integrity An examination of the data assessed for missing values, normality, linearity, homoscedasticity, and univariate outliers before conducting analyses. After inspecting the data, the authors found no missing values or other complications with the data. Thus, no adjustments to the data were necessary (Wilcox, 2003). Therefore, to test if any violation of statistical assumptions occurred, the authors conducted an homogeneity of variances analysis. Homogeneity of variances could be assumed for all of the variables assessed. Thus, the simple statistical models were used for all analyses. Research Question 1 A one-sample t-test was conducted to determine, “Did the GPAs of the participants involved in our study significantly improve from the fall of 2011 to the spring of 2012?” There was a significant difference in score t(222) = 52.73, p < .05, with GPA being higher during the spring of 2012 (M = 2.39, SD = .68), than the fall of 2011 (M = 1.38, SD = .49). Thus, participants significant improved in GPA over the course of the study. Research Question 2 The authors assessed the second research question through a linear regression: Does ACT score predict GPA among college undergraduates on academic probation? The authors examined participants’ GPA in fall F(1, 221) = 1.77, p > .05, R2 = .01, spring F(1, 221) = 3.68, p > .05, R2 = .02, and change (i.e., spring-fall) F(1, 221) = .85, p > .05, R2 = .00. The analysis found none of these variables as significant predictors of ACT scores. Research Question 3 Due to mixed results from the two research questions (research question two not being statistically significant, thus not having the results expected), the research team proposed a new (third) research question. Specifically, the authors sought to determine if a correlation existed between college undergraduates’ (on academic probation) ACT scores and second semester GPA. The authors assessed this research questions via standard Pearson’s two-tailed product-moment correlation. Results suggest no significant linear relationship existed between ACT score (M = 21.24; SD = 3.35) and GPA (M = 2.43; SD = .68), r (192) = .433, p > .05. Discussion This study examined the relationship between second semester GPA and ACT score among individuals on academic probation. Although we expected to find a clear relationship between students’ ACT scores and GPAs amongst a population FALL 2013 • VOLUME 21, NUMBER 1 51 of students that were on academic probation (i.e., students’ whose ACT score would correlate with students’ GPA), our data did not discover this relationship. Furthermore, there was also no relationship between improvement in GPA and ACT score. Although the authors attempted to have a conservative research approach (i.e., small effect sizes for analyses, large sample sizes, predictions and correlations), possible alternative explanations existed. For example, underlying relationships between the variables potentially affected results. Succinctly stated, conclusions regarding the contributory relationships between the variables cannot be certain. Thus, the researchers recommend future longitudinal and experimental studies need to further examine the relationships with similar and dissimilar populations. By doing this, it may be possible to explore whether these findings generalize to other populations or settings. Additionally, there was not a significant relationship between second semester GPA among college students on academic probation and ACT score. This is a surprise given these scores (i.e., ACT and/or SAT) were designed to predict academic success. Furthermore, universities historically tend to heavily rely on ACT or SAT scores when making admissions decisions (Mississippi Business Journal, 2012). However, the results found in this study correspond with several studies (Lemann, 2000; Sedlacek, 2004; Vars & Bowen, 1998). Therefore, this study aligns with research suggesting ACT scores are not a significant indicator of future academic performance. Although ACT scores did not correlate with second semester GPA among college students on academic probation, a significant difference did exist. For instance, the study revealed a relationship between the average ACT score for college freshman admitted to the host university (where the research was conducted) during the 2012 academic year and participants in this study. According to Yates (2012), the mean ACT composite score for college freshman was 23.9, which was the highest average to date. The mean ACT scores for the participants of this study (n = 192) ranged from 14 to 33 (ACT composite score) and were as follows: M = 21.24 Composite Score, M =21.60 English, M = 20.46 Math, M = 21.78 Reason and M = 22.09 Reading. Therefore, the participants of this study averaged significantly lower ACT composite scores than the average college freshman admitted during the 2012 academic year. However, when comparing the scores of the participants in this study with national and state statistics, the results are dissimilar. For example, according to the Mississippi Business Journal (2012), the average score for students taking the ACT in the nation was 21.1 in 2012, which indicates the participants of this study averaged .14 points higher than the average student taking the ACT in 2012. Furthermore, when contrasting participants’ average scores against the state average (which was 18.7 during 2012), another interesting situation occurs. For instance, the students on academic probation involved in this study scored 2.54 points higher. These results suggest ACT composite scores of the participants involved in this study were comparable to the average ACT test taker. Additionally, participant scores were higher than the average student taking the examination within the state. Although individuals may take the ACT for a number of reasons other than 52 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION attending college (e.g., peer pressure), the focus of this study was on individuals who took the assessment for admission into an institution of higher learning. In summary, this study compared ACT scores of individuals on academic probation with the larger student body, national averages, and state averages. The researchers wanted to examine the predictive and correlation value of ACT scores amongst first year college freshman on academic probation. Although further research needs to use larger sample sizes to increase generalizabilty, the results of this study suggest the average ACT composite score was significantly lower for individuals on academic probation when compared with the larger student body. This suggests the ACT is able to make some distinction between students that will and will not succeed in college. However, this study indicates a number of contradictory findings. First, ACT scores of students on academic probation aligned with the national average. Second, participants’ scores, on average, surpassed the state average. Third, results show no correlation with second semester GPA. Hence, the ACT examination did not seem to be an effective predictor of college success. References Astin, A. W. (1993). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education. New York: American Council on Education. Bernaus, M., & Gardner, R. (2008). Teacher motivation strategies, student perceptions, student motivation, and English achievement. The Modern Language Journal. 92(3). 387-401 Bridgeman, B., McCamley-Jenkins, L., & Ervin, N. (2000). Predictions of freshman grade-point average from the revised and recentered SAT I: reasoning test (College Board Report No. 2000-1). New York: College Entrance Examination Board. Brooks, M., Jones, C., & Burt, I. (2012). Are African American male undergraduate retention programs successful? An evaluation of an undergraduate African American male retention program. Journal of African American Studies. doi: 10.1007/s12111-012-9233-2 Bush, E., & L. Bush (2010). Calling out the elephant: An examination of African American male achievement in community colleges. Journal of African American Males in Education, 1, 40–62. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. DeBerard, M. S., Spielmans, G. I., & Julka, D. L. (2004). Predictors of academic achievement and retention among college freshmen: A longitudinal study. College Student Journal, 38, 66–80. Dimmock, S.G. (2012). Background risk and university endowment funds. The Review of Economics and Statistics, 94, 789-799. Lemann, N. (2000). The big test: The secret history of the American meritocracy. New York: Farrar, Straus and Giroux. FALL 2013 • VOLUME 21, NUMBER 1 53 Lenth, R. V. (2001). Some practical guidelines for effective sample size determination. The American Statistician, 55, 187-193. Maruyama, G. (2012). Assessing college readiness: Should we be satisfied with ACT or other threshold scores? Educational Researcher, 41, 252-261. Mississippi Business Journal. (2012, August 23). State students ACT scores remain lowest in nation. Retrieved from http://msbusiness.com/ National Center for Higher Education Management Systems (NCHEMS). (2007). Retention rates: First-time college freshmen returning their second year (ACT). Retrieved from http://www.higheredinfo.org/ Prochaska, J. & DiClemente, C. (1986). The transtheoretical approach. In J.C. Norcross (Ed.) Handbook of eclectic psychotherapy (pp.163-200). New York Brunner/Mazel. Sedlacek, W. E. (2004). Beyond the big test: Noncognitive assessment in higher education. San Francisco: Jossey-Bass. Vars, F., & Bowen, W. (1998). Scholastic aptitude test scores, race, and academic performance in selective colleges and universities. In C. Jencks & M. Phillips (Eds.), The Black-White test score gap (pp. 457–479). Washington, DC: Brookings Institute Press. Wilcox, R. (2003) Applying Contemporary Statistical Techniques. USA: Elsevier Science. Wolfe, R. N., & Johnson, S. D. (1995). Personality as a predictor of college performance. Educational and Psychological Measurement, 55, 177-185. Yates, M. (2012). ACT score of UM freshman is higher than ever. In The DM Online. Retrieved October 20, 2012, from http://thedmonline.com/ Zwick, R., & Sklar, J. C. (2005). Predicting college grades and degree completion using high school grades and SAT scores: The role of student ethnicity and first language. American Educational Research Journal, 42, 439–464. 54 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION TABLE 1 Summary of Participant’s Demographics Variable N % Marital Status Married or Partnered 22 11.5 Single 169 88 Widowed 1 .5 Employment Status Full Time 2 1 Part Time 40 20.8 Not Working 149 78.1 Sex Female 117 60.9 Male 75 39.1 Biracial Yes 11 5.7 No 181 94.3 Culture Native American 2 1 Asian or Pacific Islander 4 2.1 African American 60 31.3 Caucasian 122 63.5 Hispanic/Latino 4 2.1 Sexual Orientation Gay 1 .5 Lesbian 1 .5 Bisexual 3 1.6 Heterosexual 183 95.3 Prefered not to disclose 4 2.1 Age 17 2 1 18 93 48.4 19 91 47.4 20 5 2.6 21 1 .5