NODA Cover 19_2.indd 32 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION Using Student Learning Outcomes to Evaluate Orientation Programs Hollie Gammel Smith, Brooke Rodine, and Audrey Williams This article examines student learning outcomes of the new student orientation program at Stephen F. Austin State University (SFA). The study describes differences in pre-orientation and post-orientation surveys that were completed via SFA’s orientation website using the Student Voice higher education assessment platform. Participants were prospective freshmen students who planned to attend SFA during Fall 2009. The pre- orientation survey was completed by 834 students, and the post-orientation survey was completed by 331 students. Results indicated that students increased their knowledge and understanding of SFA’s academic policies, requirements, and services. Limitations and recommendations for future implications are noted in the study. Colleges and universities understand the importance of introducing new students to their campus community and their academic programs (Boening & Miller, 2005). For many years, these institutions have developed and implemented orientation programs for incoming students to assist them with adjusting to the college environment. Daddona and Cooper (2002) stated that orientation programs are only beneficial for making the students feel more comfortable in their new environment. Utilizing and assessing student learning outcomes can help orientation programs ensure that the needs of the students are being met, and they are equipped to succeed in the transition process. Literature Review Many studies have been conducted in regards to freshman orientation, the experiences of students during this period of transition, and how participation in an orientation program can positively impact students’ assimilation into college. In a study conducted with primarily at-risk students, the authors found that informing the students about social and affective issues, such as emotional and self-esteem concerns, increased the students’ coping skills and eased their transition to college (Baruch-Runyon, VanZandt, & Elliott, 2009). In this article, the authors also noted how new student orientations should include information relevant FEATURED ARTICLE Hollie Gammel Smith (hsmith@sfasu.edu) is the Assistant Dean of Student Affairs for Programs at Stephen F. Austin State University. Brooke Rodine (rodinebm@sfasu.edu) is the Director of Orientation at Stephen F. Austin State University. Audrey Williams (awilliams19@leo.tamu-commerce.edu) is a Counselor at the Counseling Institute of Greater Dallas at Texas A&M-Commerce. SPRING 2012 • VOLUME 19, NUMBER 2 33 to the transitioning of minorities and first-generation students to the campus culture. Cowart (1990) found that participating in orientation provided students with higher levels of social integration than those found in students who had not attended orientation. These students were found to have greater interaction with faculty and staff members and higher levels of participation in campus extracurricular activities. As colleges compete for students and the needs of incoming students continue to evolve, orientation programs have become more creative and comprehensive. When the programs begin to address more concepts, there is an increased need for assessing whether the students are being significantly impacted by their experience at orientation. Boening and Miller (2005) noted that diversity in orientation programs is essential, and that that orientation should address cultural and diversity issues. In addition to diversity, retention has also become a major focus of colleges and universities. In their study, Derby and Smith (2004) examined the effects of orientation programs in regards to retention. They found that incoming students are more likely to be engaged and remain in college if they complete the orientation process. The process serves as an important impact on the students, and it can also have ramifications on the institution’s funding. Some states are utilizing retention and graduation rates as a factor in accountability programs for financial support of state institutions (Tinto, 2006). Tinto (1988) describes the transition of new college students with three stages of student departure: separation, transition, and incorporation. When students attend orientation, they are starting to separate from their past communities of high school and their hometown. This process will affect students differently; if their previous communities do not value getting a college education, then separation will be more of a shock than for those students whose previous communities accept and encourage attending college. Orientation programs have the opportunity to aid the students’ separation, but it also provides information about resources to assist in the transition and incorporation stages. Orientation programs provide students with their first exposure to social and academic norms of the institution, as well as guide students into the incorporation stage. The influence of orientation programs is the most effective in positively impacting student retention in the first year (Tinto, 1988). Hodum (2007) addressed this purpose of orientation programs, saying, “Because of the difficulties new students may encounter upon starting college, an orientation program’s ability to address students’ needs could potentially persuade them in their decision to persist or leave” (p. 6). Pascarella, Terenzini, and Wolfle (1986) concluded that orientation’s influence on social integration leads to retention through the students feeling more connected to the university. Their report stated: Controlling for differences in precollege characteristics and initial commitments, students attending orientation had significantly higher levels of social integration and subsequent commitment to the institution than those students not attending orientation. These latter two variables, in turn, had the largest significant direct effects on freshman persistence of all variables 34 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION in the model. This suggested a non-trivial indirect influence of orientation on freshman year persistence. (p. 166) In order for orientation programs to positively influence student retention rates, the students’ need for social integration and commitment to the institution must be paired with the resources necessary to help them succeed. Parents are also an important part of new student orientation, and they can be a resource to help students persist through college. Most colleges offer orientation for parents and family members to obtain information regarding the university. Coburn and Woodward (2001) described several parent orientation programs that offer creative, fun quizzes, and scavenger hunts for parents during orientation. Most importantly, parent orientation programs should emphasize concrete information so that parents (as well as students) are able to utilize the universities’ resources (Coburn & Woodward, 2001). In that way, parents of incoming students can be utilized as an additional resource for students in transition. Orientation Surveys Beck and Davidson (2001) investigated scores from the Survey of Academic Orientations (SAO) to determine if there was a relationship between SAO scores and the grades of freshmen at the end of the first semester. They found that the SAO can be used as an early warning predictor of low grades among freshmen. Nevertheless, Scholastic Achievement Test (SAT) scores and high school percentage rank should be taken into consideration when utilizing SAO scores to predict future grades (Beck & Davidson, 2001). Dadonna and Cooper (2002) completed a study measuring the effectiveness of orientation programs in comparison to the perceived needs of college freshmen before and after participating and completing an orientation program. The authors utilized a survey to assess the perception of the student’s need in regards to a week-long orientation. The most important needs were found in areas related to academics, and least important needs were related to the social needs of the new student. Student learning outcomes can be beneficial in assessing individual orientation programs to determine whether incoming students are truly learning the information they need while attending orientation. Otter (1995) explained the benefit of using learning outcomes: “The increase in student numbers has placed considerable pressure on the traditional forms of assessment in higher education, and for many staff the major advantage of outcomes may be to offer a means of managing assessment effectively” (p. 283). The student learning outcomes are used in the development of the orientation program and then utilized again directly in pre-orientation and post-orientation surveys to determine the impact of sessions and information provided during the orientation program. Purpose The purpose of this study is to examine the extent to which students SPRING 2012 • VOLUME 19, NUMBER 2 35 demonstrated changes in learning as measured through learning outcomes of the orientation program at SFA. The research questions guiding this study are as follows: 1. Is the orientation at SFA successfully integrating new students academically to the university? 2. Is the orientation at SFA successfully integrating new students socially to the university? The following questions were used to determine the level of academic integration into the university: a) Are first time-freshmen able to identify the core curriculum requirement for their academic major and minor after participating in orientation? b) As a result of attending orientation, are new students comfortable communicating with their academic advisor regarding their major? c) Is the orientation at SFA providing first-time freshmen with an understanding of the career options related to their academic major and minor? d) Are first-time freshmen able to identify SFA’s academic requirements, expectations, and policies as a result of attending orientation? The following questions were used to determine the level of social integration into the university: a) Is the orientation at SFA successfully teaching incoming students about the traditions of the university? b) Can new students state the location and services provided by at least two campus departments after attending orientation? c) Is SFA’s orientation program successfully facilitating the creation of new relationships between students who did not know each other prior to orientation? d) Are first-time freshmen able to identify co-curricular opportunities after participating in SFA’s orientation program? Hypothesis The following two hypotheses were tested at the .01 significance level in this study: 1. There is no difference in the pre- and post-test scores in regards to the academic integration into the university. 2. There is no difference in the pre- and post-test scores in regards to the social integration into the university. 36 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION Method Participants The sample population consisted of prospective first-time, full-time students. Of the 2,221 first-time, full-time freshman students that attended SFA’s orientation program during the summer of 2009, 834 students participated in pre-orientation assessment. The post-orientation assessment was completed by 331 students. The total student body consisted of 12,845 full- and part-time students for the Fall 2009 semester. There were 2,364 first-time, full-time freshmen enrolled for the Fall 2009 semester. This first-time freshman class was comprised of 845 male and 1,519 female students. The average ACT score of these students was 21, and the average SAT score was 977. This quantitative study evaluated the impact of new student orientation on achievement of student learning outcome goals. A week or two prior to attending orientation, students were advised via e-mail to go to the SFA Orientation Programs website to complete a pre-orientation survey. The incoming students then attended a two-day orientation program, which concluded with the students registering for courses. After completing orientation, students were e-mailed again and asked to complete a post-orientation survey on the same website (no incentives were offered for students to take the pre- or post-orientation surveys). Instrument The pre- and post-orientation assessments were online surveys administered through Student Voice. Student Voice is an online assessment platform for institutions of higher education, which allows surveys to be submitted, reviewed, and distributed, and the results to be analyzed. The assessment consisted of nine questions addressing the major learning outcomes of the orientation program, including knowledge of academic requirements, ability to effectively communicate with their academic advisor, understanding of university traditions, and ability to identify campus resources. In addition to the pre-and post-orientation assessment results, grade point average (GPA) for the students was gathered to determine if participation in orientation has any correlation with GPA earned in the first semester. The first fall semester GPA information was gathered from the university’s student record database system. Statistical Package for the Social Sciences (SPSS), a statistical analysis software system, was utilized to run t-test statistics to analyze results of the pre- and post-orientation surveys. Pre- and post-orientation assessments were matched according to their student ID numbers, resulting in complete assessment information from 128 students. Responses were coded as 0 = Not Applicable, 1 = Strongly disagree, 2 = Moderately disagree, 3 = Neither agree nor disagree, 4 = Moderately agree, and 5 = Strongly agree. SPRING 2012 • VOLUME 19, NUMBER 2 37 Results Analyses focused on examining the differences in the pre- and post-evaluation results. T-tests were used to analyze the results, and the results for all nine research questions were statistically significant. For both the pre- and post-evaluations, each student was asked to indicate his or her level of agreement with each statement. Results from the t-test on pre- and post-tests are presented in Table 1. Based on the results, it was concluded that the three learning outcomes with highest difference between pre- and post-tests were: understanding how to use the SFA course registration system, being able to identify three co-curricular opportunities at SFA, and being able to state the location and services provided by at least two campus departments. While still statistically significant, the learning outcomes with the least difference between pre- and post-tests were: giving examples of the different career opportunities related to the student’s academic major and minor, followed by feeling comfortable communicating with their academic advisor regarding their major. These learning outcomes also had the highest mean on the pre-test which could account for the smaller difference with the post-test scores. Discussion Limitations There are several limitations to this study. One limitation is that less than half of the students who completed the pre-orientation survey completed the post- orientation survey. Students self-selected into the study as they made a decision to attend orientation, as well as to participate in the study. In addition, the study was limited by time restrictions based on the period allotted for data collection. Although the students were asked to fill out the post-survey immediately following orientation participation, it is possible that participation in orientation will continue to impact them throughout their remaining college experience. Additionally, the online administration of the survey could have had an impact on the number of responses compared to if the surveys were administered in person at the beginning and end of each orientation session. Offering an incentive for filling out the pre- and post-orientation surveys could also have raised response rates, including the number of matched surveys. Conclusions and Recommendations Overall, the results of this study indicated that students increased their knowledge about SFA’s academic programs and services. In addition, there was an increase in student’s knowledge of the social and personal aspects of transition to college. Of the two days that the students are at orientation, the first day was dedicated to social integration and the second day was almost wholly focused on academics. In order to obtain a better understanding of the orientation process, it 38 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION TABLE 1 Learning Outcome Question, Pre-Orientation Survey Mean, Post-Orientation Survey Mean, and t-test value (df=127, p < .01) Question Pre- Post- t SD Orientation Orientation Mean Mean Each question began with “Please indicate your level of agreement with the following statements.” 1. I can identify the core 2.89 4.20 -10.58 1.39492 curriculum requirements for my academic major and minor. 2. I feel comfortable 3.94 4.52 -5.09 1.28351 communicating with my academic advisor regarding my major. 3. I can give examples 3.785 4.19 -4.16 1.20895 of the different career opportunities related to my academic major and minor. 4. I can identify the 3.12 4.23 -9.61 1.30556 university’s academic requirements, expectations, and policies (e.g., grades and graduation requirements). 5. I understand how to use the 2.13 4.49 -20.03 1.33242 SFA course registration system. 6. I can state at least two 2.96 4.39 -11.04 1.47285 traditions of the university. 7. I can state the location and 2.35 4.16 -14.29 1.43494 services provided by at least two campus departments. 8. I can name three people that 3.27 4.42 -8.52 1.53406 I am friends with at SFA. 9. I can list three co-curricular 2.26 4.11 -15.49 1.35210 opportunities at SFA. SPRING 2012 • VOLUME 19, NUMBER 2 39 is recommended that program directors encourage students to complete the post- orientation survey immediately after orientation has concluded. Murphy, Hawkes, and Law (2002) have suggested that surveys may not accurately represent students if they are delivered online. Due to the potential lack in accurate representation and the probability of lower response rates to online surveys, orientation professionals may consider conducting post-orientation surveys immediately after orientation before students leave campus. Paper and pencil surveys may then be entered online if necessary. Due to the importance of establishing and utilizing learning outcomes to assess orientation programs, orientation professionals are encouraged to determine which methods will best fit their program. Whether online or paper-and-pencil surveys are administered, the information gathered will allow for improvements in the efficacy of orientation programs, thus better aiding students in their academic and social transitions to college. References Baruch-Runyon, A., VanZandt, Z., & Elliott, S. A. (2009). Forging connections: An investigation of new students’ perspectives of their transition to the university. NACADA Journal, 29(1), 31–42. Beck, H. P., & Davidson, W. D. (2001). Establishing an early warning system: Predicting low grades in college students from survey of academic orientations scores. Research in Higher Education, 42(6), 709–723. Boening, C. H., & Miller, M. T. (2005). New student orientation programs promoting diversity. Community College Enterprise, 11(2), 41–50. Coburn, K. L., & Woodward, B. (2001). More than punch and cookies: A new look at parent orientation programs. New Directions for Student Services 94, 27–38. Cowart, C. V. (1990). The contribution of freshman orientation programs to academic and social integration of sophomore students (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 9116728) Daddona, M. F., & Cooper, D. L. (2002). Comparison of freshmen perceived needs prior to and after participation in an orientation program. NASPA Journal, 39(4), 300–318. Derby, D. C., & Smith, T. (2004). An orientation course and community college retention. Community College Journal of Research & Practice, 28(9), 763–773. Hodum, T. L. (2007). An investigation of how students, faculty, and administrators within a particular liberal arts college perceived a new-student orientation program’s effect on students’ social integration and retention (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 3322702) Murphy, C., Hawkes, L., & Law, J. (2002). How international students can benefit from a web-based college orientation. New Directions for Higher Education, 117, 37–44. 40 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION Otter, S. (1995). Learning outcomes in higher education. In J. Burke (Ed.), Outcomes, learning and the curriculum; Implications for NVQ’s, GNVQ’s and other qualifications (pp. 273–284). London: RoutledgeFalmer. Pascarella, E. T., Terenzini, P. T., & Wolfle, L. M. (1986). Orientation to college and freshmen year persistence/withdrawal decisions. The Journal of Higher Education, 57(2), 155–175. Tinto, V. (1988). Stages of student departure: Reflections on the longitudinal character of student leaving. The Journal of Higher Education, 58(4), 438–455. Retrieved from http://www.jstor.org/stable/1981920 Tinto, V. (2006). Research and practice of student retention: What next? Journal of College Student Retention: Research, Theory & Practice, 8(1), 1–19. doi: 10.2190/4YNU-4TMB-22DJ-AN4W