36 The Journal of College Orientation and Transition ARTICLE A Longitudinal Analysis of Standards Used to Evaluate New Student Orientation at a Case Institution Daniel P. Nadler, Michael T. Miller, and Beverly G. Dyer Student orientation programs are important tools designed to assist new students in adapting to their new college or university environments (Mullendore, 1992; Twale, 1989). Over the past decade, however, orientation programs have shifted from conveying institutional expectations of new students and building community among new student groups to being used as a retention tool. Indeed, undergraduate student retention has emerged as a hallmark of the accountability movement within higher education, as colleges and universities use four-, five-, and six-year graduation rates as indicators of how much an institution cares for students. Simplified, higher graduation rates seem to indicate that an institution is qualitatively better and more caring of students so that they graduate at a better rate than others. Although such logic is fuzzy at best, it does reflect a change in thinking about new student orientation programming. The content of new student orientation programs has been a frequent topic for student affairs professionals during the past three decades (Gardner & Hansen, 1993). The conversation became so relevant that the Commission for the Advancement of Standards (CAS) worked with a committee of senior student affairs professionals to identify core standards for new student orientation programs. The resulting CAS Standards for New Student Orientation (CAS, 1988) included 20 specific activities that new student orientation programs should include. These ‘standards’ (see Table 1) represent a wide-variety of interests, ranging from promoting campus safety and healthy lifestyle choices to the technical aspects of registering for classes and getting to know faculty, staff, and continuing students. The Standards for New Student Orientation were intended to form a blueprint for new student orientation programs, and as such, many programs simply made efforts to make certain that each of the 20 functional areas identified were somehow covered in the orientation. This has been particularly difficult in environments where new student orientation takes the form of a one- to two-day program held during the summer before enrollment, and is particularly difficult to cover in the typical one-half day transfer student orientation. At one private, highly selective university in an urban southern city, the CAS Standards were used as a guide for developing the weeklong new student orientation program. As the primary criteria for including specific events, the Standards were directed to the student participants and used as an evaluative mechanism for the orientation program. Specifically, students who completed the new student orientation Daniel P. Nadler is the Associate Vice President for Student Affairs at Tulane University. Michael T. Miller is an Associate Professor of Higher Education Leadership at the University of Arkansas, and Beverly G. Dyer is an Assistant Professor of Higher Education Administration at the University of Alabama. 37Spring 2004 • Volume 11, Number 2 program were asked to what extent the program did in fact satisfactorily address the standard. This procedure provided the data for the current study, which included an assessment of the new student orientation program from 1995 through 2001. Orientation: More Than an Introduction to Campus New student orientation programs have evolved considerably in the just the past two decades (Nadler & Miller, 1999). These programs initially provided an overview of institutional life and the level of expectations a campus held for new students, and in some cases, served as an initial weeding-out process for students who might be marginally qualified (Kramer & Washburn, 1983). Throughout the 1980s, new student orientation programs shifted in their overall effort, and increasingly attempted things like building a sense of community, teaching time management, note taking, and priority setting. The technical aspects of campus life were also included in orientation, as registration and testing were common elements of early new student orientation programs. By the 1990s, new student orientation had evolved into a compendium of activities, all loosely arranged around the ideas of transitioning new students to campus. The inclusion of the CAS Standards did provide some direction for institutions, but often the approach to program development was simply of one adding on to previous year’s efforts. Further, the CAS Standards brought to mind new components to student orientation, and the result on some campuses were increasingly burdensome, less directed and targeted approaches to learning about campus life. A second conceptual shifting took place during the past decade, as problems with higher education accountability led to a considerable amount of attention being directed at undergraduate student retention. New student orientation programs were seen as a partial answer to the retention problem, and again, the transition programs were modified to represent this thinking (Pascarella & Terenzini, 1991). On some campuses, new student orientation became a multiple semester activity, with a summer registration program, a community-focused multi-day program just before the start of a first semester, and then one, two, and in some cases three semesters of university life related coursework (Stephenson, 1997). At least part of the difficulty associated with clarifying a core set of missions or values for new student orientation is that a successful transition is difficult to quantify in any meaningful fashion (Twale, 1989). This is why so many in the student services were quick to take on the retention issue, as it was a clear-cut mechanism for proving how accountable, and perhaps important, their work has become. The problem with using new student orientation programs for such a wide variety of purposes is quite simply that the programs are asked to do so much, that they end up do little with any level of depth (Miller & Viajar, 2001). This occasional confusion over a purpose for new student orientation as well as a need to try and provide some sort of evaluation or assessment of an orientation program led to the initial questioning about the CAS Standards as criteria to measure the 38 The Journal of College Orientation and Transition orientation program against. The current study is a direct result of that conversation, and has provided important and valuable data to the case study institution about what new student orientation is succeeding at doing, and what areas need to be re-examined. Research Design A case study research protocol was adapted to work with the highly selective university located in a major Southeastern urban area. The case study institution enrolled, during the time of study, approximately 10,000 students across a broad- spectrum of academic fields, including professional studies such as law and medicine, and had an over century-old history of serving this urban location. Of the 10,000 students, approximately half were undergraduates during any given year of data collection, and entering classes ranged from approximately 1,000 to 1,500 new, first-time college first-year students. The majority of each entering first-year class was from over 500 miles from where the student graduated from high school. The research protocol was tested in 1995 using a sample of students who completed a four-day, pre-semester new student orientation program. The survey was determined to be reliable and valid, and initial reliability indices for the instrument were in the .6052 to .6988 range. Further reliability testing on data used in the current study yielded an alpha level of .91 and a standardized alpha of .92, indicating strong levels of internal reliability. Following the pilot use of the instrument in 1995, data were collected annually from 1996 through 2002, with approximately 75% to 85% of each entering class completing the survey instrument. The instrument was distributed and collected at the same time for each orientation program, that is, at the end of the four-day pre- semester orientation when the new students reported back to their residential housing location for the final orientation meeting with their resident directors and assistants. The survey was comprised of the 20-items that were the CAS Standards for New Student Orientation and some basic demographic data. Students were asked to rate the orientation program they had just completed and how well they agreed that the program accomplished each of the CAS Standards. To rate each item, students were provided a 1-to-5 Likert-type scale, where 1=Strong Disagreement that the CAS Standard was met, progressing to 5=Strong Agreement that the CAS Standard was met. Findings Over the seven years of data collection, 7,258 surveys were completed and returned for inclusion in data analysis. Due to some missing data, 7,250 surveys were used in computing mean scores for the CAS Standard ratings. The range of participation was from a low of 835 completed surveys in 1998 to a high of 1,220 in 1999. There was an average of 1,035 completed surveys from new students at the case institution each year during the seven years of study. As shown in Table 1, there were consistently half male and half female respondents to the survey. Overall, 53% of the respondents were female, with a high of 56% of all 39Spring 2004 • Volume 11, Number 2 respondents coming in 2000, and a low of 49% in 1999, which was the same as the 49% who were male in 1999. As shown in Table 2, four items were consistently rated higher than others, with seven-year grand mean scores of over 4.0, meaning that new students agreed to strongly agreed that these CAS Standards were met by the new student orientation program. The items included “assisted me in developing positive new relationships with other new students” (GM=4.21), “provided information concerning academic policies, procedures, requirements, and programs” (GM=4.12), “promoted an awareness of non-classroom opportunities” (GM 4.06), and “provided appropriate information on personal safety and security” (GM=4.06). Few items were rated below 3.0 in any given year, and only four items had overall seven-year mean ratings of 3.5 or lower. The four CAS Standards that had from neutral perceptions of satisfaction to agreement that the Standard was met through new student orientation were: “provided information about opportunities for self-assessment” (GM=3.55), “provided referrals to qualified advisers and counselors” (GM=3.52), “assisted me in identifying costs of attending the university, both in terms of dollars and personal commitment” (GM=3.49) and “assisted me in developing positive relationships with university faculty” (GM=3.44), Although there were variations in agreement levels for each standard during various years, mean ratings were generally consistent changing no more than .2 or .3 in either direction. The orientation participants who participated in 1997 and 2002 had the highest overall mean ratings for the CAS Standards (with 3.84 each), and those in 1996 and 2001 had the lowest overall mean ratings (overall mean ratings of 3.73 each). Implications for Practice The value of the current research is in the effort to assess an orientation program rather than the individual mean ratings of any given CAS Standard. With each year’s results, a workshop was conducted with student life staff responsible for different aspects of new student orientation and with orientation team leaders. These workshops focused on the results of the survey, that is, how students perceived the orientation to have been successful in meeting the goals and objectives of the orientation program. When items were rated with lower mean scores, that item was discussed at length and consideration was given as to whether or not the current practice should be changed. For example, in the initial survey of 1996, the orientation program received a mean rating of 3.41 for helping new students develop relationships with the university’s faculty. The orientation team leaders and professional staff met and discussed strategies for involving more faculty and involving them in different ways. The result was something of a departure from the previous year’s ‘tips from the professor’ hour-long session, and incorporated faculty into various sessions related to success in both the classroom and at the university in general. The initial change resulted in a mean rating bump during the next year to 3.57, and no change was made for the next year. In 1999, however, the item dropped to 3.37, and the orientation professionals and team leaders 40 The Journal of College Orientation and Transition then talked at length about what had happened. They generally agreed that getting faculty to participate was a substantial challenge, and was not something they were being successful in doing. They changed strategies somewhat to work through various academic deans rather than just relying on faculty volunteers, and the following year there approximately were the same number of faculty participating in the same way. Those involved re-tried to work through academic deans in 200 and 2001, but students continued to rate the orientation program’s success at accomplishing this objective low (3.43 and 3.32, respectively). Those involved with orientation then focused on asking deans and involved faculty how they could be involved in different and perhaps more effective ways, and although no substantial change was made to the orientation program, the orientation team increasingly has become aware of the need to involve faculty for the sake of helping student learn to be successful. Another approach that the team could have utilized would be to talk to students who had recently gone through orientation, and ask them what kinds of faculty exposure would be helpful or meaningful to their academic and campus life success. Similar examples of exploration of what orientation is for, and what specific objectives these programs are suppose to accomplish are the true value of the assessment effort undertaken at this university over the seven year period reported here. All too often, programming of any sort in higher education becomes driven by precedent, and the risk to new student orientation programs is that higher education institutions begin to fail the students, that is, they begin to focus on their own needs rather than on the needs of students. This behavior is both altruistic and reflective of the need for higher education to better define an intentionality of not only the philosophy of higher learning, but the intention of the sum of the parts. What is new student orientation supposed to accomplish? What are the objectives of an orientation program or system within this framework of goals? And, the orientation professional must ask how can these objectives be measured in a meaningful fashion that is consistent with the intent of higher education. The era of unintentional behavior for higher education institutions has come to an end, particularly for those institutions that receive public support, and student affairs professionals in general, and those in orientation in specific, must be meaningful, intentional, and purposive in their actions. If they fail to present their behaviors and programs in a rational, intentional manner, they may indeed find an increasingly loss of autonomy and freedom. References Council for the Advancement of Standards. (1988). Standards and guidelines for new student orientation programs. Washington, DC: Author. Gardner, J. N., & Hansen, D. A. (1993). Perspectives on the future of orientation. In M. L. Upcraft (Ed.), Designing Successful Transitions: A Guide for Orienting Students to College (pp. 183-194). Columbia, SC: Freshman Year Experience. Kramer, G., & Washburn, R. (1983). The perceived orientation needs of new students. 41Spring 2004 • Volume 11, Number 2 Journal of College Student Personnel, 24, 311-319. Miller, M., & Viajar, P. (2001). The integration of technology in new student orientation programs. Journal of College Orientation and Transition, 9(1), 33-40. Mullendore, R. H. (1992). Student based programming in orientation. In D. P. Nadler (Ed.), Orientation Director’s Manual (pp. 43-52). Statesboro, GA: National Orientation Directors Association. Nadler, D. P., & Miller, M. T. (1999). Designing transitional programs to meet the needs of multi-ethnic first-year students. Journal of College Orientation and Transition, 6(2), 20-27. Pascarella, E. T., & Terenzini, P. T. (1991). How college affects students. San Francisco: Jossey-Bass. Stephenson, G. D. (1997). The impact of a student success course on academic achievement and persistence. Unpublished doctoral dissertation, University of Alabama, Tuscaloosa. Twale, D. J. (1989). Social and academic development in freshman orientation: A time frame. NASPA Journal, 27, 160-167. 42 The Journal of College Orientation and Transition TABLE 1 Gender Distribution of Participating Students 1996 1997 1998 1999 2000 2001 2002 TOTAL n= 903 1,043 835 1,220 1,048 1,041 1,160 7250 Gender Female 463 557 424 605 594 514 611 3768 Male 424 469 302 599 439 515 542 3290 43Spring 2004 • Volume 11, Number 2 TABLE 2 CAS Standards 1996-2002 1996 1997 1998 1999 2000 2001 2002 All n= 903 1,043 835 1,220 1,048 1,041 1,160 7,250 Assisted me in 3.79 3.87 3.84 3.81 3.83 3.80 3.88 3.83 understanding the purpose(s) of CU1 (i.e., academic or career) Assisted me in 3.63 3.72 3.67 3.66 3.72 3.71 3.79 3.70 understanding the mission of CU (i.e., research, teaching, and service) Assisted me in 3.54 3.66 3.63 3.67 3.70 3.62 3.73 3.65 determining my purpose(s) in attending CU. Assisted me in 3.41 3.57 3.52 3.37 3.43 3.32 3.53 3.44 developing positive relationships with CU faculty. Assisted me in 3.66 3.67 3.64 3.57 3.58 3.52 3.67 3.61 developing positive relationships with CU staff. Assisted me in 4.14 4.23 4.21 4.25 4.28 4.18 4.23 4.21 developing positive relationships with other new students. Assisted me in 3.56 3.53 3.53 3.48 3.67 3.61 3.75 3.60 developing positive relationships with individuals from my community. 44 The Journal of College Orientation and Transition TABLE 2 (CONT.) CAS Standards 1996-2002 1996 1997 1998 1999 2000 2001 2002 All n= 903 1,043 835 1,220 1,048 1,041 1,160 7,250 Assisted me in 3.83 3.9 3.9 3.92 3.93 3.92 3.98 3.91 understanding CU’s expectations of me. Provided 3.44 3.62 3.54 3.56 3.57 3.53 3.62 3.55 information about opportunities for self-assessment. Assisted me 3.45 3.62 3.48 3.55 3.47 3.38 3.48 3.49 in identifying costs of attending CU, both in terms of dollars and personal commitment. Provided an 3.74 3.96 3.9 3.88 3.85 3.84 3.92 3.87 atmosphere and sufficient information that enabled me to make reasoned and well-informed decisions. Provided 4.07 4.09 4.14 4.14 4.13 4.14 4.18 4.12 information concerning academic policies, procedures, requirements, and programs. Promoted an 4.0 4.12 4.19 4.08 4.03 3.95 4.09 4.06 awareness of nonclassroom opportunities. 45Spring 2004 • Volume 11, Number 2 TABLE 2 (CONT.) CAS Standards 1996-2002 1996 1997 1998 1999 2000 2001 2002 All n= 903 1,043 835 1,220 1,048 1,041 1,160 7,250 Provided 3.55 3.67 3.62 3.54 3.46 3.37 3.48 3.52 referrals to qualified advisers and counselors. Explained 3.78 3.83 3.89 3.84 3.74 3.64 3.79 3.78 the process for class scheduling and registration. Assisted me in 3.87 4.04 3.99 3.99 4.06 3.79 4.04 3.97 developing familiarity with the physical surroundings. Provided 3.78 3.9 3.86 3.84 3.86 3.76 3.91 3.84 information and exposure to available institutional services. Created an 3.81 3.99 3.92 3.93 3.98 3.87 3.95 3.92 atmosphere that minimized anxiety, promoted positive attitudes, an stimulated an excitement for learning. Provided 4.07 4.09 4.19 4.03 3.98 3.99 4.11 4.06 appropriate information on personal safety and security. Provided 3.61 3.8 3.71 3.73 3.75 3.77 3.82 3.74 opportunities to discuss expectations and perceptions with continuing students. 1CU=Case University