NODA Journal 16_1.indd SPRING 2009 • VOLUME 16, NUMBER 1 25 Proving Our Value: Developing a New Orientation Assessment Plan Brittney Black and Brenda McKenzie “First year programs that survive and thrive likely share a common link—a strong outcome assessment agenda that is closely connected to program goals” (Swing, 2001). Many orientation and transition programs are being challenged to “prove” their value. Do they achieve their goals? Do they positively impact student satisfaction and retention? Shortly after redesigning the fall program at Kent State University, we realized a need to answer these questions. In 2004, we implemented a fall orientation program which went from 2–5 days in length, incorporated more academic sessions and student development programs, and was funded through a matriculation fee. Given the potential for increased scrutiny of this redesigned program, we believed it necessary to create a plan to assess its effectiveness. This article will outline the process we followed and the lessons learned. Feedback Does Not Equal Success Since the inception of the August orientation program in 1996, Scantron evaluations had been administered. These evaluations simply asked students to rate their satisfaction; they didn’t solicit detailed feedback or suggestions. While the results provided positive feedback, they were not helpful in determining success in meeting program goals. Hence, we had little quality data to use as a foundation for building future programs. It was therefore determined that we needed to develop a more comprehensive assessment plan for the Week of Welcome (WOW) program. Process As with most assessment activities, we started with the end in mind. In this case we had to determine what exactly we were trying to gain from the assessment process. Initially, this involved researching and benchmarking against other institutions and reviewing literature to gain a better understanding of what assessment involved and produced. We then reviewed our new student orientation program mission and goals, from which we identified two goals to focus on in the first year of our assessment plan. FEATURED ARTICLE Brittney Black (brblack@kent.edu) is an Assistant Director of Alumni Relations/Outreach at the Kent State University. Brenda McKenzie (bmckenzi@kent.edu) is an Associate Director for the Center for Student Involvement at the Kent State University. 26 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION The first goal was to increase the awareness of available resources, both on and off campus, that promote students’ academic success and personal development. This goal correlated with a number of mandatory sessions for new students during the Week of Welcome program. Focusing on required programs allowed us to reach a larger number of students with our assessment instruments. The identified programs also directly related to the institution’s focus on student success. The second goal assessed was to provide information on, and opportunities for, campus and community involvement. This goal specifically relates to the overall mission of the office out of which WOW is coordinated. It also supports the importance of literature related to involvement and retention (Astin, 1993; Kuh, Schuh, Whitt, & Associates, 1991; Pascarella & Terenzini, 1991, 2005; Tinto, 1987). Astin’s theory on involvement can be simply stated as “Students learn by becoming involved” (Pascarella & Terenzini, p. 50). This theory illustrates that student-to-student interactions have positive correlations with their satisfaction with student life (Astin, 1993). As Kuh, Schuh, Whitt, and Associates (1991) state, “the goal is to create environments or situations where all students have opportunities to participate in ways that contribute something to, as well as take some from, an experience” (p. 317). Most of the WOW programming related to this goal was optional for new students; however, it typically had above an average attendance. After identifying the goals, we determined which programs offered during Week of Welcome addressed those goals. The end result was a lengthy list of possible sessions and events to assess. At this point, it became clear that additional guidance and assistance were needed. We were fortunate to identify and work with two staff members in Kent State’s Research, Planning and Institutional Effectiveness (RPIE) office. The primary purpose of RPIE is to provide departments on campus with survey design and administration assistance as well as analysis of data. Without the assistance of these staff members, we would have floundered through the process for quite some time and probably would not have achieved such comprehensive results. Our initial meeting with RPIE pertained to how we had evaluated our programs in the past. In addition, we discussed our focus goals and possible programs to target. While we thought we were advancing through the process, we found this was only the beginning. From that initial meeting, our “homework” was to take each of the identified sessions/events, write a description for each, list goals for the session/event, identify who was to be impacted, and develop possible research questions; in other words, what we hoped to learn by assessing that particular session/event. We spent quite a bit of time writing and rewriting neutrally worded survey questions. Working with institutional research staff was critical to ensuring we were asking appropriate questions. After we finished our “homework,” the process actually became easier (no less time-intensive, but easier). We narrowed the list of sessions and worked with RPIE to create a plan. This primarily consisted of developing paper surveys to be distributed at the identified events, as well as a follow-up, Web-based survey to be SPRING 2009 • VOLUME 16, NUMBER 1 27 administered one month into the semester. We also met with staff in the university’s Bureau of Research Training and Services. This office provides survey development, copying, results tabulation, and transcription to researchers for a fee. When our survey development was under way, the process hummed along smoothly and we were ready to go well before WOW. We ultimately decided to distribute surveys at six sessions/events during WOW. The sessions included one specifically for commuters, a faculty panel, diversity awareness, getting involved, and an activities fair. The final survey assessed the new students’ overall impressions of WOW. The surveys were completed on Scantron to allow for easy and affordable results processing. The first four were distributed at the end of the related session. The activities fair survey was administered by staff walking around at the final event of WOW, and the overall survey was distributed at the final required program. The results proved to be extremely positive and provided beneficial information for the staff to use in future program planning. What We Learned In addition to gaining valuable information from the results, we learned a number of lessons throughout the process. 1. Be prepared to spend significant time and energy identifying what you want to gain from the assessment. Although it was a time-consuming process, we have no regrets. You have to commit to the process and be willing to follow through. We never imagined the amount of up-front work that would be required to create an effective assessment process, nor how difficult it would be to put our intentions and goals into words. 2. Use questions to help focus your planning. Thanks to the staff in our institutional research area, we identified and addressed the following questions to help us create the most effective assessment process: a. What is your purpose for doing assessment (in other words, “So what if you know this?”)? Will it affect the program or facilitate change? If you won’t use the data, you should seriously consider whether or not it is worth the investment. b. Is one of your goals to get data about sessions from multiple perspectives (i.e., new students, families, presenters, staff)? c. From your perspective, what is the most practical way to gather assessment data about events: paper-and-pencil, Web-based, or on-the-spot electronic surveying? d. What financial resources do you have available? e. How much detail do you want to have in your event assessments? Keep in mind it was our use of a simple Likert scale process that got us to the point of needing a new assessment plan. We determined that we needed more detail. f. Do you have the resources or means to manage the process and the results that come from it? 28 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION g. With whom will you share the results? 3. Our decision to administer six surveys focused on two different goals was too much. It generated a huge amount of data to review, sift through, and find common themes. Once we started to do cross-tabulation it became even more overwhelming. For example, we compared responses of students who attended all or most WOW events to those who did not. Having so much data became unwieldy and challenging to share with others. 4. Assessment efforts are expensive. We had several options available to us for the actual tallying of the results. We could have saved money by not paying another department to tabulate and transcribe the data. However, this would have added a tremendous amount of strain on our staff; we simply did not have the resources to transcribe comments. It would have also required more time from the staff in RPIE to tabulate and interpret results. In the long run, it was worth the expense. We saved by developing our own surveys rather than purchasing something that had already been developed. Redirecting the savings also allowed us to tailor the questions to our specific programs. 5. If you have an event that is fluid (i.e., with people walking around), paper surveys are not effective. We had an extremely low rate of return on our surveys at the activities fair. Our staff was uncomfortable approaching students to fill out the surveys, and it was inconvenient to try and fill in Scantrons while holding a clipboard. Hindsight being 20/20, would we have undertaken the development of a more comprehensive assessment plan for our orientation programs? Absolutely, as the results gave us insight on ways to improve our programming efforts. We shared the results with our student coordinators, which provided them with direct student input on the events they were developing. And, as one of the few areas within student affairs to create a comprehensive assessment, our efforts set us apart as a progressive program. Best of all, the results supported what we were doing! The second year through the process was easier and less time-intensive. We chose to focus on one of the same goals—to provide information on, and opportunities for, campus and community involvement. This allowed us to use some of the same surveys from 2005. We also cut the number of paper surveys to three, contracting with StudentVoice to administer two surveys via Personal Digital Assistants (PDAs), which proved to be successful. The students enjoyed the novelty, for our campus, in providing feedback in this way. As we move forward, we will continue to work with the staff from RPIE, having developed a very effective relationship with them. We now have a solid understanding of the resources they can provide; they understand what we do and believe in our programs. Through this process we were able to make connections across campus, and by working with other departments and utilizing our campus resources, we were able to create a very effective tool for our programs. From this experience, we learned a great deal about what we offered our constituents, in addition to what we as professionals found to be key elements of our SPRING 2009 • VOLUME 16, NUMBER 1 29 programming, and we made great progress towards proving our overall importance to the institution and our students! References Astin, A. W. (1993). What matters in college? Four critical years revisited. San Francisco: Jossey-Bass. Kuh, G. D., Schuh, J. H., Whitt, E. J., & Associates. (1991). Involving colleges: Successful approaches to fostering student learning and development outside the classroom. San Francisco: Jossey-Bass. Pascarella, E. T., & Terenzini, P. T. (1991). How college affects students: Findings and insights from twenty years of research. San Francisco: Jossey-Bass. Pascarella, E. T. & Terenzini, P. T. (2005). How college affects students, Volume 2: A third decade of research. San Francisco, CA: Jossey-Bass. Swing, R. L. (2001). Introduction. In R.L. Swing (Ed.), Proving and improving strategies for assessing the first college year (pp. ix–xii). Columbia, SC: National Resource Center for the First-Year Experience and Students in Transition. Tinto, V. (1987). Leaving college: Rethinking the causes and cures of student attrition. Chicago: University of Chicago Press.