NODA Journal Spring 2008.indd SPRING/SUMMER 2008 • VOLUME 15, NUMBER 2 69 Student Learning Outcomes Assessment for an Orientation Program: A “SOARing” Transformation Erin Bentrim-Tapio and Kim Sousa-Peoples Like most universities and Student Affairs divisions, those of us at The University of North Carolina at Greensboro are charged with assessing student learning outcomes. As pressure from external constituencies grows, so does the internal emphasis on measuring and documenting learning outcomes. As a result, each department within the Student Affairs division is closely re-examining their assessment plans to ensure that they not only measure, but also produce the desired learning outcomes. Being purposeful and using a systematic process will ultimately result in improved programs and student experiences. Orientation & Family Programs had just completed their five-year divisional program review carried out by a cross-divisional committee using CAS Professional Standards for Higher Education (2006) for Orientation as the framework. One of the recommendations from the final report was to develop student learning outcomes and the assessment of such for the Spartan Orientation, Advising, and Registration (SOAR) program. We believed this would be an opportune time to restructure the SOAR survey. The mission statement of the Office of Orientation & Family Programs was the primary starting point for identifying target areas to transition from a satisfaction-heavy survey to one that also encompassed student learning outcomes. We focused on two additional core documents as well: the departmental five-year program review and Learning Reconsidered (2004) domains. The key points in our mission statement that lend themselves to learning outcome measurement are transition, preparedness for academic rigor, University traditions and culture, and appreciation of a diverse community. We also wanted to draw on the aspects of our mission statement that are clearly congruent with several Learning Reconsidered (2004) domains. These include knowledge acquisition, integration, and application as well as humanitarianism, persistence and academic achievement, and civic engagement. Once the framework and purpose were established, the next step was to determine the process and plan for making modifications. One challenge related to evaluating an orientation program is that the program office often serves as a CAMPUS NOTES Erin Bentrim-Tapio (e_bentri@uncg.edu) is the Director of Student Affairs Assessment and Kim Sousa-Peoples (k_sousap@uncg.edu)is the Director of Orientation & Family Programs, both at The University of North Carolina at Greensboro. 70 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION “clearinghouse” for academic and support units to provide information to new students. The office is under immense pressure to ensure quality programs over which they have no direct control. This basic premise was foremost in our minds as we began redesigning the SOAR assessment instrument. Another primary concern was how to manage the anxiety of departments outside our reporting structure and territorial wrangling that are inextricably associated with the assessment process. We decided the best way to diminish any apprehension and concern that might arise was to pilot the new version with programs for which the Office of Orientation & Family Programs are solely responsible for planning and implementing. We were also aware of the importance of presenting the new assessment format to participating departments in such a way that did not create suspicion or resistance. Part of this plan included talking with various groups that contribute to the development of our program, including the Orientation Planning Council (university-wide council which meets monthly for SOAR planning) and Advising Council (university-wide council of advising center and enrollment services representatives). In addition to these conversations, we sent a detailed e-mail outlining the plan to measure learning outcomes for all components of our program and that we would start with our own home-grown sessions. By modeling the behavior we hoped to see, we believed if we went through the process first, it would show good faith in being open to critique. In following years, other departments will be selected as the focus for learning outcomes assessment and those areas will be heavily involved in creating survey items that best meet the needs of that particular department. We chose four SOAR programs to address: “Got Classes,” “Life on Campus,” “Student/Family Dessert Reception,” and “Campus Traditions.” Subsequent to our review of the Learning Reconsidered (2004) domains, we revisited the purpose of each program session by asking ourselves fundamental questions: “How will the student who attends this program differ from the student who does not?” or “As a result of attending this session, what do we want a student to be able to explain/ know/do/identify?” (Although our focus was not to redesign the session itself, we did ascertain whether the desired outcomes were aligned with the program). For the purposes of this article, we will provide one example from the “Got Classes?” session. There was only one survey question about this session in the 2006 survey, which read: Please select your answer to the following questions using the scale below. 1. Very poor 2. Poor 3. Neutral/No opinion 4. Good 5. Very good 6. N/A (didn’t attend session) Got Classes? 1 2 3 4 5 6 SPRING/SUMMER 2008 • VOLUME 15, NUMBER 2 71 We revised the 2007 survey by asking a series of questions about the “Got Classes?” session. An example with results is provided below. Got Classes? Please answer each of the following statements on a 1-5 scale, where: 1. Completely disagree 2. Somewhat disagree 3. Neither agree nor disagree 4. Somewhat agree 5. Completely agree I will be able to apply information provided to planning my academic degree program. Frequency Percent Valid Percent Cumulative Percent Completely disagree 4 0.8 0.8 0.8 Somewhat disagree 6 1.2 1.2 2.0 Valid Neither agree nor disagree 53 10.4 10.5 12.5 Somewhat agree 181 35.6 35.8 48.2 Completely agree 262 51.5 51.8 100.0 Total 506 99.4 100.0 Missing System 3 0.6 Total 509 100.0 From this example, one can see the transition from satisfaction to learning outcomes measurement a much more valuable critique of this particular orientation presentation. In total, for the selected four programs, we shifted from only four questions which assessed satisfaction only to a total of 13 questions, 11 of which focused on student learning outcomes items. This more comprehensive and meaningful look at our program gives us a better sense of what to change, add, or delete from our program. We disseminated the online survey to all students who attended SOAR (with the exception of the last session) in summer 2007. The survey was designed in and administered through Zoomerang zPro (an online survey software). Surveys were e-mailed the day after the student attended SOAR. Students were given 14-17 days to complete the survey and were also sent two reminders if they had not yet completed it by a particular date. Of the 2,503 survey invitations that were sent, 509 useable surveys were received (20% response rate). A drawing for a $100 bookstore gift certificate served as an incentive tool. 72 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION As with all social science research, the design and methodology of the SOAR survey had some inherent limitations. For example, this was a self-report survey instrument to which participants could choose whether to respond or not. In addition, it was not feasible to obtain a baseline level of knowledge to use for comparative purposes. It is our contention that this particular group of students does not have extensive knowledge (inputs) about the campus and its resources prior to SOAR. However, this is not controlled research and our intention is not to present it as such. The impact of confounding variables was diminished to the greatest extent possible. Findings did provide empirical evidence of student learning outcomes. Moreover, since we used a mixed approach of including satisfaction questions with learning outcomes measures, we were able to amass practical, satisfaction data along with the developmental outcomes necessary for program improvement. The feedback from this specific instrument will impact change not only in individual sessions, but also the SOAR program in its entirety. Our fundamental goal is to develop and build an orientation program based primarily on desired learning outcomes. In addition, we will use the results to include and inform participating departments about desired outcomes, the means by which to achieve those outcomes, and the assessment process that will determine if we are achieving what we hope to achieve. As with most orientation programs, assessing student learning will continue to be a collaborative effort. We will strive to educate our partners in this process and support them as they work to intentionally articulate the outcomes they want students to attain. Ultimately, moving to an assessment instrument which measures student learning outcomes will allow us to create an action plan and a new vision for programs. An intentional focus on learning outcomes will inform programmatic and policy decisions based on sound evidence. In other words, it will help us to answer the question of how we can improve student learning and growth. References American College Personnel Association and National Association of Student Personnel Administrators. (2004). Learning reconsidered: A campus-wide focus on the student experience. Council for the Advancement of Standards in Higher Education. (2006). CAS professional standards for higher education.(6th ed.). Washington, DC.