THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION1 EMERGING RESEARCH Longitudinal Comparison of Orientation Leader Learning Outcomes Lisa Gruszka, University of Minnesota – Twin Cities Evan Witt, University of Minnesota – Twin Cities Erin Tower, University of Minnesota – Twin Cities Orientation leaders (OLs) play a vital role on college and university campuses. They are primarily responsible for transitioning new students into institutions, yet little is known about the learning and development outcomes of the experience. This study compares longitudinal data from 10 years of orientation leader cohorts measuring their pre-, mid-, and post-learning across a set of seven institutionally endorsed student development outcomes (SDOs). Findings indicate growth across all of the seven SDOs within each year but not from year to year. The study makes a case that the orientation leader experience has a significant impact on student leader development. Orientation programs are a common method to transition new students to an institution. They usually contain informational programs and a peer leadership component led by orientation leaders (OLs). The scope of the OL role varies greatly by campus, but all OLs contribute to the focus of orientation programs as outlined by the Council for the Advancement of Standards in Higher Education: Orientation Programs (OP) must facilitate the transition of new students into the institution; prepare students for the institution’s educational opportunities and student responsibilities; and initiate the integration of new students into the intellectual, cultural, and social facets of the institution. Central to the mission of OP is the inclusion of parents/ guardians and families in support of the new student. OP must also contribute to institutional enrollment management, including retention. (2014) 2VOLUME 26 NUMBER 2 The cycle of an orientation leader program includes recruitment, selection, training, and job delivery. When recruiting and selecting these student cohorts, the demographics of the institution should be reflected in addition to seeking skills such as dependability, flexibility, and communication (Pretty, 2004). Training of an orientation leader can cover many topics but will generally cover skill-building in addition to the programmatic knowledge needed to deliver an orientation program (Pretty, 2004). It is believed that training has a positive impact on the students but there was not data or research to prove it. In 2008, this study created an assessment to measure growth in seven institutionally endorsed student development outcomes over the course of the OL program. This longitudinal study had two primary hypotheses: first, that OLs within each cohort would exhibit significant levels of growth over the course of the program; and second, that more recent cohorts would exhibit significantly more growth than less recent cohorts. Literature Review Few studies have looked at the specific growth and outcomes of orientation leaders. The structures of programs can vary widely and create a challenge in finding results that can be applied across contexts. One qualitative study that explored the areas of growth for orientation leaders was conducted by Gansemer-Topf and Economos (2012). In this study, they conducted a series of focus groups with orientation leaders after the completion of their role. After coding responses, four main themes emerged that were labeled as intrapersonal growth, skill development, understanding of others, and understanding of and pride in their institution (Gansemer-Topf & Economos, 2012). Intrapersonal growth included self-exploration, while skill development included areas such as learning skills and applying skills. Understanding of others addressed connecting across different backgrounds, while the institutional understanding related more to services, programs, and policies of the institution. The variety of skills and understanding demonstrates the wide scope of orientation leaders’ roles and responsibilities. Another qualitative study was created based on the work of self- authorship (Baxter Magolda, 2008; Kegan, 1994) to explore the ways in which self-authorship was developed through the orientation leader experience. THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION3 Hodges and Tankersley (2013) invited an OL cohort to participate in two semi-structured interviews to explore student experiences in the role. Five key themes emerged from these interviews. They were related to scheduling, diversity, trust, training, and independence (Hodges & Tankersley, 2013). These findings provide insight into the nuances and complexity of the OL role, which requires students to be adaptable, value diversity, and work independently. This study by Hodges and Tankersley positions this student employment as not only employment, but a significant learning opportunity. Taking a wider approach to orientation leader outcomes, learning outcomes in student leader education were formalized through the Learning Reconsidered publications delivered by NASPA (Keeling, 2004, 2006). Learning Reconsidered (Keeling, 2004) established that all college students should develop both inside and outside of the classroom, in addition to calling for staff to actively work toward delivering outcomes related to student learning. The 2006 version of the publication advanced the national conversation related to this work and gave concrete examples of how to create and implement learning outcomes. Campus Context This research was conducted at a land-grant, R1 Doctoral University with over 31,000 undergraduate students in the Midwest. The university’s orientation leader position has been in existence since the 1980s. These students represent the university during the summer orientation and, more recently, in the welcome week programs for first-year students. The OL cohort represents the larger campus community by including students from each of the seven freshman-admitting colleges, ethnic diversity, transfer students, and previous leadership experience, to name a few. Students must be at least in their sophomore year of school and cannot graduate until one semester after the position concludes. The rigorous selection process includes an application review, a group interview, a college of enrollment interview, and an individual interview with two student program coordinators. Orientation leaders begin their leadership development training, grounded in the Social Change Model of Leadership Development (Astin et al., 1996), by registering for a three-credit, upper-division elective course followed by a two-week summer orientation logistics training. OLs then work 4VOLUME 26 NUMBER 2 as a team to facilitate an average of 22 summer orientation programs in June and July. After a few weeks of vacation, they come back to campus in mid- August for a week of training to prepare each of them to train and supervise a group of 15-25 welcome week leaders. Instrument Development In 2005, the university’s vice provost of student affairs presented a set of student development outcomes created during their time as assistant dean and director of undergraduate programs in the business school. In this role, corporate recruiters were interviewed about what skills and attributes they were looking for in the hiring process. From that research, a set of student development outcomes (SDOs) were created. The idea was that learning occurred not only in the classroom but also outside the classroom, and these outcomes would provide a guide for students in developing the skills necessary to secure employment upon graduation: Undergraduates will demonstrate: • Responsibility and accountability by making appropriate decisions on behavior and accepting the consequences of their actions. • Independence and interdependence by knowing when to collaborate or seek help and when to act on their own. • Goal orientation by managing their energy and attention to achieve specific outcomes. • Self-awareness by knowing their personal strengths and talents and acknowledging their shortcomings. • Resilience by recovering and learning from setbacks or disappointments. • Appreciation of differences by recognizing the value of interacting with individuals with backgrounds and/or perspectives different from their own. • Tolerance of ambiguity by demonstrating the ability to perform in complicated environments where clear-cut answers or standard operating procedures are absent. THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION5 The vice provost encouraged departments to utilize the student development outcomes to assess one or two areas with their student employees/leaders. The department which conducted the longitudinal study presented in this paper decided to assess all seven areas and focused efforts on the orientation leader position. That year the process began of creating an assessment tool with the assistance of the university’s survey research center. Several versions of the assessment were piloted before being finalized. In the first two years of using this tool, adjustments needed to be made with the questions around goal orientation. In addition, a section was added to the post-test that asked OLs to self-report how they felt they progressed in each of the student development outcomes (improved/changed a lot; improved/ changed a little; no change; and changed for the worse). In 2008, the survey instrument was finalized and used to complete this longitudinal study. Methods Participants: Participants in this study were undergraduate students selected as orientation leaders between 2008 and 2017. Each of the 10 cohorts were similar in size, ranging from 24 to 28 OLs per year. Data collection: Data was collected using paper surveys administered in person to OLs. Each year the current cohort completed three surveys over the course of nine months. The pre-test was administered in January on the first day of the spring semester course, the mid-test was administered in May on the last day of the semester, and the post-test was administered in September on the last day of the position. Surveys consisted of 33 items assessing the degree to which specific skills and behaviors described the OL, measured on a five-point scale ranging from “not at all” to “to a huge degree.” Data was entered into Qualtrics, an online research tool that can build, distribute, and analyze survey responses. It was then exported to SPSS for analysis. Data analysis: A mean score was calculated for each OL for each SDO at the time of the pre-, mid-, and post-tests by averaging the ratings to individual survey items that correspond to each outcome. For each OL, the difference in mean ratings was calculated between pre- and post-tests and mid- and post-tests to create a “pre-post change score” and a “mid-post change score,” respectively. 6VOLUME 26 NUMBER 2 The first hypothesis was that OLs within each cohort would exhibit significant levels of growth over the course of the program. In order to test this hypothesis, paired sample tests to compare pre- and post-test ratings for each survey item were used, as well as the outcomes’ mean scores. The second hypothesis was that more recent cohorts would exhibit significantly more growth than less recent cohorts. This hypothesis was based on the idea that significant changes and improvements in the training program over the course of the study would lead to more growth in the more recent cohorts, To test this hypothesis an ANOVA and independent-samples Kruskal-Wallis tests were run to determine whether pre-test mean scores differ by cohort; pre-post change scores differ by cohort; and mid-post change scores differ by cohort. A Bonferroni adjustment was used to account for the multiple comparisons being carried out by ANOVA and Kruskal-Wallis tests. Finally, OLs’ baseline scores were different across cohorts as another component of variation between cohorts. Results Hypothesis 1: OLs exhibit significant levels of growth over the course of the program Data analysis results support our first hypothesis that OLs exhibit significant levels of growth over the course of the program. Results indicate that within each cohort, OLs exhibited statistically significant development across SDO areas at the p < 0.05 level. The paired samples tests found that in nine of the 10 cohorts, OLs’ post-test ratings were statistically significantly higher for all outcomes areas compared to pre-test ratings. In the 2015 cohort, the Goal Orientation outcome area exhibited a non-significant result with a p-value of .058. Additionally, on average, 24 of the 33 individual survey item ratings were statistically significantly higher at the time of the post-test than at baseline. These findings support our hypothesis that OLs within each cohort exhibit significant levels of growth over the course of the program. THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION7 Table 1. 2017 cohort pre- and post-test mean ratings and paired-samples t-test significance values at p<.05 Student Development Outcome Pre-test Post-test Sig. (2-tailed) Accountability & Responsibility 3.95 4.49 .000 Independence & Interdependence 4.08 4.35 .019 Self-Confidence & Humility 3.81 4.38 .000 Goal Orientation 3.92 4.22 .016 Resilience 4.04 4.32 .006 Appreciation of Differences 4.09 4.34 .003 Tolerance of Ambiguity 3.88 4.43 .000 Hypothesis 2: More recent cohorts exhibit more growth than less recent cohorts Data analysis results failed to confirm our second hypothesis that more recent cohorts exhibit significantly more growth in the SDO areas than less recent cohorts. Results indicate that the degree to which OLs develop over the course of the program remained relatively constant across cohorts. The pre-post change scores for six of the seven SDOs did not significantly differ across cohorts. In contrast, the results indicate that overall, cohorts were exhibiting similar levels of development over the course of the program. Similarly, the mid-post change scores for six of the seven SDOs did not significantly differ across cohorts. While these results indicate significant differences between select cohorts, they do not support our hypothesis that, overall, more recent cohorts exhibit greater levels of growth than less recent cohorts. Additionally, the pre-test mean scores for all SDOs remained consistent across cohorts and ANOVA and Kruskal-Wallis tests indicated there were no significant differences in pre-test mean scores by cohort. This finding indicates that all cohorts had similar baseline results, and all OLs began the program at roughly the same level of development. 8VOLUME 26 NUMBER 2 Figure 1. Average rating across all SDO areas at Pre-, Mid-, and Post-test by cohort Discussion The data collection journey for this project began in 2008 without knowing where it would take us. We knew that we wanted to evaluate the impact of the training and implement assessment measures for these SDOs. As time went on and data collection continued, it became clear that there was a unique opportunity for longitudinal comparison. Over the past 10 years there have been many organizational changes, including the implementation of a Welcome Week program and the addition of a staff member dedicated to student leader training and learning. The first hypothesis that was tested was to look for statistically significant outcomes within the pre- and post-assessment for our OLs. The analysis of this hypothesis found statistically significant growth each year in the student development outcomes from the pre-test to post-test for each cohort. This statistical significance speaks to the OL experience as a high-impact practice resulting in the development of students based on the institution’s outcomes. THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION9 The training program for Orientation Leaders is extensive and begins with a three-credit academic course focusing on leadership and personal development that is offered each spring and begins with a two-night overnight retreat. This course has a strong focus on identity development, social justice, diversity, inclusion, and leadership education. Upon the completion of this course, Orientation Leaders participate in an intensive two-week training program that prepares them for all of their summer job responsibilities and includes both training and testing components. The content in summer training focuses on understanding program logistics, learning presentations, memorizing scripts for a student life play, and mastering campus tours. Upon completion of this training they deliver five weeks of Orientation programming before a month-long break. After this break they return to campus for one last round of training for our Welcome Week program before they supervise a team of 10-15 volunteers and provide support during the Welcome Week program itself. This training and experience arc is delivered over nine months and has many components. Completing our data collection at the end of the experience speaks to the many roles in which these students perform, including as a learner, presenter, facilitator, performer, supervisor, and team member. When we ran our analysis to compare OL outcomes across the years, one of our hypotheses was that our changes to our training program and the additional professional staff member impacted OL learning. The data analysis did not show statistical significance between the outcomes of specific OL cohorts. Looking at each of the cohorts, their outcomes were similar regardless of the year they participated in our program. The consistency across the cohorts was surprising to us, especially considering the changes that have been made to the program over the years. In the early stages of the research, the Orientation Leader training was delivered by one Orientation Director who also had a full range of programmatic responsibilities. In 2014, a professional staff member was hired to solely focus on the recruitment, hiring, and training of student leaders including the Orientation Leaders. While our data told a different story than expected, the consistency of developing student outcomes each year is notable. To see that for 10 consecutive years our OLs achieved significant learning and growth through employment in our program is significant. Over those 10 years, changes 10VOLUME 26 NUMBER 2 continued to make improvements in both the program structure and staffing. The lack of difference between the cohorts indicates to us that the overall process is what promotes student learning. Implications for Practice Across our campuses the call for data to support our programs continues to increase. A major takeaway from this assessment project was the validation of our student leader training program as evidenced by the growth by OLs each year. It is important to have tangible outcomes of student leaders to be able to show this growth beyond simple anecdotes. This has helped us to advocate for additional resources for training and justification for hiring an additional staff member to solely support the training and development of our student leaders in 2014. It has also given us language to demonstrate across campus the impact of our programs. As orientation practitioners it can be challenging to find the time to develop detailed assessment plans with the competing interests of programming. It is important to remember that these plans ultimately not only improve our programs, but also help us better serve our current and incoming students. Another implication is that the assessment of the student development outcomes gives language to students about the explicit skills they are developing through their roles. Our training program lasts around six months, in addition to the three months of work responsibilities. Through this time it can be a challenge for students to grasp all of the skills they are learning; these outcomes have helped to ground the experience. It is affirming when we hear OLs talk about their skills in working through “ambiguity” or developing “resilience.” As practitioners and administrators, we urge you to be explicit in the learning outcomes you create and to prioritize the assessment of these outcomes. Because our analysis demonstrates significance of growth within each year but not across the years, we are left with the question, “How do we increase the learning?” As members of this field we need to look for more innovative ways to meet the changing needs of our students. In our next set of assessment efforts we have set the goal of demonstrating difference across upcoming cohorts by making our training curriculums more complex and incorporating more topics around personal development. We are having THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION11 discussions about how we can change our conversations and our work to include more about diversity, social justice, and inclusion to meet the changing landscapes of our society. We also are looking at new ways to deliver content that promote experiential learning and leveraging community partnerships to support students’ transitions after graduation. Limitations and Future Research Opportunities Our research project included some limitations. First, our assessment method relied on self-reported data. The quality of this data relied on our OLs to respond to the survey instrument in an honest and self-aware way. The data could be easily skewed to reflect aspirational outcomes rather than current developmental realities. Second, as we conducted our yearly assessment we did not document all of the training and staffing changes that could have had an impact on student learning. This part of the process would have been helpful in accounting for changes, or lack thereof, between cohorts of orientation leaders. As discussed earlier, over the past 10 years we have added professional staff and a welcome week program, which could have had an impact on the OL experience. At this time we are moving past our 10-year data collection on the SDOs and are planning to phase in a competency-based leadership assessment model (Seemiller, 2013). This approach will allow us to map competencies back to the student development outcomes while giving us the flexibility and specificity to focus on key leadership behaviors. We believe this new assessment approach will usher us into the next level of outcomes-based learning in our programs as we focus on competencies such as conflict negotiation, responding to change, and responsibility for personal behavior. Our hope is that this article contributes to the conversation about student leader outcomes in orientation programs. Our review of the literature demonstrated that little has been published on this topic even though orientation offices across the country contribute a lot of time and resources to these roles. There is a large research gap to be filled for campus practitioners to demonstrate high-impact practices in student training. In the next few years we hope more research and data is published to help better support the learning experiences of our student leaders. 12VOLUME 26 NUMBER 2 References Astin, A., Astin, H., Boatsman, K., Bonous-Hammarth, M., Chambers, T., Goldberg, S., et al. (1996). A social change model of leadership development: Guidebook (Version III). Los Angeles: University of California at Los Angeles, Higher Education Research Institute. Baxter Magolda, M. B. (2008). Three elements of self-authorship. Journal of College Student Development, 49(4), 269-284. Council for the Advancement of Standards in Higher Education (2014). Orientation Programs. Gansemer-Topf, A., & Economos, K. (2012). What Student Leaders Learn from the Student Orientation Leader Experience. The Journal of College Orientation and Transition, 20 (1), 19-30. Hodges, J., & Tankersley, C. (2013). Becoming an Orientation Leader: A Catalyst for Self-Authorship and Development. The Journal of College Orientation and Transition, 20 (2), 46-60. Keeling, R. P. (Ed). (2004). Learning Reconsidered: A Campus-wide Focus on Student Experience. Washington, D.C.: American College Personnel Association and National Association of Student Personnel Administrators. Keeling, R. P. (Ed.) (2006). Learning Reconsidered 2: Implementing a Campus- Wide Focus on the Student Experience. Kegan, R. (1994). In over our heads: The mental demands of modern life. Cambridge, MA.: Harvard University Press. Pretty, B. A. (2004). Staff selection: Effective people for effective programs. In M. J. Fabich (Ed.), Orientation planning manual (pp. 11-14). Flint, MI: National Orientation Directors Association. Seemiller, C. (2013). The student leadership competencies guidebook: Designing intentional leadership learning and development. San Francisco, CA: Jossey-Bass.