THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION1 FEATURED ARTICLE Belonging and Mattering in the First-Year Welcome Experience: A Comparison Study Before and During COVID-19 Ling Ning, University of Colorado Boulder Kimberly A. Kruchen, University of Colorado Boulder Crystal S. Cyr, University of Colorado Boulder Institutions of higher education value knowledge about the programs and services that enhance the collegiate experience for students, particularly related to their sense of belonging and mattering. This knowledge has become pivotal given the recent transition of programs and services to a virtual environment due to the COVID-19 pandemic. This study employed machine learning methods to analyze and compare student responses to a Fall Welcome Survey in 2019 and 2020. The purposes were threefold: (a) to identify, rank, and contrast the top contributors to their sense of belonging and mattering; (b) to understand the impact of the pandemic on students’ welcome experience; and (c) to introduce and review an AI-leveraged analytical visualization tool for key influencer analysis. Results indicated a sharp decline in students’ experiences of connection, belonging, and mattering between fall 2019 and fall 2020 due to the pandemic. Results also revealed the three most common and significant contributors to students’ expressing a high level of belonging and mattering were opportunities to build connections, an overall positive move-in experience, and a positive welcome experience. Keywords: Fall welcome experience, key influencer, belonging, mattering, impact of the pandemic As with most of the world, the COVID-19 pandemic greatly altered students’ experiences coming to higher education institutions. Besides having to adjust to virtual classes, it also created profound changes in traditional orientation programs and services. Institutions worked quickly to change the modality, identify the interface, and develop the content necessary to ensure a successful on-campus pandemic experience. While there are many phases to a student’s collegiate career, we focus on the concentrated and intentional set of experiences that begin as they move onto campus and proceed through a series of Welcome Week programs. Establishing an effective welcome experience is important, and institutions provide a variety of options, including moving into a residence hall, wellness events, an academic welcome, concerts, and barbeques, among others. Semi-structured organic experiences such as meeting new friends and setting up a residence hall room are also important to include in the experience. A pressing question for all institutions is which of these formal programs and services are most impactful during the welcome experience. These questions have become even more important based on the challenges posed by the pandemic, which has caused institutions to make rapid decisions about student support programs and services. Understanding the outcomes associated with moving key supports to a virtual environment is critical. Done well, a week that welcomes students is a cadre of programs and activities that lays a foundation for a 2VOLUME 28 NUMBER 2 successful transition to college. Our institution approaches programs offered during the student’s first week with purpose and intentionality, seeking to establish a strong foundation for the college experience. An institutional team composed of several functional units works to achieve an effective student welcome by focusing on three main themes that guide the overarching experience: preparation, belonging, and mattering. Specific cross- functional outcomes, also developed by this institutional team, are necessary to make strategic decisions and assess the effectiveness of the experience. The Council for the Advancement of Standards’ First-Year Experience Cross-Functional Framework (Young & Keup, 2019) emphasizes the importance of “attention to quality and integrity of practice” and “strategies and tactics for addressing improvement of student learning and program/institutional outcomes” (p. 6). Specifically, the cross-functional framework suggests that a “plan must describe data needed to address the profile of the current cohort” (Young & Keup, 2019, p. 25). The institutional plan collects information from a variety of sources, including incoming student data gathered during the summer, an event rubric, micro-surveys targeting event- specific data, and a comprehensive end-of-the-experience survey. These sources allow the group to interpret the achievement of the program outcomes and are critical to analyzing overall experience effectiveness. For this more comprehensive analysis, we use the Microsoft Power BI tool. A Historical and Theoretical Review of First-Year Welcome Experiences Supporting students in their transition to postsecondary education has been the responsibility of higher education professionals for centuries as the transition from a high school student into a college student can be a significant moment in the lives of young adults. Research from various scholars, including Rentz and Saddlemire (1988), tells us that orientation and new student programs provide students with an opportunity to determine institutional fit. Institutions provide a variety of experiences to welcome students into their communities. According to the Council for the Advancement of Standards in Higher Education (2015), Harvard was one of the first colleges to formalize transition support and offer peer-to-peer mentoring services for incoming students, while the first freshman week took place at the University of Maine in 1923 (Packwood, 1977). Since then, there have been significant changes to orientation and welcome programs. Programs have evolved in myriad ways, including online delivery, with events happening closer to the start of the academic year. Regardless of the format, orientation and welcome programs have expanded to include academic and social integration into the college or university—two elements of the postsecondary experience that Tinto (1987) deemed crucial to student retention and persistence. College orientation programs across the country “encapsulate the essence of their institutions by introducing new students to the academic life, culture, traditions, history, people, and surrounding communities” (Mack, 2010, p. 20). Noel et al. (1986) suggested orientation programs create a staying environment for students. As such, this campuswide endeavor was a call to action for many campus administrators who tied it to retention efforts. Extensive research has been conducted in recent decades to better understand the role of transition programs such as orientations and welcome weeks on student retention, undergraduate achievement, and overall student success. Generally, institutions ensure that programs and activities offered during these transition experiences foster connection and socialization, building on the foundation of shared experiences, to help students make friends and develop a sense of community within the institution. Numerous studies have found that the successful transition to a higher education environment depends widely on the individual student’s perception of the transition, coping strategies, and supports moving forward THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION3 (Goodman et al., 2006). Schlossberg’s transition theory describes transitions as events or non-events resulting in changed relationships, routines, assumptions, or roles. They can take the form of anticipated transitions, unanticipated transitions, or non-events (Goodman et al., 2006). Research describing the impact of a pandemic on the college student transition is still in progress. However, the fall 2020 transition for many incoming first-time, full-time students on residential college campuses may be considered an unanticipated transition due to the uncertainty caused by the pandemic. Others might describe their fall transition as a non-event because established expectations for move-in and fall welcome experiences were not met due to the stringent policies in place from the COVID-19 pandemic. Despite the differences in transition type, Goodman et al. (2006) suggest that factors influence the individual student’s ability to cope with and move successfully through the transition: situation, self, support, and strategies, also known as the 4 S’s of transition. Situation includes trigger, timing, control, role change, duration, previous experience with a similar transition, concurrent stress, and assessment (Goodman et al., 2006). Orientation and welcome programs play a critical role in delivering activities, events, and opportunities that address these factors in offering students support. The psychological resources that factor into self, specifically student self-efficacy, commitment and values, and resiliency, are addressed through a variety of programming in which students discover support from the institution and their peers. When students transition from their familiar environments and networks to the uncertainty of a new setting with unfamiliar people, they often experience feelings of marginality (Schlossberg, 1989). During this period of loneliness and social isolation, students strive to feel that they matter to someone, so they seek opportunities to feel noticed and valued (Schlossberg, 1989). Welcome programs intentionally create activities for students to connect with other new students, current student leaders, faculty, and professional staff in a personalized setting. These efforts address the factors necessary for establishing a sense of mattering, including attention, importance, dependence, and appreciation (Schlossberg, 1989). When students feel like they matter, institutions can facilitate a sense of belonging through socialization and intentionally constructed experiences and environments (Astin, 1999). Maslow (1962) describes belonging as a basic human motivation in which all people share a strong desire to belong to a group or community. Welcome programs offer extensive opportunities for students to develop an “affiliation and identification with the university community,” creating, in turn, a sense of belonging that may impact retention and persistence through graduation (Hoffman et al., 2003, p. 228). Institutional Context A significant amount of the institutional introduction and adjustment to student life happens during a diligently planned, multi-week experience. Specifically, our institution considers the online orientation, the move-in process, and welcome events all part of the Fall Welcome experience. Online orientation includes various modules to be completed in our learning management system before registration. The move-in process begins with the communication a student receives prior to their campus arrival, and before the pandemic, an opportunity to have members of the campus community help move their belongings into their room. In 2019, the Fall Welcome experience included a range of in-person events, including 10 signature events. During fall 2020, the pandemic required us to prioritize the safety and health of students and deliver all events via an online platform. After these events, students received a comprehensive survey to assess their participation and rate their experiences. While this is an unprecedented time for higher education, the constraints posed by the pandemic may 4VOLUME 28 NUMBER 2 continue, even after it has ended. Resources, specifically time and funding, will continue to be limited. It is incumbent upon institutions to use this information to ensure students have access to programs and experiences proven to increase their sense of belonging and mattering. To this end, three research questions guided the present study: 1. Which Fall Welcome events or experiences contributed to students’ expressing a high level of belonging and mattering? 2. How did the impact of the Fall Welcome events or experiences vary across different student subpopulations? 3. How did COVID-19 affect students’ overall welcome experience? Method The purpose of the key influencer analysis was to examine and compare the extent to which the 2019 and 2020 Fall Welcome events and experiences facilitated a student’s sense of belonging and mattering. The goal was to understand how the pandemic influenced the effectiveness of fall welcome programming and how we might help students feel connected to the campus despite the situation created by the pandemic. The analytic tool used to construct the explanation model in this study was the machine learning method embedded in the key influencer visual within the Microsoft Power BI Desktop software. Microsoft Power BI is a data analytics and reporting platform that connects software services and applications to turn data sources into coherent, interactive, and insightful information. Powered by artificial intelligence (AI), the key influencer feature allows student affairs professionals to easily leverage machine learning capabilities to gain insight into student data. What makes the use of the key influencer feature within Power BI more appealing than traditional analytical applications are that • the machine learning methods embedded in the feature do not require strong data distribution hypotheses, which educational and observation data often fail to meet; • machine learning methods are capable of handling very large data sets, including thousands of rows (i.e., student records) and columns (i.e., variables); • the iterative process incorporated in machine learning can minimize estimation error, leading to more accurate and robust modeling; and • the key influencer tool provides visualizations that facilitate the interpretation of modeling results. The advantages of the key influencer tool have made it a valuable application for student affairs professionals to explain relationships and make predictions. For this study, survey data were collected at a public research university located in the western region of the United States. This residential institution enrolls approximately 35,000 undergraduate and graduate students annually. The annual Fall Welcome week generally occurs during the first two weeks of each new academic year. A combination of entertaining and informative events offers students the opportunity to explore the campus environment and resources and to build connections, thus facilitating a sense of belonging and mattering. In 2019, Fall Welcome occurred between August 20th and August 30th. The 2020 Fall Welcome lasted for about two weeks, from mid-August to early September, to meet pandemic-related health and safety guidelines. Each year immediately following the conclusion of events, a survey is sent to all incoming first-year and transfer students. The survey contains six major aspects for evaluation: pre-arrival communication, the online orientation prior to arriving on campus, moving into a residence hall, Fall Welcome event participation and experience, connection to campus, and students’ self-reported sense of belonging and mattering. The present study examined the relationship between major components in the survey to (a) understand the significant THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION5 factors that contributed to students’ expressing a high sense of belonging and mattering upon their arrival to campus; (b) examine the impact of the pandemic on the students’ welcome experience; and (c) introduce and review an AI-leveraged analytical visualization tool for key influencer analysis. Survey Participants The 2019 Fall Welcome Survey was sent via email to more than 9,000 incoming first-year and transfer students. The total number of responses received was approximately 2,200, resulting in a response rate of 20%. During fall 2020, 8,000 incoming first-year and transfer students received an email invitation to fill out the Fall Welcome Survey, and 1,440 students responded, resulting in an 18% response rate. The response rate varied by question for both surveys because all responses were voluntary. Both surveys were incentivized to increase the response rate. Table 1 presents the primary demographic breakdown for each survey. The distribution shows that in fall 2019 and fall 2020, responding students had a similar demographic composition. Table 1. Demographic Information of Participating Students for Each Survey Variables Fall 2019 Survey (N = 2,200) Fall 2020 Survey (N = 1,400) Gender identity Male 28% 31% Female 43% 45% Gender non-conforming 1% 0.6% Prefer not to state or missing 28% 23% Race/ethnicity White 65% 66% Hispanic/Latino 14% 14% Asian 12% 12% Other (e.g., American Indian, Black/African American, Pacific Islander, International, Unknown) 9% 8% Entry type First-year undergraduate student 86% 86% Transfer student 14% 14% First-generation status (%Yes) 19% 19% Out-of-state student (%Yes) 41% 36% Disability (%Yes) 25% 17% Variables Independent variables. The survey gathered feedback about the signature Fall Welcome to inform future event design and implementation. Students who reported participating in these events were asked to rate, on a scale 1 to 6, how satisfied they were with the quality of each event they attended. Tables 2 and 3 present the descriptive information for the ratings of each event in fall 2019 and fall 2020. 6VOLUME 28 NUMBER 2 Table 2. Satisfaction Ratings for Signature Fall Welcome Events, Fall 2019 Signature Fall Welcome events N Participation rate % Satisfaction M SD Football game 1,026 47% 90% 5.50 0.80 Carnival 1,093 50% 77% 5.07 0.94 Kickoff rally 1,204 55% 73% 4.99 0.91 Student organization fair 1,132 51% 72% 4.95 0.93 Tailgate 447 20% 70% 4.99 0.97 Concert 1,073 49% 67% 4.71 1.42 Recreation Center open house 878 40% 65% 4.79 1.06 College days in academic major 1,188 54% 56% 4.53 1.06 Bystander intervention training 1,601 73% 55% 4.52 1.02 Convocation 1,279 58% 32% 3.78 1.33 Note. % Satisfaction is the sum of % Satisfied and % Very Satisfied. Table 3. Satisfaction Ratings for Signature Fall Welcome Events, Fall 2020 Major Welcome Events N Participation Rate % Satisfaction M SD Virtual historical ghost tour 15 1% 100% 5.40 0.61 Self-led neighborhood tour 41 3% 95% 4.63 0.72 Disney trivia 24 2% 79% 4.21 1.12 Affinity groups virtual meet-up 60 4% 85% 4.40 1.02 Sexual health comedy program 39 3% 100% 5.23 0.77 Video gaming 44 3% 93% 5.09 0.90 Convocation 136 9% 86% 4.43 0.97 Celebrity guest speaker 156 11% 93% 5.20 1.02 College days in academic major 102 7% 90% 4.58 0.93 Student organization fair 152 11% 85% 4.59 1.01 Guided painting class 44 3% 96% 5.23 0.93 Note. %Satisfaction is the sum of %Satisfied and %Very Satisfied. Other variables included in the machine learning model were students’ overall move-in experience, overall Fall Welcome experience, satisfaction with the management of traffic and parking on campus, and satisfaction with the advanced communication. A 6-point response scale was also used in these four questions, with 1 indicating strongly disagree and 6 indicating strongly agree. Table 4 presents detailed descriptive information about the variables measuring the major aspects of the Fall Welcome experience. One important variable is whether students established connections with peers over the Fall Welcome week(s). The question invited students to report whether they had met any other students during Fall Welcome with whom they expect to stay in contact, and if so, approximately how many. Figure 1 has the distribution details of the responses to the question from both surveys. THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION7 Table 4. Satisfaction Ratings for Major Aspects of Fall Welcome Experience Statement N % Agreement M SD Fall 2019 Fall 2020 Fall 2019 Fall 2020 Fall 2019 Fall 2020 Fall 2019 Fall 2020 Overall, I had a positive move-in experience. 1,530 728 86% 86% 5.26 5.23 0.91 0.93 Overall, Fall Welcome was a positive experience for me. 1,553 725 74% 52% 4.91 4.30 0.96 1.28 The traffic and parking on campus during move-in was managed well. 1,522 722 66% 79% 4.75 5.08 1.24 1.03 Overall, I was satisfied with the advance communications I received about the move-in process. 802 423 74% 75% 4.86 4.89 0.99 1.04 Note. %Agreement is the sum of %Agree and %Strongly Agree. Figure 1. The number of other students respondents expected to stay in contact with. 8VOLUME 28 NUMBER 2 Dependent variables. Students’ sense of belonging and mattering is a 6-item, self-reported instrument consisting of two independently scored scales of students’ perceived sense of belonging and mattering. All items were measured on a 6-point scale, for which the response categories were 1 for strongly disagree and 6 for strongly agree. Construct validity and internal consistency were examined through a series of studies. Cronbach’s α ranged from 0.92 to 0.94 for the belonging scale and from 0.87 to 0.90 for the mattering scale, suggesting the items have a high degree of internal consistency. Confirmatory Factor Analysis was conducted to investigate construct validity. The model fit indices of CFI (comparative fit index) and TLI (Tucker-Lewis Index) gave values of 0.97 and 0.94. respectively. These values exceed the threshold value of 0.90, indicating a two-factor model was a good fitting model to the data. Table 5 has the descriptive information of the responses to the scales in both surveys. The responses to the sense of belonging and mattering items were then recalibrated into three levels based on the average score across the respective statements (i.e., Low, M ≤ 3.5; Medium, 3.5 < M <5; High, 5 ≤ M ≤ 6). Table 6 presents the percentage and counts for each level of belonging and mattering. Table 5. Descriptive Statistics of Responses to Belonging and Mattering N M SD Statement Fall 2019 Fall 2020 Fall 2019 Fall 2020 Fall 2019 Fall 2020 Sense of Belonging I belong at the institution. 1,735 869 4.94 4.69 0.95 1.08 I fit in well at the institution. 1,735 872 4.77 4.18 1.03 1.28 I feel a connection with the institution community. 1,739 870 4.63 4.48 1.07 1.13 Mattering There are people on campus who are genuinely interested in me as a person. 1,600 868 4.74 4.56 1.03 1.01 People on campus are generally supportive of my individual needs. 1,602 872 4.72 4.39 0.91 1.21 There are people on campus who care about my future. 1,599 871 4.58 4.52 1.11 1.13 Table 6. Descriptive Information of Levels of Belonging and Mattering N Percentage Fall 2019 Fall 2020 Fall 2019 Fall 2020 Level of Belonging High 446 176 28% 20% Medium 957 543 60% 63% THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION9 Low 184 145 12% 17% Level of Mattering High 370 160 23% 19% Medium 1,083 589 68% 68% Low 134 115 8% 13% Control Variables. The students’ demographic variables (i.e., gender identity, race/ethnicity), disability status, entry type (i.e., first-year undergraduate student or transfer student), residency status (i.e., in-state or out-of-state), and first-generation status were included in the final machine learning model to control for their potential effects related to the impact of Fall Welcome experiences on students’ expressed sense of belonging and mattering. These variables were included in the analysis to estimate the variation of impact within each major student subgroup. Analytic Procedures Analysis was conducted using descriptive statistics to clean and summarize the responses collected. The Power BI reporting platform was used to tabulate and graph summary results. After confirming the consistency and accuracy of the outcomes measured, the key influencers analytics tool within Power BI Desktop was used to construct machine learning models. The purpose was to identify, rank, and contrast the factors contributing significantly to the outcome metrics. After installing Power BI Desktop and importing the key influencers visualization, the analytical process was simple and straightforward. We provided the key influencers tool the outcome variable of interest, the three levels of belonging and mattering, and the explanatory variables, including the above-mentioned independent variables and control variables. The analysis was conducted for the two outcomes, sense of belonging and mattering, using the data from the fall 2019 and 2020 surveys. Results Declines in Fall Welcome Event Participation Due to COVID-19 Participation rates for fall 2019 events ranged from a minimum of 20% at the football tailgate to a maximum of 73% for bystander intervention training. The satisfaction rating also varied substantially from one event to another, with satisfaction ranging from 32% to as much as 90%. Due to pandemic restrictions, signature events were adapted for virtual environments in fall 2020. The participation rate declined sharply, and most events had less than a 5% participation rate. The celebrity guest speaker and student organization fair had the highest participation rates at approximately 11%. However, among the students who did participate in the events, more than 85% rated them good or very good, except for a trivia event. Only 79% of participants rated this program good or very good. Declines in Connection Opportunities Due to COVID-19 The restrictions in place due to the pandemic led to the accelerated use of social media and virtual applications (e.g., Zoom, Google Meet, Skype) in facilitating social interactions and events. As a result, students missed out on organic opportunities to interact with their peers. Survey responses demonstrate a sharp decline in the number of quality connections among students. Approximately 20% of the fall 2019 survey respondents said they met no other students with whom they expected to stay in contact. Among the fall 2020 survey respondents, 64% said they did not meet any other students. Moreover, only 7% of responding students in fall 2020 indicated they met five or more students with whom they planned to stay in contact, compared to 21% in fall 2019. 10VOLUME 28 NUMBER 2 Declines in Sense of Belonging and Mattering Due to COVID-19 Restrictions The average for each of the statements that measure sense of belonging and mattering dropped significantly from fall 2019 to fall 2020, except for one mattering measure: There are people on campus who are concerned about my future. The mean and standard deviation of the responses to that statement were almost identical between the two surveys. One possible reason is students might have interpreted the word “concerned” differently, and such differences were canceled out when we averaged the responses. The tabulation of the levels of belonging and mattering showed that about 20% of the students expressed a high level of belonging and mattering, 60% to 70% a medium level, and 10% to 17% a low level. A comparison also indicated that students reporting a high level of belonging decreased by 8% from fall 2019 to fall 2020. More students expressed a low level of belonging in fall 2020 than in fall 2019. Similar trends were observed for the responses to the mattering statements. Key Influencers for Belonging and Mattering Across both surveys, meeting five or more new students with whom they expected to stay in contact, having a very positive overall move-in experience, and being satisfied with the Fall Welcome experience had the greatest impact on students’ perceived level of belonging and mattering. Analysis of the survey responses from fall 2019 indicated that if students met no other students with whom they expected to stay in contact, they were 2.8 to 3.1 times more likely to experience a low sense of mattering and belonging, respectively. In fall 2020, the likelihood of experiencing low mattering and belonging increased to 4.3 times and 5.4 times, respectively, if students did not meet any peers with whom they hoped to stay in contact. The other most impactful factor to mattering was a student’s perception of the overall move-in experience. Based on the responses in fall 2020, the likelihood of them experiencing a low sense of mattering increased by 4.4 times if they disagreed that their overall move-in experience was positive compared to 2.6 times in fall 2019. Another impactful contributor to a strong sense of belonging and mattering is their response to having an overall positive Fall Welcome experience. In particular, the findings showed that for Fall 2020, the likelihood of experiencing a strong sense of mattering would increase by about 3 times if the respondents agreed they had a positive Fall Welcome experience. Whether students ate meals with their roommates or cohort mates/hall mates also contributed significantly to a sense of belonging and mattering. Table 7 and Table 8 present the key influencers related to belonging and mattering for fall 2019 and fall 2020. Table 7. Key Influencers to the Level of Belonging and Mattering Based on 2019 Fall Welcome Survey Data Key influencers Level of sense of belonging Level of mattering High Medium Low High Medium Low Likelihood sense of belonging would be high/medium/low Likelihood mattering would be high/ medium/low Satisfaction = 6 Satisfaction = 5 Satisfaction ≤ 4 Satisfaction = 6 Satisfaction = 5 Satisfaction ≤ 4 Major Events Convocation 2.32x 1.29x 2.20x 1.20x 1.97x Recreation Center open house 1.73x 1.25x 1.78x 1.75x 1.19x 1.71x THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION11 College days in academic major 2.08x 1.48x 2.05x 2.10x 1.34x 2.50x Kickoff rally 2.24x 1.43x 2.37x 2.32x 1.35x 2.05x Carnival 2.05x 1.29x 2.32x 2.05x 1.25x 2.21x Bystander intervention training 2.14x 1.60x 1.75x 2.16x 1.42x 2.30x Concert 1.86x 1.20x 2.47x 1.78x 1.17x 1.90x Student organization fair 1.87x 1.32x 1.67x 2.11x 1.30x 2.40x Tailgate 1.02x 1.44x 2.68x 2.10x 1.48x Football game 2.16x 1.24x 2.93x 1.91x 1.20x 2.30x Note. x denotes times. For example, the likelihood of students’ experiencing a high sense of belonging would increase by 2.32 times if students were very satisfied with the Convocation event when keeping all other factors constant in the model. Table 7. Continued. Key influencers Level of sense of belonging Level of mattering High Medium Low High Medium Low Likelihood sense of belonging would be high/medium/low Likelihood mattering would be high/ medium/low Satisfaction = 6 Satisfaction = 5 Satisfaction ≤ 4 Satisfaction = 6 Satisfaction = 5 Satisfaction ≤ 4 Experience rating Overall positive move- in experience 2.41x 1.30x 3.09x 2.55x 1.27x 2.59x Overall positive fall welcome experience 1.37x 1.29x 1.49x 1.59x Did you meet any other new students during Fall Welcome that you expect to stay in contact with? If so, approximately how many? 2.38x (If meet 5 or more other students) 1.67x 1.91 x (If meet One student)/ 3.06 x (If meet No students) 2.12x (If meet 5 or more other students) 2.80x (If meet No students) / 1.91x (if meet One student) I ate meals with my roommates and/or hallmates. 1.82x (if yes) 2.47x (if no) 12VOLUME 28 NUMBER 2 The traffic and parking on campus during move-in was managed well. 1.74x 1.31x 1.48x 1.70x 1.15x Overall, I was satisfied with the advance communications I received about the move-in process. 1.83x 1.19x 1.91x 2.02x 1.19x Student demographics Student status (first- year / transfer) Gender (M/F) 1.34x (If Female) First-Gen status (Y/N) 1.31x (if No) Disability status (Y/N) 1.30x (If No) 1.75x (If Yes) 1.76x (if Yes) Race/ethnicity (White/Hispanic/ Asian/Other) 1.21x (If White) 1.13x (If Asian) 1.58x (If Hispanic) Note. x denotes times. For example, the likelihood of female students’ experiencing a low sense of belonging is 1.34 times higher than male students when keeping all other factors constant in the model. Table 8. Key Influencers to the Level of Belonging and Mattering Based on 2020 Fall Welcome Survey Data Key influencers Level of sense of belonging Level of mattering High Medium Low High Medium Low Likelihood sense of belonging would be high/medium/low Likelihood mattering would be high/ medium/low Satisfaction = 6 Satisfaction = 5 Satisfaction ≤ 4 Satisfaction = 6 Satisfaction = 5 Satisfaction ≤ 4 Major event(s) Disney trivia 1.26x 1.15x Experience rating Overall positive move- in experience 1.52x 1.74x 2.95x 4.41x Overall positive fall welcome experience 2.98x 1.62x 1.72x 2.84x 1.33x THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION13 Did you meet any other new students during Fall Welcome that you expect to stay in contact with? If so, approximately how many? 1.92x (If meet 5 or more other students) 1.77x (If meet 2-4 other students) 5.40 x (If meet No students) 2.10x (If meet 5 or more other students) 4.32x (If meet No students) Level of connection with your cohort community 1.66x(if moderate or strong) 1.82(if not strong) 3.34x(if no) 2.28x(if minimal to strong) 2.97x(if no connection) Attended Fall Welcome events with my roommates and/ or cohort 2.63x(if yes) 3.93x(if no) 2.33x(if yes) 3.32(if no) I ate meals with my roommates and/or cohort mates. 2.20(if yes) 1.93x(if no) 1.45(if yes) 1.10(if yes) 2.09(if no) The traffic and parking on campus during move-in was managed well. 2.03x Overall, I was satisfied with the advance communications I received about the move-in process 2.22x 1.10x 1.97x 1.41x 2.59x Student demographics Student status (first- year / transfer) 1.56x (If first-year) First-Gen status (Y/N) 1.59(If non- First Gen) Disability status (Y/N) 1.27(If No) Race/ethnicity (White/Hispanic/ Asian/Other) 1.48x (If White) 1.41x (If Asian) 1.38x (If White) Note. x denotes times. For example, the likelihood of students’ experiencing a high sense of belonging would increase by 1.26 times if students were very satisfied with the Disney Trivia virtual event when keeping all other factors constant in the model. The key influencer analysis of the Fall 2019 survey revealed that some signature events were more 14VOLUME 28 NUMBER 2 impactful than others, contributing significantly to a strong sense of belonging and mattering. The degree of impact estimated for Fall Welcome events and move-in experiences was also found to vary by students’ demographics. Table 7 contains the details of the estimated overall impact. For example, as it relates to transfer students, the top contributing factors were connection (whether they met five or more new students with whom they expected to stay in contact) and satisfaction with college day with academic major, bystander intervention training, and the student organization fair. Transfer students did not think events like convocation, the kickoff rally, and the football game were as important as first-time, first-year students did. The connection with other new students was particularly important to first-generation students. When all other factors were kept constant, if the first-generation students did not meet other new students they expect to stay in contact with, the likelihood of their experiencing a low sense of mattering would increase by 2.5 times. For Hispanic/Latino students, connection with other new students and an overall positive move-in experience were also particularly important. Almost none of the virtual events hosted during the 2020 Fall Welcome had participation rates high enough to be estimated as impactful on students’ sense of belonging and mattering. Trivia was found to be impactful to some degree: a good rating for the event would increase the likelihood of students experiencing high belonging by about 1.3 times. However, a careful review of the descriptive distribution of the satisfaction ratings of all events by the participating students showed that a vast majority rated the events as good or very good. For future planning, communicating about events and maximizing student outreach opportunities to encourage and promote participation would be helpful to obtain sufficient participation rates, allowing for the possibility to quantify the impact of (virtual) events on student outcomes. Discussion Research suggests that connection and socialization are important to students developing a strong sense of belonging and mattering on college campuses. Our analysis not only supports the literature (Strayhorn, 2012) but also suggests that a key influencer to first-year student’s sense of belonging and mattering is developing five or more new connections during their welcome experience. The pandemic has profoundly changed the college experience, particularly for first-year students who are navigating an already challenging transition in search of new connections and community. The decline of students’ reported sense of belonging and mattering between fall 2019 and fall 2020 made this especially evident. When providing virtual experiences for students during the welcome experience, it is important to intentionally structure opportunities to meet and connect with other students in a meaningful way. Clearly communicating the next steps for encouraging students to nurture these new relationships beyond the structured activities in more organic settings (e.g., eating meals together) facilitates opportunities for increasing belonging and mattering. The response to the pandemic required prioritizing the health and safety of students and staff, which meant shifting events and experiences to a virtual environment. Unforeseen consequences, such as lack of knowledge about events and the modality in which events took place, likely contributed to lower participation rates. In turn, students experienced lower rates of belonging and mattering. Communication about these types of details is critical for student participation. For example, traditional marketing strategies such as posters or other visual techniques were not as impactful, and an over-reliance on email may have caused students to overlook event marketing. Technology was another challenge. Students reported difficulty accessing the platforms used to host events. Comments about challenges included not locating the correct link or difficulty joining the virtual event. Institutions continue to learn and eliminate barriers. THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION15 The findings also reveal the importance of creating a very positive overall move-in and Fall Welcome experience for all students, particularly for underrepresented student populations. Compared to the majority of responding students who were White, non-first generation, and had no disabilities, students from underrepresented groups did not experience the same level of belonging and mattering despite attending the same events. It is imperative event coordinators design events that provide comparable, inclusive, and equitable experiences for all students. We continue to advance our assessment capabilities through the use of new statistical tools to predict outcomes. The analysis technique described in this study provides an accessible yet powerful tool that can increase the speed of analysis and allow professionals to make decisions more quickly. While it is valuable to have some familiarity with statistical modeling to use this tool, it is not required, opening up the possibilities to a variety of users. Limitations The key findings of this study should be considered within the context of several limitations. The findings outlined were generalized only to the students who responded to the surveys at one particular institution. Other strategies should be contemplated when selecting survey samples and achieving adequate response rates before generalizing the findings to the whole student population. Moreover, each institution has a unique student population and priorities related to a transition experience. Given the diversity of higher education institutions, these findings may not be relevant to other institutions in different geographic regions. There is a need to consolidate and standardize quality measures and conceptualization of college orientation and fall welcome to incorporate a broader and generalizable student experience. Another limitation is the narrow focus on sense of belonging and mattering. Other outcomes such as student engagement, involvement, connection, and transition should be assessed to gain a more comprehensive picture of the impact of a fall welcome experience, particularly during the pandemic. Sample size sparsity also presented a challenge when estimating the impact of events, particularly for students from underrepresented groups. Lastly, the low participation in virtual events related to COVID-19 health and safety protocols limited our ability to quantify the event’s impact on student outcomes. Conclusion This analysis identifies the degree to which the COVID-19 pandemic has influenced the effectiveness of Fall Welcome programming and students’ Fall Welcome experiences, particularly their sense of belonging and mattering, given the transition of most programs and services to a virtual environment. This study employs a machine learning method embedded in Microsoft Power BI Desktop (i.e., the key influencer analytic tool) to analyze and compare student responses to a Fall Welcome Survey in 2019 and 2020, before and during the COVID-19 pandemic. The study shows a sharp decline in Fall Welcome event participation from fall 2019 to fall 2020. Additionally, the use of social media and virtual applications to facilitate social interactions and events has deprived students of organic interaction opportunities, resulting in feeling a lack of connection to other students, as indicated by responses to the fall 2020 survey. The study also identifies the three most impactful and consistent factors that contribute to students’ expressing a high level of belonging and mattering before and during the pandemic. The key influencer analysis reveals that certain events during the Fall 2019 Welcome had a greater impact than others in helping students feel a strong sense of belonging and mattering. However, due to the low participation rate, we could not estimate the impact on students’ sense of belonging and mattering for most events in fall 2020. Further studies with robust data may help identify the impact of virtual Fall Welcome 16VOLUME 28 NUMBER 2 events on students’ outcomes. Future research may also focus on creating organic and meaningful interactions amid virtual experiences that enhance students’ connection and facilitate their sense of belonging and mattering. The COVID-19 pandemic challenged colleges and the communities they serve to shift and evolve programs and services. Despite these challenges, colleges have produced innovative programs to support students’ sense of belonging and mattering during the welcome experience. The willingness and flexibility of colleges and universities worldwide to pivot quickly to support first-year students’ overall transition experience and provide opportunities to safely meet and connect with other new students in both structured and organic formats is a testament to the innovative spirit of higher education professionals. Authors’ Note The authors want to acknowledge and thank the dedicated staff at our institution. This year has been unlike any other, and our staff have risen to the challenge. The transition team worked tirelessly to support our students and provide them the best possible experience. We are grateful for their willingness to collaborate on this assessment journey. You are amazing, and our institution is lucky to have you. THE JOURNAL OF COLLEGE ORIENTATION, TRANSITION, AND RETENTION17 References Astin, A. W. (1999) Student involvement: A developmental theory for higher education. Journal of College Student Development, 40(5), 518-529. Council for the Advancement of Standards in Higher Education. (2015). CAS self-assessment guide for orientation programs. Goodman, J., Schlossberg, N. K., & Anderson, M. L. (2006). Counseling adults in transition: Linking practice with theory (3rd ed.). ProQuest Ebook Central https://ebookcentral.proquest.com Hoffman, M., Richmond, J., Morrow, J., & Salomone, K. (2003). Investigating “sense of belonging” in first-year college students. Journal of College Student Retention: Research, Theory & Practice, 4(3), 227-258. Kuh, G. D., Schuh, J. H., Whitt, E. J., & Associates. (1991). Involving colleges: Successful approaches to fostering student learning and development outside the classroom. Jossey-Bass. Mack, C. E. (2010). A brief overview of the orientation, transition, and retention field. In J. A. Ward-Roof (Ed.), Designing successful transitions: A guide for orienting students to college (Monograph No. 13, 3rd ed., pp. 3-10). University of South Carolina, National Resource Center for The First-Year Experience and Students in Transition. Maslow, A. H. (1962). Toward a psychology of being. von Nostrand Reinhold. Noel, L., Levitz, R., Saluri, D., & Associates. (1986). Increasing student retention. Jossey-Bass Packwood, W. T. (1977). College student personnel services. Charles C. Thomas. Rentz, A. L., & Saddlemire, G. L. (1988). Student affairs functions in higher education. Charles C. Thomas. Schlossberg, N. K. (1989). Marginality and mattering: Key issues in building community. In D. C. Roberts (Ed.), Designing campus activities to foster a sense of community (New Directions for Student Services, No. 48, pp. 5-15). Jossey-Bass Tinto, V. (1987). Leaving college: Rethinking the causes and cures of student attrition. University of Chicago Press. Strayhorn, T. L. (2012). College students’ sense of belonging: a key to educational success for all students. Routledge. Young, D. G. & Keup, J. (2019). CAS cross-functional framework for first-year experience. P. J. Carretta (Ed.). Council for the Advance