NODA Journal 19_1.indd 124 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION Assessing the Advising Experience: Using Homegrown Assessment Tools to Improve Advising Practices Rachael Switalski and Jeffrey Birou Introduction and Problem Statement In 2008, a new associate dean was appointed in the student services office at Drexel University. In the same year, the director was also planning to leave. Aware of the impending change, the five advisors in the office sought other opportunities to advance their careers. This led to a 100% central staff turnover within one academic year. We (the authors) both began working in the student services office as academic advisors in Spring 2008 while other advising positions were also being re-staffed. We were part of the “cross over” advising staff who decided to continue current office policies and procedures for the fall term because the existing structure had worked in the past. However, we were soon to learn that the way things “had always been done” was not necessarily the best way. Students arriving at the beginning of the fall term seemed to be in a state of perpetual panic. They repeatedly asked the same questions, didn’t understand how their curriculum was structured, weren’t aware whom to contact for help, and didn’t know who their advisor was or how their advisor might help. This situation left us feeling harried and burnt out. Based on this anecdotal information, we suspected that this sense of disconnect stemmed from inadequacies in our initial outreach to the incoming class; therefore, we decided to build a unique assessment in an attempt to identify weaknesses and understand the students’ perceptions of their advising experiences. One advantage to having 100% advising staff turnover is the ability to look at all activities with fresh eyes and provide new suggestions for programmatic changes. Thus, we used this opportunity to examine the standing practices and reformulate them as needed. That fall, the office secured funding for a new team member to attend the National Resource Center for the First-Year Experience’s National Conference on CAMPUS NOTES Rachael Switalski (rswitalski@coe.drexel.edu) is the Associate Director in the Undergraduate Advising Center at Drexel University College of Engineering. Jeffrey Birou (jbirou@coe.drexel.edu) is an Academic Advisor in the Undergraduate Advising Center at Drexel University College of Engineering. First-Year Assessment. Applying the methods learned at the conference was key in helping use assessment for our internal purposes. We structured the first assessment to determine: • whether students knew who their advisors were, • if students were satisfied with their advising experiences, • if students felt that they received accurate information in advising sessions, • whether students utilized walk-in hours or e-mail to obtain advising services, • if students were notified of deadlines, and • whether students felt that their advisor had been available when needed. We intended to use the data gleaned from the survey to refine interactions with incoming freshmen. As it turned out, we needed an overhaul. Why Assessment? Conducting an assessment can often feel laborious, time-consuming, and irrelevant to day-to-day operations. An institution may participate in a multitude of surveys for freshmen, ranging from homegrown student self-assessment efforts to nationally recognized tools such as the Cooperative Institutional Research Program (CIRP) or the National Survey of Student Engagement (NSSE), which are taken annually by hundreds of thousands of freshmen entering into two- and four-year institutions. While student self-assessment surveys (SAS)—CIRP, NSSE, and others—provide insight into matters relevant to various higher education functions, this article discusses a supplement to those efforts, what we refer to as a “short n’ sweet” assessment of one specific office’s services. These assessments can validate current practices, provide individualized feedback to professional staff, and identify areas for improvement. There are several established academic advising assessments, such as the ACT Survey of Academic Advising and the Developmental Advising Inventory, as detailed by Cuseo (n.d., pp. 1–5). These are valuable resources and can provide useful information about advising methods, including longitudinal data that can be compared with other institutions. However, we were interested in the unit- specific concerns and goals built into our assessment model, which an internally developed assessment could uniquely address (Campbell, Robbins, & Nutt, 2004; Demetriou, 2005). As stated by Light (2004, in Demetriou, 2005), the interactions an office has with a student comprise part of their collegiate experience; effective assessment can help identify ways “to improve [that] overall student experience.” It is important to keep assessment on a small scale, both in terms of the number of questions asked and the type of information collected. Too many questions carry a risk of alienating potential respondents: “the longer an instrument is (i.e., the more reading time it requires), the less time students devote to writing and, consequently, fewer useful comments are provided” (Cuseo, n.d., p. 6) When the assessment asks broad questions in an attempt to address the operations of an entire department or college, the data will also turn out to be FALL 2011 • VOLUME 19, NUMBER 1 125 broad and general in a way where specific follow-up actions cannot be identified. “The point,” according to Gardner et al. (2001, p. 7), “is to keep the process manageable… instead concentrate on selected components you deem most in need of your immediate attention.” Asking too many questions also can result in the audience disengaging before completing the assessment. Instead, consider “what data points are needed in order to conduct a thorough problem analysis, and where [those data are] available” (Gardner, 2001, p. 8). In other words, be economical and intentionally write questions that will provide hard, actionable data. An example of a wasted question, initially included in our Fall 2009 assessment, was: “Are you a fan of our Facebook or Twitter pages?” While the utilization of these social media resources is something about which we were curious, we could have just as easily looked at the number of members who joined those pages online, information that is freely available on both sites. Initial Assessment Our questions were drafted to investigate our weak connection with freshmen and to determine specifically the areas for improvement. We asked 12 questions, including “Do you know who your advisor is?” and “How many times have you been in e-mail contact with your advisor during the fall term?” Three of the questions were a variation of the latter, asking about walk-in advising and appointments. Our greatest concern was advisor recognition; we did not know if we had done a good job of making students aware of our services. We sent out the assessment via e-mail to freshmen about three-quarters through their first term, after registration for the following term. We believed that at this point in the term students would know who their advisor was. We left the assessment open for three weeks and reminded students of the assessment via e-mail and our Facebook page. There were 218 respondents from a 750-person target population, a 28% yield. It is important to consider such statistical factors as standard deviations and confidence intervals (also referred to as “margins of error”), and to know beforehand whether or not these are of concern for a particular assessment. Higher response rates yield smaller margins of error, making data more statistically accurate. For example, the data we collected from this initial assessment provided a 5.02% margin of error, which is fairly high, but for this assessment we were looking for general trends, which our data provided. Table 1 shows a sampling of our key assessment points. 126 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION We had not set specific goals for this study, but were using this initial assessment to help inform where realistic goal posts should be. For instance, if we were only seeing 20% of our target population coming for walk-in advising, it would not be a reasonable goal to increase that number to 80% in the following year. That being said, we were not satisfied with these results. The fact that fewer than 40% of our respondents indicated that they came in for walk-in sessions or had an appointment was disappointing; we were missing a significant portion of our student body. Particularly telling was how often students claimed they were in e-mail contact with their advisor: 28% claimed they had never exchanged e-mails with their advisor. Moreover, the results regarding advisor recognition were worrisome; 17% of the students could not identify their advisor. Planning for a New Outreach Model Given these findings, we determined that an overhaul to our outreach plan was necessary; we had suspected this for some time but now had data to reinforce it. In order to construct a new, multipronged effort, we needed to begin planning early. Our next fall term began in late September 2009, so our advising staff spent several staff meetings in early 2009 analyzing the data from our assessment and FALL 2011 • VOLUME 19, NUMBER 1 127 TABLE 1 Key Assessment Points, Results from 2008 Question Answer 2008 Do you know who your Yes 83% academic advisor is? No 17% How many times did you 0 62% utilize the academic 1-3 36% advising walk-in hours 4+ 2% during the fall term? How many times have you 0 63% made an appointment to 1-3 36% meet with your academic 4+ 1% advisor during the fall term? How many times have you 0 28% been in e-mail contact with 1-3 52% your advisor during the fall 4+ 20% term? discussing what sort of changes we could make and how we could implement them in a manner that provided a more developmental experience for students. In these weekly strategic planning meetings, we created goals for our office that were both measurable and qualitative. For example, one aim for Fall 2009 was that 95% of first-term freshmen would know who their advisor was by the time the fall assessment was given in November 2009. An outreach plan was developed to reach students from as many directions as possible in order to communicate the message: “We are your advisors, and we are here to help!” Each piece was designed with the aim that students would receive specific and helpful information from their advisors, so any “blast” e-mails were sent to the students from a general college e-mail address. Instead of one lengthy e-mail sent the week before classes to cover all relevant information, we broke up the information into smaller batches, what we sometimes refer to as the “Web 2.0” or “Twitter” approach to content delivery. We were cognizant of how the millennial generation interacts with content on the Internet—in short bursts of information— and conformed our strategy accordingly in hopes that students would be more likely to read our communications. Overall, the main goals of the outreach, designed to address the most problematic results of the survey, were: • to have students be aware of their advisor, • to have students want to go to their advisor for help, and • to ensure that students know how to get in contact with their advisor. In order to stagger these areas of communications, we needed to back up our outreach plan to begin much earlier than it did in the previous year. Implementation of Outreach Plan Listed below is the outreach plan and a brief explanation of each activity. June 2009 Personalized letter via United States Postal Service from advisor A letter introducing the office and signed by the advisor was mailed to each student in mid-June. This letter was sent via “snail mail” with the intent that the parents would also see it and encourage their students to reach out to the advisors with questions. The letters arrived before the first orientation session so that students would already know who their advisor was during orientation and would be able to introduce themselves. July 2009 Orientation The academic session in the university orientation was restructured so that a significant piece was aimed at letting students and parents know that advisors are here to help. 128 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION August through September 2009 Weekly newsletters via e-mail and Facebook We divided the pre-fall term outreach efforts into weekly e-newsletters, e-mailed every Friday after the final orientation session through the beginning of classes. The content of each e-newsletter was relevant to the week it was delivered. For example, the first e-newsletter, distributed by e-mail, covered information about advanced placement scores and evaluations; the e-newsletter sent the week before the term started was about walk-in advising availability during the course add/drop period. Because these digital newsletters were developed internally, it was easy to address frequently asked questions that advisors were encountering, such as how to buy books or how the dining hall works. These newsletters and important pieces of information were posted to Facebook and Twitter. Freshman Welcome Party with t-shirt giveaway After classes began, the advisors hosted a party to welcome freshmen to the campus. Each freshman was able to pick up a t-shirt designed by one of their classmates as part of a college-specific design contest. These t-shirts were only given to incoming freshmen to foster a sense of identity within the freshmen class. October 2009 Advisors visit each UNIV 101 class As a group, the academic advisors visited each UNIV 101 (Freshman Experience) course to introduce themselves to students, answer questions, and make sure students were familiar with the office location and walk-in hours. Advisors also handed out magnets with contact information, referral cards for tutoring resources on campus, and candy to generate goodwill. November 2009 Advising Assessment Survey sent to students An assessment survey was sent to the 2009 freshman class to determine if the outreach plan had been successful. Anecdotally, it seemed clear to the two advisors from the previous summer that the relationship with this incoming class was different. Students were sending multiple e-mails a day to the advising office, seemed to have the information about advising that they needed for the fall term, and continued coming in to ask questions about minors, schedule adjustment, joining clubs, and other types of planning. It seemed to the advisors that the workload had shifted from being heavy at the beginning of fall term to being distributed into August and September. This was an indication that students were reading the newsletters (although specific “open rate” data were not collected until 2010 when we switched to a new software program that included this feature), and they were asking questions about the information as they received it rather than trying to sift through mountains of information when they arrived on campus. The following assessment results show that the outreach plan made a large difference in how this freshman class perceived FALL 2011 • VOLUME 19, NUMBER 1 129 their advisors. Assessment Results, 2009 After the new outreach plan had been in place for one cycle (from before new student orientation through course registration for the second term), a revised assessment instrument was sent to the freshmen. The results in Table 2 compare the 2008 and 2009 findings. 130 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION TABLE 2 Assessment Results, 2008/2009 Question Answer 2008 2009 I have received Disagree/Strongly 22% 3% accurate information Disagree about courses, programs, and Agree/Strongly Agree 78% 97% requirements through academic advising. Sufficient prior notice Disagree/Strongly 17% 1.8% has been provided Disagree about deadlines related to policies and Agree/Strongly Agree 83% 98.2% procedures.* Academic advising Disagree/Strongly 14% 1.8% has been available Disagree when I needed it.** Agree/Strongly Agree 86% 98.2% *Question wording was revised slightly in 2009 to read: “Please respond to the following: I have been given sufficient prior notice about deadlines related to Drexel policies and procedures.” **Question wording was revised slightly in 2009 to read: “Please respond to the following: my academic advisor has been available when I needed her/him.” The responses indicated in Table 2 point to improved student satisfaction between 2008, our baseline year, and 2009. We credit our electronic outreach efforts for these improved student perceptions. The e-newsletters distributed in a more digestible form helped to establish a student-advisor relationship, which continued into the first term. Additionally, the number of students who e-mailed the advising office at least four times grew nearly three-fold. Students also self-identified as using the walk- in hours more, and the number of students who claimed that they never came in for a walk-in advising session dipped to 14.5%. Most importantly, 98% of first- year students were able to identify their advisor by the time the assessment was delivered (See Table 3). One area of concern was that the number of students who identified as having scheduled an appointment with their advisor did not increase to our satisfaction. Efforts will continue to address this issue. “Lather, Rinse, Repeat” Since the second annual assessment suggested that the new outreach methods were working as intended, our office felt confident in utilizing the same approach with the 2010 freshmen. This is where we entered what we affectionately term our “Lather, Rinse, Repeat” stage. Some minor changes included changing the web service used for our e-newsletters; these adjustments were made to refine the process based on student feedback, not replace it. After reviewing the results from the third annual freshman survey in November 2010, we were surprised that the proportion of respondents to total population was ~22%. Table 3 shows the results, along with the 2008 and 2009 results, for the sake of comparison: FALL 2011 • VOLUME 19, NUMBER 1 131 TABLE 3 Key Assessment Points, Results from 2008 through 2010 Question Answer 2008 2009 2010 Do you know who Yes 83% 98% 99.5% your advisor is?* How many times did 0 62% 46.5% 58.3% you utilize the academic 1-3 36% 52.6% 39% advising walk-in hours 4+ 2% 0.9% 2.7% during the fall term?** How many times have you 0 63% 61.6% 64.2% made an appointment to see 1-3 36% 37.4% 33.2% your advisor in the fall term?*** 4+ 1% 1% 2.7% How many times have you been 0 28% 6.1% 5.4% in e-mail contact with your 1-3 52% 35.3% 36% advisor during the fall term?**** 4+ 20% 58.6% 58.6% *Question was changed in 2009 and 2010 to “Who is your academic advisor?” with a multiple choice answer. **Question as changed in 2009 and 2010 to “How many times have you come in for a walk-in meeting with an advisor?” ***Question was changed in 2009 and 2010 to “How many times have you made an appointment to see your advisor?” ****Question was changed in 2009 and 2010 to “How many times have you been in e-mail contact with your advisor?” We were surprised to find that our face-to-face interactions decreased—fewer students indicated meeting with advisors either with an appointment or walk-in advising. The potential significance of this is addressed in the conclusion. Advisor Feedback: An Unexpected High Point As students supplied feedback about specific advisors, they were also able to use the comment box to write notes of appreciation or criticisms of their advising interactions. Comments included: • He responds to e-mails promptly and is easy to approach. Very understanding and helps set up a plan towards success. • She helped me with my academic probation; with her help, I was able to bring my GPA up to a 2.0. • She does an outstanding job. She makes you feel like you’re the only student she is advising. Each advisor is given his or her individual feedback. Having this feedback and knowing that students appreciate their advisors can create a bright spot in an otherwise stressful day or week. Conclusion After two assessment cycles following the launch of the outreach plan, data indicate that the outreach efforts improved advisor recognition, increased knowledge of advising resources and important information, and raised satisfaction with advising availability and services. Interestingly, one factor that has not been positively affected to a significant degree is face-to-face interactions between advisors and students. Both appointments and walk-in sessions for advising remain fairly static from year to year. It is possible that the current generation of students is satisfied with e-mail interactions and does not feel the need for face-to-face contact unless significant problems arise; we are planning to survey this level of satisfaction in the upcoming assessment cycle. The office will continue to brainstorm ways to encourage students to come in for advising by offering incentives and trying various programming to put advisors in front of students. As summarized by Nutt (2003), research done by Astin, Tinto, and Habley indicates that students are more likely to seek people with whom they are familiar when a problem arises. Tinto (1987, in Nutt, 2003) argued that “academic advising is the very core of successful institutional efforts to educate and retain students”. Until there is research that shows that digital or online connections have the same result, the office will continue to strive to increase the number of personal interactions between students and advisors. 132 THE JOURNAL OF COLLEGE ORIENTATION AND TRANSITION References Astin, A. W. (1977). What matters most in college: Four critical years. San Francisco: Jossey-Bass. Astin, A. W. (1993). What matters most in college: Four critical years revisited. San Francisco: Jossey-Bass. Campbell, S., Robbins, R., & Nutt, C. (2004). Guide to assessment in academic advising for assessment of academic advising seminar. Kansas State University: National Academic Advising Association. Cuseo, J. (n.d.). Assessment of academic advisors and academic advising programs. Informally published manuscript, Department of Psychology, Marymount College, Palos Verdes, CA. Retrieved from http://www.uwc.edu/administration/ academic-affairs/esfy/cuseo/Assessment of Academic Advisors and Academic Advising Programs.doc Demetriou, C. (2005). Beyond student satisfaction surveys: Designing a student focus group to assess academic advising. Pennsylvania State University. The Mentor: An Academic Advising Journal, 7(1), Retrieved from http://dus.psu.edu/mentor/050211cd.htm Gardner, J. N., Barefoot, B. O., & Swing, R. L. (Eds.). (2001). Guidelines for evaluating the first-year experience at four-year colleges. Columbia, SC: The National Resource Center for the First-Year Experience and Students in Transition. Habley, W. R. (1994). Key concepts in academic advising. In summer institute on academic advising session guide (p. 10). Available from the National Academic Advising Association, Kansas State University, Manhattan, KS. Light, R. (2004) Changing advising through assessment. NACADA Journal, 24 (1 & 2), 7–16. Nutt, C. L. (2003). Academic advising and student retention and persistence. Retrieved August 2, 2011, from the NACADA Clearinghouse of Academic Advising Resources: http://www.nacada.ksu.edu/clearinghouse/advisingissues/retention.htm Tinto, V. (1987). Increasing student retention. San Francisco: Jossey-Bass. FALL 2011 • VOLUME 19, NUMBER 1 133