ANALYTIC TEACHING AND PHILOSOPHICAL PRAXIS Vol. 30 No.1 31 Improving Academic Quality through Outcomes Assessment and Active Learning Strategies – a Model for Effective Institutional Change Theresa R. Moore & Mary C. Hassinger ABSTRACT: Whether responding to recommendations by accrediting bodies or to demands for higher academic distinc- tion by various stakeholders, becoming more student-centered often involves large scale, institutional change. Viterbo Uni- versity was recommended by the Higher Learning Commission to improve its “culture of assessment.” As a response to this recommendation, Viterbo University launched a project to improve outcomes based assessment and active learning, funded by a $1.8 million Title III federal grant. The focus of this article is to describe the results for this five year project as related to the two overarching objectives of the grant: 1) to improve academic assessment and 2) to improve the use of active learning strategies. Data indicates not only a strong participation rate on behalf of faculty in the faculty development activities, but also high satisfaction and long term commitment to active learning and assessment. Furthermore, faculty perceptions coupled with objective measures are indicative of improved student learning. The article aims to inform readers of ways that faculty development efforts can improve academic quality and also be sustainable. Theoretical framework Colleges and universities are increasingly being challenged to become more “learner-centered.” What does this mean for faculty and more importantly, what does this mean for students? Many universities are in- vesting large amounts of resources on issues related to improving teaching and learning pedagogies such that there is now much more diversity of teaching than straight lecture only and the “teach as I was taught” model. Best practices support a wide variety of teaching pedagogies that are correlated with improved student learning, however, the onus of institutional change remains a daunting one that faculty may find threatening to their aca- demic freedom in the classroom. Faculty attitudinal considerations are just one important factor for successful institutional change. Institutional change cannot be discussed without unearthing the fact that this theoretical framework as- sumes that the “covering of content” or dissemination of information is not adequate to what higher education institutions should be teaching students. In fact, the goals of a liberal arts institution are much more existential and hearken back to directives articulated in the American Constitution regarding civic virtues such as “respect for the individual and commitment to opportunity; respect for the views of others; the belief that individual right and privileges are to be exercised responsibly (as cited in the Wingspread Group on Higher Education, 1992). Furthermore, “the liberal [arts] education teaches us to think critically and to question our intellectual, social and political premises” (Katz, S. as cited in the Wingspread Group on Higher Education, 1992). In es- sence, in order to become citizens that can fully prescribe to Constitution law requires a fundamentally different type of education, one that requires students to learn how to learn and how to think. Student –centered learning is based on these concepts, and evidence of student-centered learning is based on various assessment methods widely practiced in higher education. Best practices in outcomes-based assessment (Banta et. al., 1996, Estabrooks et al., 2002, Huba and Freed, 2000) assert that institutional change occurs most effectively when faculty have direct engagement in the proc- ess. Travis (1995) corroborates that change directed from top management is not nearly as effective as change ANALYTIC TEACHING AND PHILOSOPHICAL PRAXIS Vol. 30 No.1 32 cultivated among the ranks of those who are expected to adopt the innovations. However an individual institu- tion decides to radically shift the pedagogical culture, faculty still bear the major responsibility for this change to happen. (Davis, 1993) Further ingredients for successful institutional change continue to highlight considerations of the faculty. The late Donald Farmer, Academic Vice President for Academic Affairs at King’s College in Pennsylvania, was able to institutionalize competency growth plans and a coherent curriculum within each field of study to improve student learning based on the following premises for success. His approaches have been modeled by many insti- tutions seeking the same improvements: Never to ask faculty to implement a change which faculty believe they are unprepared professionally to • implement; Faculty self-confidence is essential to gaining…commitment;• Directed and focused faculty is a significant ingredient in preparing faculty to successfully implement • academic change; and Permanent and significant changes usually require the acquisition of new resources. (Farmer, 1998)• Whatever the initial impetus is to become “student -centered,” faculty are being called to take on a major pedagogical shift - to stop viewing teaching as only “covering the content” and to start viewing it as emphasizing “helping the students learn” (Svinicki, 1990). Faculty development efforts are being used in new and exciting ways to enable faculty to consider adaptations to their teaching through an array of initiatives such as instruc- tional grants, workshops and discussion groups, classroom observations, and microteaching opportunities (We- imer, 1990). Not only is faculty buy-in and commitment to these initiatives important, the commitment of resources on behalf of the institution is integral to the change process. For example, a resource library on best practices in assessment and active learning, hardware technology and support, and consultative personnel are important in augmenting any existing resources that the institution may already have. The cross section of resources is intended to offer faculty significant and “cutting edge” training, and hence support and confidence regarding their teaching (Menges, Weimer, and Associates, 1996). Among pedagogical resources available for faculty to become more learner-centered is an array of innovative improvement strategies and classroom methods, such as cooperative learning, case method, test feedback, and videotaping. Some of these strategies have a formal struc- ture, an extensive research base, and applicability to almost any discipline. Such strategies have been described as “teaching improvement models” (Svinicki, 1990). It is integral that the institution provide accessible personnel to assist faculty in sustained training in these areas. To date, most research has focused on faculty development programs’ participation rate of faculty, as op- posed to actual measurable relationships between the programs and improved student learning (Sargent, 1999, as cited in Hickson, et. al, 2008). In contrast, Columbus State University undertook a required portion of their accreditation process known as the Quality Enhancement Project (QEP). Through a faculty needs assessment, the university decided to focus their faculty development efforts on one major initiative: improving student writ- ing. Although the goals of the quality enhancement program at Viterbo University were different than the goal at Columbus State University, their successes were driven by similar premises: 1) the initiatives in the plan were clearly linked to the university’s strategic plan of academic excellence and improved student learning and 2) a structure of accountability, recognition, and rewards reinforced that the university was committed to the activi- ties. (Hickson, et. al, 2008). The literature on facilitating institutional change to improve student learning has its underpinnings in sev- eral common facets: the importance of faculty buy-in and commitment, the need for financial resources and institutional commitment, and the provision of the necessary administrative support and expertise to facilitate the various faculty development efforts, with long term sustainability always at the forefront of the endeavor. ANALYTIC TEACHING AND PHILOSOPHICAL PRAXIS Vol. 30 No.1 33 Methodology With best practices of teaching and learning in mind, and the financial resources in hand, the Title III team of administrators at Viterbo University, which included a Director, an Instructional Designer, an Instructional Technology Specialist, and several faculty “peer coaches,” undertook five years of rigorous faculty development activities in the areas of assessment and active learning, guided by a clearly articulated timeline for success and facilitated by a team of teaching and learning experts (see appendix A). By the end of the grant cycle (September 2008), the Title III team of administration were able to analyze several forms of data that were gathered over the years, with positive results. Data indicates not only a high par- ticipation rate in the faculty development activities, but also high faculty satisfaction, commitment to assessment and active learning, and both faculty perceptions and survey data indicative of improved student learning. Results Assessment Over 100% of full time faculty were trained in learning outcomes and assessment and 92% of faculty submit- ted work reports documenting refined, piloted learning outcomes and assessment strategies in courses. Regard- ing the 49 established academic programs (both undergraduate and graduate) as of November 2008: 100% have established a plan for program-level assessment;• 92% have data on student learning and are in the process of analyzing the data;• 73% have articulated action taken to improve student learning; and• 18% have indicated follow-up measures.• Active learning 107% of faculty ( taking into consideration new faculty hires throughout the five year training period) were trained in active learning strategies, and 105% of faculty (#) submitted active learning pilot plans throughout the duration of the grant. Preliminary surveys suggest that 82% of faculty (#) have extended the active learn- ing strategies beyond those required for the work reports in the grant. This survey also documents that 79% of faculty (#) reported a high degree of commitment to active learning. As of August 2008, 50% of academic programs submitted documentation reporting a strong commitment at the program level to active learning by their faculty. Long term commitment Title III objectives Initial survey results indicate that 74% of faculty report a strong commitment to assessment (rank of 4 or 5 on a scale of 1- 5with five being the highest level of commitment). The same survey reports 79% of faculty indicate a strong commitment to active learning (rank of 4 or 5 on a scale of 1- 5 with five being the highest level of commitment). Impact on student learning Assessment reports archived in Trac Dat ©, an assessment management repository marketed by Nuventive © and adopted by the university via the Title III grant, coupled with faculty self-reports, document increased student learning in programs and courses. Fifty percent of undergraduate program assessment coordinators completed a final grant work report which asked them to ascertain and document the impact of the Title III faculty development efforts on student learn- ing. These assessment coordinators, in a variety of ways, articulate an improved assessment culture and increased ANALYTIC TEACHING AND PHILOSOPHICAL PRAXIS Vol. 30 No.1 34 use of active learning. In the same document, assessment coordinators correlated these improvements with overall improvements in student learning. These assertions are corroborated by two other methods of inquiry: 1) objective indicators offered by NSSE data and 2) student perception data. The quandary remains as to how measure actual “improvement” of student learning. Students may articulate perceptions of increased learning, but the epistemological questions remain: Did student learning really improve and how do we know? Student Data The authors discovered a correlation between student data gleaned from the NSSE (National Survey of Stu- dent Engagement) and improved student learning during the timeframe of the Title III. While a claim for causa- tion cannot be made, the parallels discovered via item analysis of the NSSE provide an interesting counterpart to the Title III data. The NSSE annually obtains information from hundreds of four-year colleges and universities nationwide about student participation in programs and activities that these institutions provide for student learning and personal development. Survey items on NSSE represent empirically confirmed “good practices” in undergradu- ate education. That is, they reflect behaviors by students and institutions that are associated with a variety of desired outcomes. (http://nsse.iub.edu/index.cfm) There are three domains in the NSSE survey in particular related to student learning that are relevant to the endeavors of the Title III project: level of academic challenge (LAC), active and collaborative learning (ACL), and student-faculty interaction (SFI). The NSSE instrument was administered to Viterbo University freshmen and seniors during the academic years 2005-2006 and 2006-2007, timing that coincided with Title III faculty development efforts. In Fall 2005, Viterbo’s incoming freshmen took the Beginning College Survey of Student Engagement (BCSSE). In Spring 2006, these same students participated with seniors in taking the NSSE. Viterbo’s overall participation rate in the NSSE was significantly higher compared to the national aver- age. The freshmen participation rate was 52% compared with 33% from all NSSE participants. Fifty-six per- cent of Viterbo seniors participated, compared with the 36% rate from all NSSE participants. Level of academic challenge (LAC) is defined as challenging intellectual and creative work central to student learning and collegiate quality. Colleges and universities promote high levels of student achievement by empha- sizing the importance of academic effort and setting high expectations for student performance. (http://nsse. iub.edu/index.cfm) Items in this category asked students questions related to: Preparing for class (studying, reading, writing, rehearsing, etc. related to academic program).• Number of assigned textbooks, books, or book-length packs of course readings.• Number of written papers or reports of 20 pages or more; number of written papers or reports of be-• tween 5 and 19 pages; and number of written papers or reports of fewer than 5 pages. Coursework emphasizing analysis of the basic elements of an idea, experience or theory.• Coursework emphasizing synthesis and organizing of ideas, information, or experiences into new, more • complex interpretations and relationships. Coursework emphasizing the making of judgments about the value of information, arguments, or meth-• ods. Coursework emphasizing application of theories or concepts to practical problems or in new situations.• Working harder than you thought you could to meet an instructor’s standards or expectations.• Campus environment emphasizing time studying and/or academic work.• ANALYTIC TEACHING AND PHILOSOPHICAL PRAXIS Vol. 30 No.1 35 Active and collaborative learning (ACL) is based on the premise that students learn more when they are in- tensely involved in their education and asked to think about what they are learning in different settings. Collabo- rating with others in solving problems or mastering difficult material prepares students for the messy, unscripted problems they will encounter daily during and after college. (http://nsse.iub.edu/index.cfm) Items in this category asked students if and how often they: Asked questions in class or contributed to class discussions. • Made a class presentation.• Worked with other students on projects during class.• Worked with classmates outside of class to prepare class assignments.• Tutored or taught other students.• Participated in a community-based project as part of a regular course.• Discussed ideas from your readings or classes with others outside of class (students, family members, • co-workers, etc.). Student-faculty interaction (SFI) is defined as opportunities for Students to learn firsthand how experts think about and solve practical problems by interacting with faculty members inside and outside the classroom. As a result, their teachers become role models, mentors, and guides for continuous, life-long learning. (http:// nsse.iub.edu/index.cfm) Items in this category asked students if and how often they: Discussed grades or assignments with an instructor.• Talked about career plans with a faculty member or advisor.• Discussed ideas from your readings or classes with faculty members outside of class.• Worked with faculty members on activities other than coursework (committees, orientation, student-life • activities, etc.). Received prompt written or oral feedback from faculty on your academic performance.• Worked with a faculty member on a research project outside of course or program requirements.• Results revealed the following regarding Viterbo University students who partook in the NSSE: Level of academic challenge (LAC), active and collaborative learning (ACL), and student-faculty interac-• tion (SFI) all improved between 2006 and 2007 for Viterbo University’s scores relative to the NSSE Top 50% mean score 2007 LAC at the senior level exceeds the 2007 NSSE Top 50% mean score by 1.4%. • 2007 ACL at the senior level exceeds the 2007 NSSE Top 50% mean score by 8.4%.• Item analysis within these three domains indicates that Viterbo students reported significantly higher rat- ings in comparison to other Catholic universities and colleges, benchmark schools, and the NSSE average. The preceding NSSE data is one indicator that Viterbo University students are reporting an improvement in certain domains related to classroom experiences and Title III objectives. Student focus group data – pilot The breadth of literature on faculty becoming student-centered is being paralleled with literature addressing the student side of the equation – helping students to learn in learner-centered environments. An important consideration for future research, the following results represent a sample of student perspectives on the peda- gogical shifts that have taken place since the inception of grant training in 2004. A cross-section of students was selected from various majors and were asked a series of open-ended questions related to student learning outcomes and assessment. The researcher’s goal was to ascertain their general aware- ness of these topics and how they experience them in their classrooms and, in particular, within their major ANALYTIC TEACHING AND PHILOSOPHICAL PRAXIS Vol. 30 No.1 36 programs of study. Eleven upper division students were interviewed (5 male; 6 female) from various disciplines on campus. Via content analysis, interview data revealed that students were familiar with outcomes-based assessment verbiage and were aware of objectives in line with Title III faculty training activities. In a focus group format, students were asked a series of ten open ended questions. Content analysis of the data revealed that the students in the focus group were able to define and describe a general awareness of student learning outcomes and assessment methods of these outcomes within their majors – especially as they related to courses. A salient level of ambiguity existed when students were asked to define/describe learning outcomes and assessment methods at the program (major) level. Not all were able to make the connection between course outcomes and program outcomes and therefore, the concept of competency growth plans within majors eluded them. Although all students interviewed were able to give examples of active learning methods used in class- rooms, disparity was also found among the rigor and types of activities. The general findings from the pilot interview reveal that students are aware of learning outcomes, assess- ment, and active learning approaches within classes in their majors. They emphasized the efficacy of active lean- ing in the classroom and made a significant suggestion as a result of the focus group discussion - they advised that general education courses make a better effort at aligning learning outcomes, activities, and assessment methods and making them more transparent to students. Discussion The results from the Title III project at Viterbo University suggest high satisfaction with the training and long term commitment to active learning and assessment beyond the tenure of the grant. Furthermore, faculty and student perceptions coupled with objective measures are indicative of improved student learning. Future research calls academicians to take not only the faculty into consideration when making huge peda- gogical shifts, but also the learners. (Doyle, 2008) Students often come to college with certain assumptions about roles and ways of learning that may not only contrast with more learner-centered models, but in fact sabotage them. This leads to the importance of student voice and transparency as to why the classroom dynamics are changing from the traditional lecture-based format. The expectations for both faculty and students’ roles need to be clear in order to assuage confusion and/or resistance. The discussion of actual “improved student learning” versus perceived “improvement” is something that needs careful consideration as student learning continues. Triangulation of faculty course data, student perceptions, and perceptions from Assessment Coordinators, lead the authors to believe that there has inarguably been a cultural shift at Viterbo, but the tangible and measurable results of student learning still await full explortion and dissemination. References Banta, T. W., Lund, J. P., Black, K. E. & Oblander, F. W (eds). (1996), Assessment in practice: putting principles to work on college campuses. San Francisco, CA: Jossey-Bass. Davis, B. (1993). Tools for Teaching. Jossey-Bass Publishers: San Francisco, CA. Doyle, C. (2008). Dancing toward transformation – mapping a pedagogical site. Narrating Transformative Learning in Education. Palgrave Macmillan. Estabrooks, C. et al. (2002). Ingredients for change: revisiting a conceptual framework. Quality and Safety in Health- care. January 2002; 11(2). Hickson, J. et al. (2008). A Quality enhancement model: connecting faculty development with the achievement of student learning outcomes and institutional goals. The Journal of Faculty Development .January 2008; 22(1):5-10 Huba, M. and Freed, J. (2000). Learner-Centered Assessment on College Campuses: Shifting the Focus from ANALYTIC TEACHING AND PHILOSOPHICAL PRAXIS Vol. 30 No.1 37 Teaching to Learning. Allyn & Bacon Publishers. Menges, R.,Weimer, M. and Associates. (1996). Teaching on solid ground using scholarship to improve practice. Jossey-Bass Publishers: San Francisco, CA: Svinicki, M., (Ed.). (1990). The changing face of college teaching. New Directions for Teaching and Learning. No. 42. Jossey-Bass Publishers: San Francisco, CA. Travis, J. (1995). Models for improving college teaching: a faculty resource. The National Teaching & Learning Forum. Publishers: James Rhem & Associates, LLC. Wingspread Group on Higher Education.1993.An American Imperative: Higher Expectations for Higher Education. Address coresspondence to: Theresa R. Moore Director of Faculty Development and Internship Coordinator Viterbo University, 900 Viterbo Drive La Crosse WI, 54601 trmoore@viterbo.edu or Address correspondence to: Mary C. Hassinger Academic Vice President Stephen’s College email: mhassinger@stephens.edu phone: (573) 876-7213 x 4213 ANALYTIC TEACHING AND PHILOSOPHICAL PRAXIS Vol. 30 No.1 38 Appendix A