Global Research in Higher Education ISSN 2576-196X (Print) ISSN 2576-1951 (Online) Vol. 6, No. 1, 2023 www.scholink.org/ojs/index.php/grhe 1 Original Paper Fostering Scholarly Approaches to Peer Review of Teaching in a Research-Intensive University: Strategic Development of a Departmental SPRoT Protocol Andrea S. Webb 1* , Harry T. Hubball 1 & Anthony Clarke 1 1 Faculty of Education, University of British Columbia, Vancouver, Canada * Andrea S. Webb, Faculty of Education, University of British Columbia, Vancouver, Canada Received: November 23, 2022 Accepted: December 5, 2022 Online Published: January 11, 2023 doi:10.22158/grhe.v6n1p1 URL: http://dx.doi.org/10.22158/grhe.v6n1p1 Abstract This article draws on a 10-year institutional initiative and examines whether and how a strategic departmental Summative Peer Review of Teaching (SPRoT) Protocol was implemented at a Canadian research-intensive university. A peer review of teaching initiative (2010-12), led by a team of UBC national teaching fellows, was prompted by institutional concerns about the quality of student learning experiences and the effectiveness of teaching in a multi-disciplinary research-intensive university context. Canadian universities have long recognized the importance of attending to the evaluation of teaching practices in their particular contexts; however, the enactment of localized scholarship directed at these practices remains very much in its infancy. Traditional approaches to the evaluation of university teaching have often resulted in the over-reliance on student evaluation of teaching data and/or ad-hoc peer-review of teaching practices with numerous accounts of methodological shortcomings that tend to yield less useful and less authentic data. Using a case study research methodology, this paper examines the strategic development of a departmental SPRoT protocol at the University of British Columbia, Canada. Issues addressed in this article include contemporary approaches to the evaluation of teaching in higher education, faculty “buy-in” for the evaluation of teaching in a research intensive university, scholarly approaches to summative and formative Performance Reviews of Teaching (PRT), faculty-specific engagement in summative and formative (informal to formal) PRT training and implementation, and strategic institutional supports (funding, expertise, mentoring, technological resources). Keywords peer review of teaching, tenure and promotion, scholarship of teaching, case study www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 2 Published by SCHOLINK INC. 1. Introduction Historically, inadequate means of evaluating teaching have undermined the consideration of teaching in tenure, promotion, and re-appointment cases within research intensive universities (Bernstein, 2008; Elen, Lindblom-Ylänne, & Clement, 2007; Hammersley-Fletcher & Orsmond, 2004). For the most part, this has been due to the lack of rigour, authenticity, and credibility in peer reviews of teaching. Peer Reviews of Teaching (PRT) for high-stakes decisions (e.g., tenure and promotion) face a number of challenges (Chism, 2007; Hubball & Clarke, 2011), including:  Lack of systematically prepared and knowledgeable assessors.  Confusion about the relationship and distinction between formative and summative PRT.  Exclusive reliance on classroom observations by peers or student evaluations of teaching.  Methodological shortcomings that result in less authentic and credible data.  Potential conflict of interest associated with peers acting as reviewers. This article draws on a 10-year institutional initiative and examines whether and how a departmental Summative Peer Review of Teaching (SPRoT) Protocol was implemented at the University of British Columbia (UBC), Canada. 1.1 Context for PRT at The University of British Columbia, Canada As a research-intensive university, The University of British Columbia educates a student population of 50,000 and has over 250 graduate degree programs with 12 Faculties, 2 Colleges (Interdisciplinary Studies and Health Disciplines), and multiple Schools (see http://www.ubc.ca/). It is routinely ranked among the top 40 universities in world and among the top 3 in Canada (Times Higher Education World University Rankings, 2022). Therefore, if PRT is to be accepted for tenure and promotion decisions, it must be consistent with the standards of quality that are characteristic of scholarly work across the university (Hubball & Clarke, 2011). For this to happen, data from PRT must be rigorous and credible, ideally addressing all of the challenges outlined above. However, while UBC has long recognized the importance of student experience of instruction, most frequently gathered in end-of -term student evaluation of teaching, localized, discipline-specific scholarly approaches to PRT remains very much in their infancy (Glassick, Huber, & Maeroff, 1997; Hubball & Clarke, 2011; Kanuka, 2011; University of British Columbia, 2009). In 2010-12, UBC embarked on a major PRT initiative that was articulated in UBC’s Place and Promise: The UBC Plan, a “10-year visioning” document for the institution (University of British Columbia, 2010a). The academic plan outlined in the document emphasizes priorities and goals for enhancing the quality of teaching and learning across the university. In addition, a new educational leadership tenure track, culminating in a Professor of Teaching stream, emerged in 2012 to situate educational practices as both scholarly and professional endeavours within this research-intensive environment. These opportunities accentuated the need for more authentic and credible data on the quality teaching. In 2018, a new strategic plan Shaping UBC’s Next Century, focused around ten goals, emphasized www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 3 Published by SCHOLINK INC. transformative learning as a core area. “Excellence in transformative teaching” is supported through Educational Renewal (strategy 11 of 15 strategies), which emphasizes instructors as “highly effective teachers […] developing their craft” (University of British Columbia, 2018). UBC considers teaching as one of three essential components of academic life (research, teaching and service). Faculty members are expected to not only be pedagogically proficient but also to develop a scholarly approach to their university teaching practices (UBC, 2020). Heads, Deans, and the Senior Appointments Committee were also seeking better data on teaching so they can more responsibly carry out their respective oversight and evaluation responsibilities in the university. Thus, in 2009-2010, a UBC faculty working group report followed by a UBC PRT Initiative (UBC, 2009), led by three 3M National Teaching Fellows, Director of Faculty Relations and 12 representatives (Dean’s nominees), one from each of the Faculties on campus, developed discipline-specific guidelines and professional development strategies for PRT in order to enhance the quality of teaching evaluations within and across the disciplines at UBC. Typically, PRT has been undertaken by experienced academics for the purpose of evaluating new/less experienced/tenure-track faculty members. In most instances, there has been a disproportionate reliance on student evaluation of teaching or hastily conceived classroom observations. When these types of data are used, peer reviews are often dismissed or given little credence for re-appointment, tenure or promotion, or other reviews of teaching at UBC (e.g., formative reviews of teaching) (Pratt, 1997). Specifically, traditional PRT practices have lacked adequate prior dialogue or follow-up around critical issues such as a clear rationale for how particular judgments are made, as well as the appropriateness of guiding frameworks (institutional, disciplinary, programmatic, and teaching) and timelines used in the process. Further, the actual practice of PRT often bears little resemblance, and makes scant reference, to the current teaching and learning literature. Rarely does PRT scholarship clearly articulate “best practices” for teaching and learning within various disciplinary contexts, often with differing philosophical underpinnings (e.g., consider the qualitative emphasis in the arts versus the more quantitative emphasis in the sciences). 1.2 Theoretical Underpinnings to the Evaluation of Teaching in Higher Education Contemporary approaches to PRT are increasingly the subject of research in higher education. For example, a 2009 report by four Australian Universities recommended that peer review of teaching should be tailored to specific institutional needs and circumstances rather than adopting generic protocols. Further, contemporary approaches draw on authentic assessment and evaluation processes including attention to criteria, standards, data collection, trained reviewers and the important relationship between formative and summative peer reviews of teaching. For example, Figure 1 illustrates the relationship between formative and summative approaches to PRT. www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 4 Published by SCHOLINK INC. Figure 1. Relationship between Formative and Summative Approaches to PRT Formative peer review of teaching emphasizes periodic feedback to faculty members to improve teaching practices (Esterhazy, de Lange, Bastiansen, & Wittek, 2021). The progressive focus of formative peer review is essential to immersive continual professional development through iterative feedback. Formative approaches range from informal, classroom or practice-setting observations with follow-up collaborative reflection and discussion among colleagues, to formal, rigorous criteria-driven processes that simulate summative evaluations (Hubball & Díaz-Cidoncha García, 2022). Building on these approaches, summative PRT uses both internal and external evaluators to provide information and feedback to faculty members about the quality of their teaching. Peer, student, and self-evaluation can be integrated to offer a more holistic and contextual assessment of teaching (Bullough & Pinnegar, 2001; Cochran-Smith & Lytle, 2004). The evaluative focus of summative PRT inform decision-making considerations for reappointment, tenure, or promotion. Researchers have cautioned that PRT, as with peer review of other forms of academic work (e.g., merit reviews), peer reviewers must be sensitive to personal agendas and frames of reference, such as, their understanding of institutional and teaching contexts, personal beliefs and values about effective teaching, and their evaluative skills, as well as the beliefs, values, and intentions of the faculty member being reviewed. While recent literature has documented a best practice approach to formative and summative peer review of teaching, this approach can easily become performative, ad hoc and/or lost within the disciplinary context or complex (and often competing) priorities of research-intensive universities. Scholarly approaches to PRT deliberately build upon current practices by tailoring peer reviews of teaching to the diverse needs, circumstances and disciplinary contexts of one’s university. But in all cases, a scholarly approach to PRT (formative and summative) requires greater commitment to the process than ad-hoc or one-off evaluations that currently characterize the practice in many research-intensive universities. www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 5 Published by SCHOLINK INC. 1.3 Scholarly Approaches to PRT in Research-intensive Universities Scholarly approaches to PRT are part of a larger process of current institutional, curricular, and pedagogical reform in higher education (Ambrose et al., 2010; Chism, 2007). A scholarly approach to PRT, as with all forms of research, is based on underlying assumptions about knowledge, in this case, knowledge about teaching and learning. We believe that there are at least three assumptions that are significant in relation to PRT specifically that knowledge is: 1) personally constructed, 2) socially mediated, and 3) inherently situated (Cox, 2004). Each assumption provides direction and caution for the peer review of teaching. For example, the first assumption cautions that individual, local, and disciplinary conceptions of the “good” in teaching will always be part of the PRT process. Therefore, peer reviewers need to make explicit their personal beliefs about effective teaching (Atwood, Taylor, & Hutchings, 2000; Pratt, 1998). Similarly, those being reviewed should be allowed to clarify their own assumptions, beliefs, and intentions related to effective teaching. This bilateral transparency is critical to making peer reviews of teaching fair, as well as authentic. The socially mediated aspect of knowledge construction speaks to the negotiated nature of gathering and presenting authentic and credible knowledge about someone’s teaching. Negotiation may be more obviously relevant in the formative review process, but it is no less relevant in summative reviews. Indeed, scholarly approaches to summative reviews of teaching often involve faculty members from different institutional levels or disciplinary traditions, e.g., when internal and external reviewers are involved (Friedman, 2008; Healey, 2000). Coming to consensus about the quality of someone’s teaching may, therefore, require give and take across disciplinary traditions and diverse communities of practice. Negotiation between various stakeholders, e.g., administrators, peers, and students, can enhance the validity, reliability, and authenticity of PRT (Wenger-Trayner et al., 2015). Finally, knowledge about teaching is also inherently situated within disciplinary traditions, learning environments, and political landscapes that frame what is acceptable in both the review process and evidence of effective teaching. Scholarly approaches to PRT must, therefore, have regard for the historical, political, and contextual factors that are beyond the control of individual teachers and reviewers, but are part of the signature pedagogies or accepted practices within which they are teaching. Acknowledging the situated nature of knowledge about teaching is, therefore, also a matter of assessing the appropriateness of an individual reviewer within a culture of teaching (Stake & Cisneros-Cohernour, 2000; Lave & Wenger, 1991). Given that very little research has examined summative PRT experiences in diverse university settings, this paper draws on a 10-year institutional initiative and examines whether and how a departmental SPRoT protocol was implemented in a Canadian research-intensive university context. Drawing on case study research methodology, the following research question was designed to guide this investigation within one department at The University of British Columbia, Canada, whether and to what extent was a strategic SPRoT protocol implemented in UBC’s Department of Curriculum and Pedagogy (EDCP)? www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 6 Published by SCHOLINK INC. 2. Methodology 2.1 Case Study Research Case study research is a systematic in-depth inquiry that investigates a phenomenon within its real-life practice context (Pearson, Albon, & Hubball, 2015). Case study research methodology internalizes theory and practice by drawing on a wide range of contextually-bound data and engaging with key personnel. Accordingly, researchers in this study included Co-chairs of the FCP and original members of the UBC institutional PRT initiative (2010-12). Case study research enables research practitioners to develop a richer and more in-depth understanding of the phenomena under investigation (Webb & Welsh, 2019; Yin, 2017). Thus, case study research is highly generative in nature and is particularly well-suited to examine whether and how a strategic departmental SPRoT protocol was implemented in a Canadian research-intensive university context. Specific prompts for this inquiry included:  How does the institutional teaching context, with its particular features such as geographic region, organizational structure, strategic objectives, and allocation of resources, shape PRT initiatives? For example:  How is PRT understood, interpreted, organised, delivered, experienced, and/or institutionalized within EDCP?  What are the most common best practices and challenges for implementing PRT initiatives in EDCP?  How can PRT be better supported in the departmental setting? 2.2 Data Collection and Analysis Integral to this inquiry, a purposeful sample of contextually-bound program data were gathered from the following:  Relevant documentation from UBC strategic planning documentation (Indigenous Strategic Plan, 2020; Place & Promise, 2010; Strategic Plan 2018-2028);  Meetings. Mixed groups (including Co-Chairs UBC FCP, EDCP Teaching Review and Awards Committee (TRC) leaders, administrators, and faculty members) with members ranging from four to six stakeholders or individuals in either face-to-face venues or by using online video conference platforms, ranging from two to three sessions in total over a 3-month period.  Samples of teaching (online and face-to-face) materials including syllabi, podcasts, video recordings, worksheets, teaching dossiers, student evaluations of teaching, feedback and grading of students’ work, teaching awards, unsolicited letters of support for teaching;  Classroom teaching observations; and  Reflective field notes from the EDCP TRC team. www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 7 Published by SCHOLINK INC. Qualitative data sources were analyzed using the constant comparative method through itemizing, categorization, and finally to thematisation (Coe, Waring, & Hedges, 2017; Cresswell, 2013). Next, member checking was utilized to establish major themes, data patterns, and to discern complex commonalities, contradictions, and interactions with respect to PRT practices. The use of iterative and multiple data sources established the trustworthiness of the research findings through triangulation. 3. Result The focus of this study examined whether, to what extent and how a strategic EDCP SPRoT Protocol was implemented in UBC’s research intensive context. First, it is important to re-assert that a strategic EDCP SPRoT Protocol was implemented in order to address primary institutional concerns for the quality of scholarly and evidence-based teaching evaluation reports that were required for the high stakes purpose of tenure and promotion at UBC. Thus, with a strong focus on SPRoT, the TRC in EDCP were guided by a number of discipline-specific questions drawn from the UBC context and the literature related to peer review of teaching:  What are critical elements of an effective summative PRT report?  What evidence/data for summative PRT are appropriate?  What criteria for summative PRT are appropriate?  What standards for summative PRT are appropriate?  What ethical principles for summative PRT are appropriate?  What engagement sequences for summative PRT are appropriate? 3.1 Frameworks for Scholarly Approaches to PRT in EDCP We argue that there is no one single institutional strategy or scholarly approach to PRT in research-intensive university contexts. Building on PRT perspectives presented in the literature and research, the organizational framework shown in Figure 2 was useful for facilitating scholarly approaches to PRT within EDCP. www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 8 Published by SCHOLINK INC. Figure 2. Operational Framework for Conducting a Scholarly Approach to PRT in a RIU Figure 2 provides a flexible and iterative framework that takes into account the PRT context, and integrates responsive institutional strategies for conducting scholarly approaches to PRT. Practical strategies, outlined briefly below for each component of the framework, are drawn from a combination of literature sources and research at UBC (Hubball & Clarke, 2011; UBC, 2009). 3.1.1 PRT Context PRT context refers to the attention and sensitivity given to the “big picture” that shapes peer review of teaching practices. This can be achieved through appropriate leadership, research and ethical considerations, consultation, dialogue, collaboration, and attention to adequate support and incentives to conduct PRT. These strategies ensure that the PRT experience is not only meaningful and relevant to the needs and circumstances of faculty members, but it is also manageable to administer, and, above all, empowers the community to engage in scholarly approaches to PRT. Within the EDCP context, leadership support, including recognition of PRT team as a departmental committee and ensuring that their contributions were recognized as service (within tripartite work contract at UBC: research, teaching, and service), lend credence to the process as a scholarly practice. Additionally, a clear distinction between formative assessment and summative evaluation clarifies the importance of PRT committee within the department processes. Faculty members on the committee are familiar with the university, faculty, and departmental processes for evaluation of teaching and policies for reappointment, tenure, and promotion 3.1.2 PRT Planning PRT planning refers to the preparation and development of short- and long-term PRT goals including the scheduling of timely (e.g., pre-assessment, assessment, and post-assessment) meetings, deadlines, and expectations with respective parties (e.g., Department Head, reviewers, and reviewed faculty member when appropriate), which, in part, drive the PRT process. For example, these meetings can be focused on discussions to clarify signature pedagogies (such as PBL in Health, Case Study in Law, www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 9 Published by SCHOLINK INC. Aesthetics in the Arts), and appropriate forms of assessment and evaluation. Situated with the larger Faculty of Education, PRT planning as a department includes annual timing to coincide with faculty reappointment, tenure, and promotion deadlines and managing subject specific perspectives that may be part of disciplinary training (e.g., historians in Social Studies education, kinesiologists in Physical education, chemists, biologists, and physicists in Science education, etc.). For any process to be fair and equitable, it must be as transparent as possible to the participating members. For that reason, a protocol for the PRT should be established and agreed upon by the members in the Department/Faculty. An example of one such protocol that was developed in one Faculty context included the following sequence: Sample protocol for summative PRT  notification of required summative PRT  meeting of PRT committee (internal and external reviewers) to discuss context, appropriate data, criteria, standards, and process expectations  meeting of candidate with PRT committee member to discuss/clarify context, data collection, criteria, standards, and process expectations  data collection to obtain a long and broad perspective of a faculty member’s teaching practice  meeting of candidate with PRT committee member to discuss and debrief outcomes from PRT  meeting of external and PRT committee to discuss data analysis and formulation of summative report that is submitted to Department Head by external PRT committee member) 3.1.3 PRT Data Sources PRT data sources are typically quantitative and qualitative in nature, and include teaching workload statistics, classroom observations of teaching, course syllabi, teaching dossier, and student evaluations of teaching. Data sources must be appropriate and sensitive to assessing a broad perspective of teaching practices, including evidence about context, process, outcomes, and impact within the institution. The inclusion of peer, student, and self-evaluation data is important to help address the PRT challenges including assumptions about knowledge in teaching and learning. Moving away from a single data source (classroom observations or student evaluations) and building in, where possible, formative cycles of feedback, provides additional data to contextualize the activities, pedagogical understanding, and personal growth of each faculty member 3.1.4 PRT Criteria and Standards PRT is guided by appropriate frameworks related to logistical and pedagogical criteria (e.g., command over subject matter and representation of recent developments in the field, preparedness, relationship between goals/objectives, student engagement strategies, and assessment of learning) and standards for effective teaching (e.g., descriptors pertaining to designated criteria for percentile or A, B, C, D ratings; exceeds, meets, or does not meet standards; strengths and weaknesses) if judgments about the quality of a faculty member’s teaching practice is to be authentic and credible within specific contexts. www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 10 Published by SCHOLINK INC. Within EDCP, departmental criteria and standards use scholarly approaches to integrate institutional requirements (Guide for Senior Appointments Committee, etc.) with the current literature and practices. Various criteria frameworks for effective teaching have been documented in the higher education literature. The following criteria are suggested in the collective agreement at the University of British Columbia (UBC, 2020 Section 3.2.1-3.2.7). Sample criteria for summative PRT  command over subject matter (how knowledgeable or authoritative)  representation of recent developments in the field (what’s in; what’s not)  preparedness (for individual sessions and for overall course/term)  presentation  student engagement  influence on the intellectual and scholarly development of students  relationship between goals/objectives and assessment and evaluation of student learning  appropriateness of course materials and requirements (given the topic and level)  articulation with other programmatic courses/elements  responsiveness/accessibility to students  demonstration of a scholarly approach to teaching  effective graduate supervision (as appropriate) Making judgments on aspects of a colleague’s teaching practice (evaluation) is typically the most challenging dimension of PRT. Evaluation challenges often emerge due to inadequate attention to communities of practice, inappropriate criteria, and vague standards. For example, judgments tend to focus on committee members’ interpretations of statements such as “Exceeds”, “Meets”, or “Does not meet” expected standards. The following descriptions (Table 1) are provided as examples and illustrate valuable specificity (further developed and contextualized by each Faculty) that, in part, overcomes idiosyncratic interpretations. www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 11 Published by SCHOLINK INC. Table 1. Sample Standards for Summative PRT A Exceeds Faculty/Departmental Expectations Reserved for the (usually) few exceptional examples of teaching practice consistent with the standard, reserved for those that the Department/Faculty would nominate for teaching awards. Both internal and external peer reviewers need to be in agreement with the evidence presented. Meets Faculty/Departmental Expectations Evidence suggests there is generally a high quality throughout the instructor’s teaching practice, no problems of any significance, and evidence of consistent attention to the following: student engagement, effective teaching practices, and developing a scholarly approach to teaching and learning. Does Not Meet Faculty/Departmental Expectations Evidence suggests there is generally an inadequate quality in the instructor’s teaching practice and serious flaws or deficits in the instructor’s understanding of teaching and learning. 3.2 Data Used for Summative Evaluation of Teaching Various types of data for summative peer review of teaching are documented in the literature (Chickering & Gamson, 1987). Building on the earlier organizational framework for conducting scholarly approaches to PRT, data sources can be strategically categorized within the parallel conceptual framework shown in Figure 3 (Hubball & Clarke, 2011). Figure 3. Potential Data Sources for PRT 3.2.1 Teaching Context Teaching Context data focus on critical structures that shape a faculty member’s teaching practice. Therefore, a comprehensive needs assessment involving consultations and collaborations between respective parties is required in order to situate a faculty member’s teaching practice within the SoTL www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 12 Published by SCHOLINK INC. literature, institutional visioning documents, pedagogies within their context and discipline, and an individual’s academic workload. 3.2.2 Teaching Process Teaching Process data focus on issues of importance that arise throughout a faculty member’s teaching practice. For example, to what extent are learning outcomes made explicitly to the students through course syllabi? To what extent are individual instructors incorporating learning-centred classes that are responsive to the needs and circumstances of the students? To what extent is the instructor drawing on an appropriate selection and sequencing of active learning methodologies and developing a reflective teaching disposition to guide further teaching development? This could also be better served with annual quality assurance and quality enhancement strategies for further developing both formative and summative departmental PRT practices. 3.2.3 Teaching Outcome Teaching Outcome data focus on immediate outcomes of a faculty member’s teaching practice. For example, what are key student learning outcomes from this particular teaching approach and how do students rate the quality of the educational experience? It might also encompass an examination of the quality of students’ work and an examination of the faculty member’s student grading practices. 3.2.4 Teaching Impact Teaching Impact data focus on the long-term (e.g., months, years) impact of a faculty member’s teaching practice. This might involve, for example, an examination of the faculty member’s long-term impact on and contributions to teaching and learning within a subject or unit. It might also encompass a longer-term analysis of the range in quality of students’ work, a longitudinal examination of grades given, and an examination of the faculty member’s responses to and subsequent changes from previous formative peer review of teaching feedback/reports. See Appendix for a Sample Summative Review Report. 4. Discussion Departmental Teaching Review and Awards Committee (TRC) leaders, collectively and individually, have progressed through various stages of PRT professional development training pertaining to scholarly PRT processes. Not unexpectedly, various challenges emerged from these PRT experiences. Initially, for example, there was a great variation in quality and rigour of initial summative PRT reports that were disseminated among the group. On review, these generated useful debates about disciplinary cultures with respect to PRT (e.g., misunderstandings about formative and summative PRT practices, initial confusion and anxiety about the rationale and interpretations for internal and external PRT evaluators, misunderstandings about PRT sources of data collection). For example, one of the most pressing challenges, has been the lack of departmental and Faculty-wide support, resources and recognition for the importance of an external reviewer in the SPRoT process (i.e., comparable to www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 13 Published by SCHOLINK INC. peer-evaluations of research productivity by colleagues in other Faculties and/or similar institutions for the purpose of institutional tenure and promotion considerations) which rendered a major limitation to authentic SPRoT practices. However, over time, inspired by cross-disciplinary discussions pertaining to current practices, significant progress was made and greater coherence agreed upon with respect to authentic (e.g., signature pedagogies) and scholarly approaches to PRT within the department that includes external reviewers (i.e., those not immediately from the unit or department of the Faculty member being review). Nonetheless, there is still considerable discussion and debate within EDCP and UBC more broadly about external reviewers. These discussions are aided by the fact that many TRC leaders have served on Faculty and institutional promotion and tenure committees. Beginning with an understanding of the quality and type of report that is required by the Dean’s Advisory and Senior Appointments Committees, has guided the development of rigour, longitudinal, and broad perspective reporting on teaching practice. This insight has guided the development of a template that includes multiple data considerations, both qualitative and quantitative sources. The EDCP SPRoT Protocol has been adopted as the Faculty-wide model. One important outcome that emerged, sped forward by the transition to online learning during the Covid-19 pandemic, was the effective and efficient use of digital technologies in order to facilitate faculty members’ engagement in scholarly approaches to PRT. The appropriate use of technology was deemed to provide greater flexibility and, in many cases, improve the quality of analysis of teaching practices. For example, the increasing use of video conferencing tools (e.g., Zoom) was deemed to facilitate pre- and post-assessment meetings, as well as e-portfolios for teaching dossiers and online teaching and digital recordings of “classroom” experiences (e.g., split screen video with one camera focusing on the instructor and the other on the class). However, it should be noted that technology cannot always adequately capture true classroom atmospheres or teacher-student dynamics and/or tensions in diverse classroom settings and sometimes posed difficulties with the technology itself. Evidence thus far suggests five issues that arose during this study: (1) significant differences have occurred in pre-post PRT reporting protocols; (2) insufficient familiarity and clarity with PRT literature and key concepts such as formative, summative, external and internal reviewers; (3) lack of understanding regarding diverse and authentic PRT data sources; ignorance around explicit PRT sequencing; (4) lack of methodological rigour of PRT processes by refining/developing reports to incorporate key PRT concepts; and (5) the need for a greater sense of community and cohesion among EDCP PRT leaders within and across the disciplinary domains (e.g., science education, art education, etc.). Scholarly approaches to PRT often require faculty members to move beyond their own disciplinary orientation and embrace broader social science methodologies, which for many is epistemologically and ethically challenging (Kanuka, 2011). Additionally, PRT may expose unknown weaknesses but not strengths in individual teaching practices (Barrios-Rodriguez et al., 2022). Although, if coupled with a www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 14 Published by SCHOLINK INC. strategically aligned formative PRT practice/policy, we are hopeful that we can develop an ongoing and iterative cycle of peer review that supports faculty development and provides additional data for the SPRoT report. Despite these inevitable challenges, the early signs for implementation of the UBC PRT initiative are very encouraging, though we are closely analyzing progress annually. Finally, where appropriate, EDCP TRC provides additional support for mentoring and nominations for teaching awards. The intent of this is to build discipline-specific protocols that are sensitive to the pedagogical norms of the discipline, profession, and field of practice in the department. 5. Conclusion This article has outlined a PRT initiative at one Canadian research-intensive university and the issues, practices, and challenges associated with implementing that initiative. We are still learning and trust that the progress made to date will provide a strong foundation for a reputable and respected PRT process within and across the disciplines into the future. We acknowledge that we will still have to overcome existing suspicions and conceptions that linger from earlier and less scholarly PRT efforts. Thus, as outlined in this paper, we have attempted to provide a more scholarly underpinning to PRT reconceptualization of the process. We hope that through this study we can inform ongoing campus-wide renewal and review. A scholarly approach to PRT in a research-intensive university is a complex and multifaceted process. It involves focused attention to the needs and circumstances of discipline-specific communities of practice, relevant literature and conceptual frameworks, systematic methodology for authentic assessment and evaluation, ethical considerations, and effective dissemination of outcomes. Summative PRT should take a broad and long perspective of a faculty member’s teaching practice with explicit procedures and guiding principles (e.g., scholarship, accuracy, integrity, transparency, diversity, credibility, and usefulness). Further, to implement formative and summative scholarly approaches to PRT, in a research-intensive university requires appropriate resourcing for peer reviewers’ time, expertise, and training costs. Consequently, if scholarly approaches to PRT are not adequately supported, confronting implementation issues can present significant challenges for many reviewers and administrators-the magnitude of which may well be an outright deterrent for some academic units to engage in scholarly approaches to PRT (Hubball & Clarke, 2011). While there are still many significant challenges and areas for improvement in this department, the growing local and institutional support, and widespread attention to scholarly approaches to the PRT within and across disciplines is a testimony to the increasing value placed on scholarly approaches to PRT in research-intensive universities. An institutional commitment to research, therefore, far from being a barrier to improving teaching, can be publicly engaged as the basis for authentic assessment and evaluation of teaching in a research-intensive university setting. www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 15 Published by SCHOLINK INC. Acknowledgements The authors would like to express their sincere thanks to institution-level educational leaders and colleagues from UBC who have provided key insights for this paper. References Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., Norman, M. K., & Mayer, R. E. (2010). How learning works: Seven research-based principles for smart teaching. San Francisco: Jossey-Bass. Bernstein, D. J. (2008). Peer review and evaluation of the intellectual work of teaching. Change, 40(2), 48-51. https://doi.org/10.3200/CHNG.40.2.48-51 Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building networked improvement communities in education. In M. T. Hallinan (Ed.), Frontiers in Sociology of Education (pp. 127-162). Dordrecht, The Netherlands: Springer. https://doi.org/10.1007/978-94-007-1576-9_7 Bullough, R., & Pinnegar, S. (2001). Guidelines for quality in autobiographical forms of self-study research. Educational Researcher, 30(3), 13-21. https://doi.org/10.3102/0013189X030003013 Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 39(7), 3-7. Chism, N. V. (2007). Peer review of teaching: A sourcebook (2nd ed.). Bolton, MA: Anker Publications. Cochran-Smith, M., & Lytle, S. (2004). Practitioner inquiry, knowledge and university culture. In J. J. Loughran, M. L. Hamilton, V. K. LaBoskey, & T. L. Russell (Eds.), International Handbook of Self-Study of Teaching and Teacher Education Practices (pp. 601-605). Dordrecht, Netherlands: Kluwer Academic Publishers. Coe, R., Waring, M., & Hedges, L. (2017). Research methods and methodologies in education (2nd ed.). Sage, London: UK. Cresswell, J. W. (2013). Qualitative inquiry & research design: Choosing among five approaches (3rd ed.). Thousand Oaks, CA: Sage Publishers. Elen, J., Lindblom-Ylänne, S., & Clement, M. (2007). Faculty development in research-intensive universities: The role of academics’ conceptions on the relationship between research and teaching. International Journal for Academic Development, 12(2), 123-139. Esterhazy, R., de Lange, T., Bastiansen, S., & Wittek, A. L. (2021). Moving beyond Peer Review of Teaching: A conceptual framework for Collegial Faculty Development. Review of Educational Research, 91(2), 237-271. https://doi.org/10.3102/0034654321990721 Glassick, C. E., Huber, M. T., & Maefoff, G. I. (1997). Scholarship assessed: Evaluation of the Professoriate. San Francisco, CA: Jossey Bass. https://doi.org/10.3200/CHNG.40.2.48-51 https://doi.org/10.1007/978-94-007-1576-9_7 https://doi.org/10.3102/0013189X030003013 https://doi.org/10.3102/0034654321990721 www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 16 Published by SCHOLINK INC. Hammersley-Fletcher, L., & Orsmond, P. (2004). Evaluating our peers: Is peer observation a meaningful process? Studies in Higher Education, 29(4), 489-503. https://doi.org/10.1080/0307507042000236380 Hubball, H. T., & Clarke, A. (2010). Diverse methodological approaches and considerations for SoTL in Higher Education. Canadian Journal of the Scholarship of Teaching & Learning in Higher Education, 1(1). https://doi.org/10.5206/cjsotl-rcacea.2010.1.2 Hubball, H. T., & Clarke, A. C. (2011). Scholarly approaches to Peer-Review of teaching: Emergent frameworks and outcomes in a Research-Intensive University. Transformative Dialogues: Teaching & Learning Journal, 4(3), Article 7. Hubball, H. T., & Díaz-Cidoncha, G. (2022). Peer review of coaching in diverse international amateur/grassroots 50-65+ small-sided football contexts. International Science & Football Association Newsletter Magazine (6th ed.). Hubball, H. T., Clarke, A., & Pearson, M. L. (2017). Strategic leadership development in research-intensive higher education contexts: The Scholarship of Educational Leadership. In P. Tripathi, & S. Mukerji (Eds.), Handbook of Research on Administration, Policy, and Leadership in Higher Education (pp. 1-19). Hershey, PA: IGI Global. https://doi.org/10.4018/978-1-5225-0672-0.ch001 Hutchings, P., Huber, M. T., & Ciccone, A. (2011). The scholarship of teaching and learning reconsidered: Institutional integration and impact. San Francisco, CA: Jossey-Bass. Kanuka, H. (2011). Keeping the scholarship in the scholarship of teaching and learning. International Journal for the Scholarship of Teaching and Learning, 5(1). https://doi.org/10.20429/ijsotl.2011.050103 Lomas, L., & Kinchin, I. (2006). Developing a peer observation program with university teachers. International Journal of Teaching & Learning in Higher Education, 18(3), 204-214. Pearson, M. L., Albon, S. P., & Hubball, H. (2015). Case study methodology: Flexibility, rigour, and ethical considerations for the scholarship of teaching and learning. Canadian Journal for the Scholarship of Teaching and Learning, 6(3). https://doi.org/10.5206/cjsotl-rcacea.2015.3.12 Rocío, B.-R., Inmaculada, S.-B., José, J. J.-M., Macarena, L.-L., Noelia, G.-C., Enrique, J. C., José, D. V. R., Rocío, O.-R., Carmen, A.-P., Sandra, M.-P., Carmen, M. G. D., Juan, P. A. M., Raúl, A. R., María, E. G.-R., & Pilar, R. (2022). Peer review of teaching: Using the nominal group technique to improve a program in a university setting with no previous experience. International Journal for Academic Development. Stake, R. E., & Cisneros-Cohernour, E. J. (2000 Fall). Situational evaluation of teaching on campus. New Direction in Teaching and Learning, 83, 51-72. The University of British Columbia. (2009, April). Report of the UBC Working Group on Peer-review of Teaching. https://doi.org/10.1080/0307507042000236380 https://doi.org/10.5206/cjsotl-rcacea.2010.1.2 https://doi.org/10.4018/978-1-5225-0672-0.ch001 https://doi.org/10.20429/ijsotl.2011.050103 https://doi.org/10.5206/cjsotl-rcacea.2015.3.12 www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 17 Published by SCHOLINK INC. The University of British Columbia. (2010). Place and Promise: The UBC Plan. President’s Office, UBC Press. The University of British Columbia. (2018). Shaping UBC’s next century: Strategic plan 2018-2028. Retrieved from https://strategicplan.ubc.ca/ The University of British Columbia. (2020). Indigenous Strategic Plan. Retrieved from https://isp.ubc.ca/ The University of British Columbia. (2020, Sept). Guide to Reappointment, Promotion and Tenure Procedures at UBC. Retrieved from https://hr.ubc.ca/sites/default/files/documents/SAC%20Guide.pdf Webb, A. S., & Welsh, A. J. (2019). Phenomenology as a methodology for SoTL research. Teaching & Learning Inquiry, 7(1), 168-181. https://doi.org/10.20343/teachlearninqu.7.1.11 Wenger-Trayner, E., Fenton-O’Creevy, M., Hutchinson, S., Kubiak, C., & Wenger-Trayner, B. (Eds.). (2015). Learning in landscapes of practice: Boundaries, identity, and knowledgeability in practice-based learning. Routledge, New York, N.Y. https://doi.org/10.4324/9781315777122 Yin, R. K. (2017). Case study research and applications: Design and methods (6th ed.). Thousand Oaks, CA: Sage Publications. Appendix Sample report template for a summative PRT The following excerpts (purposely edited) provide a working example of a summative PRT report at UBC. --------------------------------------------------------------------------------- Date: XXXX To: XXXX Head Department of XXXXX From: XXXXXX Re: Peer Review of Teaching for Summative Peer review of Teaching: Feedback Report Introduction We have worked with XXXXXXX over the past six weeks to review his/her teaching practice. The departmental guidelines for the “Colleague Review Process” define teaching practice to include post-Baccalaureate courses, graduate courses, and membership (including supervision) of graduate student thesis committees. Departmental guidelines identify key data sources and criteria when https://doi.org/10.20343/teachlearninqu.7.1.11 https://doi.org/10.4324/9781315777122 www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 18 Published by SCHOLINK INC. determining the standard (exceeds, meets, or does not meet the standard of teaching expected of faculty members in this department) upon which to evaluate teaching practices. This report looks back over XXXXXXX’s recent accomplishments and forward to professional development goals for future teaching practice. Prior to the peer review, we met with XXXXXXX to discuss the context of his/her teaching, departmental criteria and standards for effective teaching, his/her teaching and course goals, and the peer review protocol. In compiling this report, therefore, we drew on the following data sources:  our pre-instruction, post-instruction, and follow-up discussions with XXXXXXX  his/her course syllabi and lecture plans (e.g., is based on current scholarship and literature; uses authentic methods to assess and evaluate student learning outcomes; contributes to departmental/programme goals; and articulates a rationale for pedagogical approaches)  her/his scholarly teaching dossier (if available)  two peer classroom observations of XXXXXXX (e.g., breadth and depth of pedagogical repertoire; clear and helpful classroom discourse; engagement with and responsiveness to students; and inclusiveness and fair treatment of student diversity is evident)  comments elicited specifically for this evaluation from XXXX students (graduate and undergraduate) for whom XXXXXXX was a course instructor and/or supervisor (e.g., Is reasonably accessible; provides timely feedback; offers high quality guidance; knows institutional/departmental procedures)  her/his student evaluation of teaching (SEoT) open-ended comments and numerical scores on teaching practice from UBC’s 6 module teaching evaluation components  student grading practices (including distributions and justification, review of feedback on students’ assignments)  his/her 2-page reflection paper pertaining to XXXXXXX’s interpretation of his/her previous formative PRT or SEoT data from XXXXX to XXXXX XXXXXXX’s teaching expertise is in the areas of XXXXX, with a scope that is interdisciplinary and international. XXXXXXX has a strong commitment to diversity and innovation in his/her teaching practice, evident in his/her teaching philosophy statement and pedagogic goals. For example, XXXXXXX’s teaching practice draws on and uses a wide range of learning and teaching strategies (such as XXXXX) in order to recognize, acknowledge, and honour XXXXX in the student learning experience. www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 19 Published by SCHOLINK INC. Major Teaching Contributions XXXXXXX’s contributions to teaching at UBC are significant and varied. For example, she/he is the XXXXX, she/he is a graduate student advisor, and teaches in both the graduate and undergraduate programs within the Faculty. Since coming to UBC, XXXXX has been involved in direct supervision, co-supervision, and committee work of several graduate students at the XXXXX and PhD level. In the XXXXX years that he/she has been at UBC, a total of XX graduate students (XXXXX) with whom she/he worked as Principal Supervisor or Committee Member have successfully defended their research theses/dissertations. In addition to guiding his/her graduate students in their research inquiries, she/he has actively encouraged and successfully supported a number of them in developing conference proposals, presenting conference papers, and preparing articles for publication. One of her/his recent XXXXX graduates has since enrolled in the PhD Program in XXXX and she/he was successful in securing one of the prestigious 4-year XXX graduate scholarships. The following two quotes from his SEoTs demonstrate how his/her teaching (both graduate and undergraduate) has been perceived and interpreted by students at UBC. The first quote refers to his undergraduate teaching, and the second quote comes from a student who took one of his/her XXXXXX graduate courses. XXXXX (graduate student XXXXX) Graduate Student Supervision Since 2007, XXXXX has worked with several graduate students. We were able to contact XXX of these students to request feedback about XXXXX’s graduate teaching practice and supervision. All responses consistently speak of a highly dedicated and talented teacher. A representative selection of their comments (and those from a selection of undergraduate students) include the following: XXXXX l (XXXX May-June 2022). Meetings with XXXXX: Reflective Practice and Professional Development Our meetings with XXXXX proved to be valuable in that we were able to move beyond artifacts and observations to a conversation about the assumptions that underlie and give meaning to his/her teaching practice. We were impressed with the thoughtfulness, enthusiasm and care with which XXXXX talks about and reflects on his/her teaching, and the willingness to constantly seek ways of further developing her/his teaching repertoire. www.scholink.org/ojs/index.php/grhe Global Research in Higher Education Vol. 6, No. 1, 2023 20 Published by SCHOLINK INC. Student Evaluation of Teaching (SEoT) Numerical Scores on Teaching Practice The SEoT Office provided a summary report of the numerical scores for three undergraduate courses and four graduate courses XXXXX has taught while at UBC. The scores are based on student responses to a 30-item questionnaire. The report documents XXXXX’s record of achievement beginning with courses she/he taught in 2007 as an AAAAA Professor. SEoT records document XXXXX courses which Dr. XXXXX has taught since the fall of XXXXX. Since XXXXX, the three-year faculty average for undergraduate courses was XXXXX, and the graduate average was XXXXX. Dr. XXXXX was above the faculty mean for one of the undergraduate courses and above the faculty mean for all XXXXX of the graduate courses listed. Dr. XXXXX’s weighted overall average was XXXXX across XXXXX undergraduate courses, and XXXXX across all four graduate courses she/he has taught since XXXXX (SEoT Summary Report, October XXXXX). Taken overall, these results reflect a very high standard of teaching. Summary We commend XXXXX for her/his valuable contributions to the graduate and undergraduate (XXXXX) programs and for his/her commitment to embark on professional development initiatives to expand and improve his/her teaching practice in order to enhance student learning. In our opinion, when taking into account all data sources and criteria, XXXXX meets the teaching standards of our department.