Microsoft Word - akkaraju_oral_exam_final.docx Journal of Effective Teaching in Higher Education, vol. 6, no. 1 https://doi.org/10.36021/jethe.v6i1.354 The Oral Exam—Learning for Mastery and Appreciating It Shylaja Akkaraju, CUNY Bronx Community College, shylaja.akkaraju@bcc.cuny.edu Abstract. To reduce academic dishonesty and strengthen learning outcomes, I adopted in-depth oral examinations as my benchmark and summative assessments in a Human Anatomy & Physiology course taught in an online asynchronous setting. This decision led my students and me down the transformative path of mastery learning. This was a threshold experience for my students who were learning how to think and express themselves as physiologists. This was also a threshold experience for me as I explored the scope of the oral examination in promoting skill acquisition while nurturing a relationship-rich learning environment. By employing deliberate practice principles including basic drills, one-on-one weekly check-ins, and small group recitation sessions, students exceeded benchmarks for conceptual understanding, mastery of fundamentals, and application of concepts to clinical scenarios. Students consistently reported that they were happy within this learning environment. With meticulous planning, it is possible to motivate students to learn for mastery and acquire expertise by employing oral exams as the pivotal assessment strategy in an online course thereby also making academic dishonesty almost irrelevant. Keywords: oral exams; mastery learning; threshold concepts; deliberate practice; pedagogy of care Most Human Anatomy & Physiology instructors believe that academic integrity is compromised in online courses (Carrasco, 2022). However, it has been shown that oral exams can hold the key to avoiding academic dishonesty while promoting excellence (Schurgers & Qui, 2022). In fact, oral exams can deepen engagement in a way that traditional assessments cannot (Theobold, 2021). I decided to adopt the oral exam for both benchmark (monthly) and summative (end of semester) assessments not just to reduce academic dishonesty issues but also as a potentially powerful vehicle to assess mastery of the subject matter. When instruction is designed with mastery learning in mind, the oral examination can serve as a highly effective assessment vehicle to evaluate mastery learning (Carrasco, 2022; Joughin, 2010) because it provides a wealth of information about student understanding (Theobold, 2021). Mastery learning requires a great deal of scaffolding, allowing learners to engage in deliberate practice, which culminates in the demonstration of expert performance (Ericsson, 2008). The principles of deliberate practice that have been established in the training of elite athletes, musicians, dancers, and chess grandmasters have a lot in common (Coyle, 2009; Ericsson, 2008; Ericsson & Harwell, 2019). They all need unlimited trials that provide opportunities to focus on fundamentals, form mental representations, engage in repeated performance, practice slow learning, receive immediate feedback, and learn from errors (Ericsson & Pool, 2017). The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 67 For the instructor, deliberate practice requires meticulous planning and streamlining of content, making mastery learning design a threshold concept . A threshold concept is a doorway that leads to an experience that is both troublesome and transformational resulting in an entirely new way of thinking about something (Meyer & Land, 2003). A threshold concept is troublesome because at the beginning of the learning period, it may appear to be counterintuitive. It is transformational because by the end of the learning period, there is a major shift in thinking. Mastery learning and expert performance can appear to be counterintuitive for most instructors because we do not normally expect introductory biology students to “think and talk like physiologists.” We expect such behavior from college seniors or graduate students; for this reason, mastery learning is a troublesome concept. If we were to demonstrate that college students taking an introductory course can acquire expertise, then we are entering the zone of transformation. The threshold experience itself is not without struggle, a sometimes-messy process which is played out within the learning environment known as the liminal space. Those who are yet to engage in the threshold experience are said to be pre-liminal; whereas, those that have exited the liminal space fully transformed are said to post-liminal. All others are liminal (Land et al., 2005). For students, mastery learning via oral exams is troublesome because it is unfamiliar, challenging, and demands deliberate practice to succeed. It is also transformational because students gain confidence in their ability to “speak physiology” not only during oral exams and practice sessions but also in everyday life with their family and friends. Typically, within the liminal space, students struggle to achieve success as they take excursive (bold and exploratory) and recursive (timid and retreating) journeys during the threshold experience (Figure 1). Therefore, it is imperative that the instructor create a supportive liminal environment in which the students can engage and flourish (Cousin, 2006). Figure 1 Learning for Mastery is a Threshold Experience for the Instructor and Student The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 68 In the context of our collective trauma of the pandemic, the need for a highly supportive liminal environment and concepts such as pedagogical love, pedagogy of kindness, and relationship-rich education have become increasingly relevant (Akkaraju et al., 2019; Cho, 2005; Lambert & Felten, 2020; Loreman, 2011; Maatta & Uusiautti, 2012; Wilkinson & Kaukko, 2020). It has been shown that student success is attributable to a relationship-rich learning environment (Lambert & Felten, 2020). Care in pedagogy is classified as aesthetic care and authentic care (Valenzuela, 1999). Aesthetic care is superficial and limited to caring about student performance and disposition. Authentic care is the deeper version that emphasizes a reciprocal relationship between a student and teacher in which the student feels cherished. To fulfil the conditions of aesthetic care, the tone, frequency, ease, and mode of communication becomes a crucial part of the instructional design (Pritts, 2020). The use of a mobile messaging application can be a powerful way to impact student success (Sherr et al., 2019) by making the students feel cared for while gently nudging them into completing all assignments on time. Combined with the use of laughter, smile, thumbs up, and other positive emojis, text messaging allows for more relaxed communication with the students. In this article I address the following: (a) scaffolding formative assessments that are congruent with oral exam outcomes; (b) assessing for mastery; (c) structuring deliberate practice to result in mastery; (d) viewing mastery learning as a threshold experience to help create and maintain a nurturing liminal environment; and (e) creating a nurturing liminal environment which is central to mastery learning. Methods I studied the effectiveness of oral exams in promoting mastery learning for a period of three consecutive semesters in an online, asynchronous section of Human Anatomy & Physiology course at Bronx Community College of the City University of New York (CUNY). Each section consisted of 17–20 students with a total of 57 students. Student performance data on oral exams was collected for all benchmark and summative assessments. Student perceptions of weekly formative assessments, oral exams, and weekly check-ins were collected using an anonymous survey at the end of the semester. The project gained approval from the CUNY Institutional Review Board prior to the start of this project. Scaffolding Formative Assessments to Match Oral Exam Outcomes A major learning outcome for the oral exam was that the student would demonstrate conceptual clarity. As a reuslt, weekly formative assessments were geared towards enabling the student to develop knowledge and skills that would gamely meet this outcome. Content was streamlined which helped to focus on the fundamentals, deepen understanding of core concepts, and enhance the learner’s ability to readily apply physiological concepts to clinical scenarios. To achieve conceptual clarity, students were expected to engage in a series of scaffolded learning experiences that helped them to connect with the material, master the basics, dig deeper into the concepts, amplify learning through sketching and writing by hand, and teach concepts assigned for the week (Figure 2). The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 69 In step 1, students were expected to take notes from mini-lecture videos that I had created and uploaded to my YouTube channel (Akkaraju, n.d.). In step 2, students answered multiple choice or matching questions at specific checkpoints of the mini- lecture video. In step 3, students practiced vocabulary and basic conceptual information by taking a quiz that was designed with the purpose of drilling the fundamentals. In step 4, they re-wrote notes to make sure that their information was accurate. These handwritten notes were provided by me and were congruent with the questions on the oral exam. In step 5, they took a comprehensive quiz addressing deeper conceptual and application questions. I held a weekly recitation session to review the quiz that students attended on a voluntary basis. Recordings from these recitations were made available to the entire class. In step 6, students picked a time slot of their convenience to meet with me individually for 15 minutes to explain the concept of the week and to track their progress. Figure 2 Weekly Formative Assessments Congruent with Oral Exam Outcomes The formative assessments were scaffolded to maximize congruence. A core concept such as blood pressure travels through several learning experiences (formative and benchmark) by the time a student achieves conceptual clarity (Table 1). The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 70 Table 1 Concept Tracking to Maximize Congruence Learning opportunity Assessment type Software used Description of activity Learning outcome (Skill) Learning outcome (Knowledge) Notetaking Formative Explain Everything Basics & YouTube Watch mini- lecture video & take detailed notes including sketches Transcription fluency Spelling Sketching to learn Factual Check understanding Formative PlayPosit Answer questions at various checkpoints Retrieval practice Factual Vocabulary Gamify learning the basics Formative Quizlet Practice with flashcards & self- testing Retrieval practice Vocabulary Factual Conceptual & Contextual learning Formative GoodNotes Practice redrawing & sketching to engage with the concepts Retrieval practice Flow charting Sketching Conceptual Procedural Teaching to learn Formative Zoom Engage in weekly check-in session to orally explain concepts of the week Oral communication Conceptual clarity Digging deeper Formative Google form Zoom Complete a worksheet with deeper conceptual & contextual questions Retrieval practice Factual Conceptual Procedural Recitation session Formative Zoom Appear on Zoom voluntarily to practice for the oral exam Recorded sessions shared with the class Oral communication Retrieval practice Factual Conceptual Procedural Oral exam Objective test Benchmark (monthly) Summative (End-of- term) Zoom Google form Complete online objective test & appear for a 20- minute oral exam Oral communication Retrieval practice Problem solving Conceptual Clarity Factual Conceptual Procedural Use of Software to Develop Learning Opportunities I experimented with a variety of software applications to develop and deploy learning opportunities: The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 71 • To develop explainer videos in a sketch-note style, I used a software application called Explain Everything Basics (Explain Everything, 2020) which allows you to create animated videos and upload them directly to YouTube. • To create real-time questions that appear at certain checkpoints of the YouTube video, I uploaded the video to a software application called PlayPosit (PlayPosit, 2013) and used this software to create objective type questions to check basic understanding. • To gamify learning vocabulary, labeling, and basic facts, I created quizzes in a software application called Quizlet (Quizlet, 2005) and added it to a class that I had created. Students sign into this Quizlet class at the beginning of the semester to access their homework. • To demonstrate how sketch-notes can help create mental representations in biology and provide students with an open educational resource, I created a set of notes for the major concepts and clinical scenarios on a software application called GoodNotes (GoodNotes, 2011), which makes it easy to share with students. • To create a quiz with deeper conceptual and contextual questions, I used Google forms (Google, 2008). • To conduct small group recitation sessions, I used the Zoom videoconferencing platform (Zoom, 2020). Designing Deliberate Practice I incorporated deliberate practice principles into the weekly formative assessments by doing the following: (a) allowing students unlimited trials on all formative assessments; (b) providing immediate feedback wherever appropriate via apps; (c) promoting mental representations by writing to learn (notetaking and making sketches); and (d) practicing oral skills by “teaching” concepts. For all six weekly assessments, students were only given credit if they demonstrated proficiency, allowing for slow learning. Assessing for Mastery Students were provided with the oral exam questions, notes with answers, mini- lecture videos, and recitation sessions prior to each oral exam. There was full transparency. I used a score of 90% and above on the oral exam as an indicator of mastery and at least 80% on the oral exam as an indicator of proficiency. My benchmark for students achieving mastery in this course was set at 70% meaning that I expected at least 70% of the students to achieve mastery. For achieving proficiency, I set the benchmark at 80% meaning that at least 80% of the students would achieve proficiency. Each oral exam session, which lasted 20–30 minutes was scored using a rubric (Table 2). The students were required to have their camera on and were not allowed to refer to notes during the oral exam. Each student received a completed rubric within a few hours of taking the oral exam. The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 72 Maintaining a Nurturing Learning Environment I designed a caring learning environment that focused on three aspects: communication, connection, and intervention. To keep communication channels open, I used the mobile messaging application called Remind (Remind, 2014). To forge a personal connection with each student, I included 15-minute one-on-one weekly check-in on Zoom. Students were allowed to pick a time slot that suited their schedule from a wide range of choices that included evenings and weekends. These sessions were useful in tracking progress, clarifying course information, evaluating conceptual clarity via the “teach it” exercise, and getting to know each other. Throughout this process, I was guided by the threshold concept theory, which allowed me to observe the student learning experience with empathy as they journeyed from pre-liminal to liminal to post-liminal stages (see Figure 1). Following along the rules of mastery learning, students were allowed multiple attempts on the oral exam. If a student’s performance on weekly check-ins and oral exam sessions indicated that they needed intervention, additional online practice sessions were scheduled. Two weeks prior to each oral exam, students were provided with a detailed study guide that was broken up into three parts: part A consisted of a list of “essay” type questions; part B consisted of a collection of figures and graphs; and part C consisted of a long list of questions dealing with the basics reminiscent of the oral exam format (see Table 2). As a way of practicing for the oral exam, several 30- minute small group online recitation sessions were arranged during the days leading up to the exam that students attended on a voluntary basis. During these sessions, students took turns to practice answering each question with their peers helping them when they got stuck. Recordings from these sessions were made available to the entire class. Table 2 Sample Rubric for Oral Exam Criteria Concept/question Unfamiliar (60 and below) Familiar (61-79) Proficient (80-89) Master (90- 100) Comfort level (Ease of explanation) Nodal/Non-nodal action potentials Blood Pressure Cardiac Output Circulatory shock Edema Completeness (Right amount of detail to support big picture) Nodal/Non-nodal action potentials Blood Pressure Cardiac Output Circulatory shock Edema Nodal/Non-nodal action potentials The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 73 Criteria Concept/question Unfamiliar (60 and below) Familiar (61-79) Proficient (80-89) Master (90- 100) Conceptual understanding (Cause and effect relationships evident) Blood Pressure Cardiac Output Circulatory shock Edema Logical Reasoning (Can think on feet) Walk me through the Graphs Part B Basic Knowledge (Strong fundamentals) Let’s check your Basics Part C Results A major discovery of this study was that it is possible to assess for mastery using the oral exam as the assessment vehicle. The oral exam made thinking visible, allowing me to observe mental flow, fluency, and conceptual clarity. It was also possible to easily assess if the student had prepared for familiarity, proficiency, or mastery (see rubric in Table 2). It turned out that 74% of the students (n = 57) demonstrated mastery and 16% achieved proficiency in the summative oral exam (Figure 3). Altogether 90% of the students achieved proficiency, which was well above the benchmark of 80% for proficiency. Figure 3 Majority of Students Learned for Mastery 74% 16% 10% Achieved Mastery Achieved Proficiency The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 74 As for mastery, the students scored above the benchmark, which was set at 70%. Student perceptions of the oral exam was favorable in general and specifically to the following statements (Figure 4): (S.1) Preparing for and taking the oral exam was a powerful learning experience; (S.2) Taking oral exams helped me realize that I need to focus on understanding concepts rather than memorizing them; (S.3) Over the course of the semester I became more comfortable speaking about physiology; and (S.4) I like this way of testing. Figure 4 Students who Chose “Strongly Agree” or “Agree” to Statements 1–4 Students were generally positive about the oral exam format, process, and outcomes (Figure 5) when responding to the following statements: (S.5) Oral exams helped me to become more aware of my strengths and weaknesses in my approach to learning; (S.6) The oral exam felt like a conversation about physiology with my instructor; (S.7) I left each oral exam knowing my strengths and areas for improvement; (S.8) I felt nervous taking oral exams; and (S.9) The oral exam format was clear to me. Students unanimously strongly agreed to statements about the efficacy of the weekly check-ins and small group sessions (Figure 6). The students responded positively to the following statements: (S.10) The weekly check-ins helped to keep me on track; (S.11) The weekly check-ins helped to know my instructor better and discuss any issues I was having with the class; (S.12) Video recordings of the small group sessions were helpful to me in preparing for oral exams; (S.13) The “Teach it” exercise helped me a lot to master concepts and apply them to clinical scenarios; and (S.14) I am happy that I took this class. 90% 86% 79% 76% 65% 70% 75% 80% 85% 90% 95% Statement 1 Statement 2 Statement 3 Statement 4 The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 75 Figure 5 Students who Chose “Strongly Agree” or “Agree” to Statements 5–9 Figure 6 Students Unanimously Favored the Weekly Check-ins and Small Group Recitations More than 95% of the students responded favorably to the weekly formative assessments. When asked to respond to the statements, “What study tips would you give to your peers on how to approach this exercise for the best results? Describe your unique method of approaching this exercise,” many of them were eager to share their advice with their peers (Table 3). 100% 100% 100% 100% 100% 0% 20% 40% 60% 80% 100% 120% Statement 10 Statement 11 Statement 12 Statement 13 Statement 14 77% 77% 82% 86% 82% 72% 74% 76% 78% 80% 82% 84% 86% 88% Statement 5 Statement 6 Statement 7 Statement 8 Statement 9 The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 76 Discussion The students responded favorably to the organization of the weekly formative assessments, and all students agreed that the repetition of information in weekly assignments and weekly check-ins were very helpful with the oral exams. The amalgamation of intentional content, deliberate practice, and a highly supportive liminal environment had the desired effect on mastery learning (see Figure 2). More importantly, it became apparent that it is possible to assess whether a student is learning for familiarity, proficiency, or mastery based on their oral exam performance (see Table 2 and Figure 3), which is difficult to discern in a more traditional written or objective type exam. The mastery design guided by deliberate practice principles helped student engagement to the extent that students were able to think and speak like physiologists. Table 3 Study tips to Peers on how to Approach an Exercise Formative Assessment Level of Difficulty Study tips to peers on how to approach this exercise Write/Sketch Low • Good to keep specially for any oral exams • Have fun drawing and writing down the notes while you learn them! Play Posit Quiz Moderate • Watch the video at least twice • Focus on the videos first, bypass the questions until u understand. Take Many notes, small details can be important for the next step. Repetition is your friend. Quizlet Quiz Moderate • Take detailed notes of the material provided and go through the flash cards first to learn the material • Do all of them. It helps a lot! Google Quiz Difficult • Don’t get discouraged if you do not get everything right on the first try. Use what you learned from playposit as well as quizlet to complete this. Keep trying until you have a better understanding of what is being presented to you • The google quizzes helped a lot with testing my knowledge on all the information I learned with the quizlet, and playposits. I think they were really beneficial - using your notes from playposit and understanding the concepts will help you complete your google quiz’s successfully Teach it! Difficult • I love “teach it” as it applies to real life scenarios, and you need to understand the material in order to teach it thoroughly which can give clarity on the cause and effects and allows you to start recognizing patterns or even curiosity to solve different clinical scenarios that it might relate to. Viewing mastery learning as a threshold experience was reassuring to me when students would become stuck or discouraged with the process. Students found the process of mastery learning to be challenging, time-consuming, and even nerve The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 77 wrecking, thereby providing support for the troublesome nature of a threshold experience (Land et al., 2005). Most students who successfully emerged from this experience were also transformed by it (see Figure 3). For example, there was the realization that they needed to understand and not just memorize physiological concepts. Nearly all the students (90%) felt that preparing and taking oral exams was a powerful learning experience. Many were surprised that they were able to “talk physiology” to their friends and family members, which was perhaps the most transformational change, making mastery learning a threshold concept (Meyer & Land, 2003). About 90% of the students entered the liminal space. They engaged in deliberate practice hoping to achieve conceptual clarity and demonstrate the ability to think and speak like a physiologist. The majority of the students (74%) stepped over the threshold and achieved mastery, thereby showing that oral exams can motivate students to learn for mastery without raising any red flags regarding academic dishonesty—something that has also been observed by others (Schurgers & Qui, 2022). The construction and maintenance of a warmly supportive learning environment focusing on both esthetic and authentic care (Valenzuela, 1999) paid off with 100% of the students responding favorably to the pedagogy of care model (see Figure 5). They all felt that the weekly check-ins and small group recitation sessions were especially useful. This level of support may have spilled over to their learning experience because even if the deliberate practice was sometimes onerous, the students remained eager to engage at a deeper level and help their peers (Table 3). Conclusion From an instructor’s perspective, creating and maintaining optimum conditions for mastery learning to occur was both challenging and exhilarating. This was indeed a threshold experience. It was troublesome because of the extensive planning involved before the semester began and the laborious progress tracking that occurred throughout the semester. However, to observe student growth over the course of the semester during the weekly check-ins, small group recitations, monthly oral exams and the final oral exam was truly a powerful experience. With the rapid improvements in educational software, it should be possible to streamline progress tracking and simplify the workflow for both the student and the instructor, thereby making mastery learning via oral exams even more attractive. On the flip side, fears of plagiarism arising from the abuse of artificial intelligence can be allayed by a well-planned oral examination. The oral exam, when designed to humanize the learning experience, can create a learning sanctuary for students in an online environment. Conflicts of Interest The author declares that there is no conflict of interest regarding the publication of this article. The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 78 References Akkaraju, S. (2022). Pedagogical love during the age of the pandemic: Formative assessments to the rescue. The Assessment Review, 3(2). https://assessatcuny.commons.gc.cuny.edu/author/shylaja-akkaraju/ Akkaraju, S. (n.d.). Professor Akkaraju [YouTube]. https://www.youtube.com/channel/UC5UdUWw5Zyx7Nu5s_CKn_Ew Akkaraju, S., Atamturktur, S., Broughton, L., & Frazer, T. (2019). Ensuring student success: Using formative assessment as the key to communication and compassion among faculty, students, and staff. New Directions for Community Colleges, 2019 (186): 71–79. doi:10.1002/cc.20358 Carrasco, M. (2022, January 28). Concerns about online cheating decline. Inside Higher Ed. Cho, D. (2005). Lessons of love: Psychoanalysis and teacher/student love. Educational Theory, 55(1), 79–95. Cousin, G. (2006). An introduction to threshold concepts. Planet (17), 4–5. Coyle, D. (2009). The talent code: Greatness isn't born, it's grown, here's how. Bantam Books. Ericsson, A. (2008). Deliberate practice and acquisition of expert performance: A general overview. Academy of Emergency Medicine, 15(11): 988–994. Ericsson, K., & Harwell, K. (2019). Deliberate practice and proposed limits on the effects of practice on the acquisition of expert performance: Why the original definition matters and recommendations for future research. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.02396 Ericsson, A., & Pool, R. (2017). Peak: Secrets from the new science of expertise. First Mariner Books edition. Mariner Books/Houghton Mifflin Harcourt. Explain Everything. (2020). Explain Everything Basics [Mobile explainer video application software]. https://explaineverything.com/explain-everything- basics/ GoodNotes. (2011). Digital note-taking software for iPad [Mobile application software]. https://www.goodnotes.com/ Google Forms. (2008). Google forms [Mobile application software]. https://www.google.com/forms/about/ Joughin, G. (2010). A short guide to oral assessment. Leeds Metropolitan University/University of Wollongong. https://teachlearn.leedsbeckett.ac.uk/- /media/files/clt/a-short-guide-to-oral-assessment-by-gordon-joughin.pdf The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 79 Lambert, L. M., & Felten, P. (2020). Relationship-rich education: How human connections drive success in college. Johns Hopkins University Press. Land, R., Cousin, G., Meyer, J. H. F., & Davies, P. (2005). Threshold concepts and troublesome knowledge: Linkages to ways of thinking and practicing: Implications for course design and evaluation. In C. Rust (Ed.), Improving Student Learning—Diversity and inclusivity (pp. 53–64). Proceedings of the 12th Improving Student Learning Conference. Oxford Centre for Staff and Learning Development (OCSLD),. Loreman, T. (2011). Love as pedagogy. Sense Publishers. Maatta, K., & Uusiautti, S. (2012). Pedagogical authority and pedagogical love: Connected or incompatible? International Journal of Whole Schooling, 8(1), 21–39. Meyer, J. H. F., & Land, R. (2003). Threshold concepts and troublesome knowledge: Linkages to ways of thinking and practicing, In C. Rust (Ed.), Improving Student Learning—Theory and Practice Ten Years On (pp. 412-424). Oxford Centre for Staff and Learning Development (OCSLD). PlayPosit. (2013). Interactive video vlatform [Computer software]. https://go.playposit.com/ Pritts, N. (2020, June 1). Using announcements to give narrative shape to your online course. Faculty Focus. https://www.facultyfocus.com/articles/online- education/online-course-design-and-preparation/using-announcements-to- give-narrative-shape-to-your-online-course/ Quizlet. (2005). Study tool for quiz and flashcard creation [Mobile application software]. https://quizlet.com/ Remind. (2014). Remind school communication [Mobile application software]. http://Remind.com Schurgers, C., & Qui, H. (2022, December 11). Oral exams in support of academic integrity: From catching dishonesty to promoting excellence. Integrity Matters. International Center for Academic Integrity. https://academicintegrity.org/resources/blog/106-2022/december- 2022/417-oral-exams-in-support-of-academic-integrity-from-catching- dishonesty-to-promoting-excellence Sherr, G., Akkaraju, S., & Atamturktur, S. (2019). Nudging students to succeed in a flipped format gateway biology course. Journal of Effective Teaching in Higher Education, 2(2), 57–69. The Oral Exam Journal of Effective Teaching in Higher Education, vol. 6, no. 1 80 Theobold, A. (2021). Oral exams: A more meaningful assessment of students’ understanding. Journal of Statistics and Data Science Education, 29, 156– 159. DOI: 10.1080/26939169.2021.1914527 Torres, C. (2019). Assessment as an act of love. ASCD Education Update, 61(2). Valenzuela, A. (1999). Subtractive schooling: U.S.-Mexican youth and the politics of caring. State University of New York Press. Wilkinson, J., & Kaukko, M. (2020) Educational leading as pedagogical love: The case for refugee education. International Journal of Leadership in Education, 23(1), 70–85. doi: 10.1080/13603124.2019.1629492 Zoom (2011). Zoom video conferencing [Mobile application software]. https://zoom.us/