Microsoft Word - Student-Generated Learning article.docx Published by University of Minnesota Libraries Publishing Student-Generated Learning Objectives and Curricular Innovation Thomas Laux; Mark Stephens, MD; Jennifer Meka, PhD DOI: https://doi.org/10.24926/jrmc.v4i2.3637 Journal of Regional Medical Campuses, Vol. 4, Issue 2 (2021) z.umn.edu/JRMC All work in JRMC is licensed under CC BY-NC Thomas Laux, BA; Medical Student, Penn State College of Medicine, University Park, PA Mark Stephens, MD; Professor of Family and Community Medicine and Professor of Humanities, Penn State College of Medicine, University Park, PA Jennifer Meka, PhD; Director of Medical Education and Educational Research Institute and Assistant Dean for Medical Education, University at Buffalo Jacobs School of Medicine and Biomedical Sciences, Buffalo, NY Corresponding author: Thomas Laux; Address: Suite 308 1850 E. Park Ave., State College, PA 16803; Telephone: 814-689-0008; Email: tlaux@pennstatehealth.psu.edu All work in JRMC is licensed under CC BY-NC Volume 4, Issue 2 (2021) Journal of Regional Medical Campuses Original Report Student-Generated Learning Objectives and Curricular Innovation Thomas Laux; Mark Stephens, MD; Jennifer Meka, PhD Abstract Introduction Recent focus from accrediting bodies emphasizes learning objectives as a means of mapping and standardizing content coverage. While most educational training centers on faculty-derived learning objectives that are geared towards didactic lectures, student-centered teaching modalities like problem-based learning continue to gain popularity. One opportunity is the integration of student-generated learning goals in curriculum development. The educational philosophy at the Penn State College of Medicine University Park Regional Campus centers on discussion-based Inquiry Groups that focus on students’ experiential case learning which leads to student-generated learning objectives. This study examined a student- centered approach to learning objectives. Methods Our quantitative analysis explored student-generated learning objectives during the first-year curriculum. Primary outcome measures included process variables investigating the growth and change of objectives across the year that include Bloom’s taxonomy-based verb scores, verb numbers per session, and learning objective word lengths. Knowledge- based content coverage variables were compared with existing curricular models. Results Student-derived learning objectives changed substantially over the year. Taxonomy scores decreased while the learning objective verb number, word length, and calculated value per session increased significantly. Content and comparator analyses showed that coverage and verb quality met or exceeded existing curricular models. Discussion Student-generated learning objectives are not only plausible and achievable, they also provide distinct pacing and engagement benefits. Our findings serve as a model for student-centered educational innovations. Introduction Learning objectives (LO) are used broadly as a standard tool to clarify educational goals.1 Current regulatory guidelines from the Liaison Committee on Medical Education (LCME) and Accreditation Council for Graduate Medical Education (ACGME) promote LO to establish educational standards for undergraduate and graduate medical education.2-4 Historically, program, course, and session LO are written by faculty. However, with the proliferation of problem-based learning, student-generated LO are increasingly being used in medical, dental, and nursing education.5-7 To date, studies examining LO designed by both students and faculty suggest that student-generated LO are equivalent to faculty-generated LO in terms of breadth and depth of content.8-10 Given the apparent congruence between faculty and student-derived LO, a gap in the literature exists when examining the impact of student-generated LO on student learning processes and outcomes. When designing curricula, content coverage correlates poorly with learner retention and usable knowledge.11 As such, goals beyond maximizing content coverage must be considered. Therefore, LO are vital to instructional design. LO help identify the specific, measurable competencies to be achieved while also providing guidance as to the level of depth and type of transfer that is anticipated at any given stage of the learning process. Promoting long-term knowledge retention and Journal of Regional Medical Campuses, Vol. 4, Issue 2 Original Report critical thinking skills helps prepare learners for current and future professional practice. Experiential learning provides opportunities for context-specific learning and has long-term knowledge retention benefits over traditional lectures.12 Strategies such as distributed practice, interleaving, and desirable difficulty also help to promote learner retention.13 With this background in mind, in 2016 the Penn State University College of Medicine (PSCOM) sought a curriculum redesign for a new regional medical campus. From the outset, the curriculum needed to meet LCME standards for content coverage, while simultaneously creating opportunities to promote long-term retention, experiential learning, and critical thinking skills.12,13 Student-generated LO stemming from authentic clinical experiences was agreed upon as the bridging mechanism to meet these goals. Using the constructs of design thinking from the engineering realm, PSCOM partnered with 5 medical students who deferred admission for one year—Medical Student Design Partners—to pilot curricular strategies to build the PSCOM University Park Regional Campus (UPRC).14 Respecting the unique design of this curriculum, the present study explores process and content changes related student-generated LO across the first year of medical school. We predicted that LO verbs per session would change in quantity and quality throughout the year to fit growing learner content capacity, and that overall verb quality would be comparable to faculty- derived samples. We also hypothesized that our class sessions would be as good or better than comparator faculty-driven sessions from a content coverage perspective. Methods The Inquiry Group Cycle This study examines the inaugural UPRC class during the 2017-2018 academic year. Institutional Review Board reviewed and exempted the study prior to data analysis. The UPRC first-year curriculum consists of 2 complementary semester-long courses that utilize student-derived LO based on the problem-based learning model.17 Twelve incoming students were divided into two 6-person inquiry groups (IQ). Students were immersed into local primary care practices for several clinical sessions each week (Figure 1). Students acclimated to the practices by initially working with front desk staff to learn about clinical workflow and gain health systems knowledge. They progressed to working with nursing staff obtaining vital signs and helping to guide patients to examination rooms. Finally, students progressed to more traditional clinical roles: reviewing patient information from the electronic medical record, conducting a history and physical, and then working with an assigned preceptor to develop the assessment and plan for each patient. By the end of the first year, each student engaged in approximately 100 clinical half-days. Figure 1: The Inquiry Group (IQ) Weekly Cycle This immersive experience was designed to provide students with authentic patient narratives to bring to IQ each week to share with their peers. Students brought these patient-based narratives to class to generate LO for weekly study. On Mondays, students prepared a summary narrative from a patient encounter of their choosing.18 After each student presented their selected case, the IQ group voted to select 2 cases for further study to discuss on Wednesday and Friday of that week. Using the PSCOM 4 pillars model—biomedical sciences, clinical sciences, health systems sciences, and health humanities—students created LO considering each pillar for each of the 2 cases.19 The LO creation was student- led. Experienced faculty helped to facilitate the process using Bloom’s Taxonomy as a shared framework, and while taxonomic charts were posted in classrooms no explicit Bloom’s-based training was given for students. Students then used the co-created LO as the roadmap for their preparations, reconvening on Wednesday and Friday to discuss each case at a much deeper level. The selected cases and associated LO were designed entirely by the students based on their clinic experiences. Drawing on contextual learning theory, this intentionally asynchronous model of coursework leveraged in vivo experiential clinical learning to maximize student engagement and ‘ownership’ of both content and process to enhance transfer and retention. The IQ design was intended to promote distributed practice and interleaving relative to traditional coursework.13 Tuesday-Friday: LO-guided Independent Study for IQ Sessions Wednesday: IQ Session Case 1 Friday: IQ Session Case 2 Previous Monday-Friday: Primary Care Clinical Immersions Inspire Cases Monday: Case Presentations, Selection, and Student-Generated Learning Objectives Journal of Regional Medical Campuses, Vol. 4, Issue 2 Original Report Process Variables—LO Verbs and Bloom’s Taxonomy Process variables illustrate the quality of LO and their value to learner thinking and development in the class session. Process variables including average LO score, number of LO verbs per IQ session, total LO value per IQ session, and LO word count analyzed the structure of LO over time. As part of each Monday IQ session, there were 3 to 7 student-generated LO per session. Some LO included multiple verbs. For example, in a session on lung cancer one LO stated “Review the literature on the relationship between socioeconomic status and health- care delay and treatment. Develop medical and/or public health strategies which might respond to the evidence you discover in the literature”. Each of these 2 distinct verbs, “review” and “develop”, comprised an individual LO-related element for the purposes of analysis. Once the number and distinct nature of LO verbs were established for a session, each verb was individually scored using the Bloom’s Taxonomy of Measurable Verbs.20 This corresponds to a score of 1 to 6 for each verb along the progressive continuum of knowledge, comprehension, application, analysis, synthesis, and evaluation. LO and verbs were scored by one primary rater using the standardized scoring system, and were reviewed by the 2 other authors.20 A small number of verbs from the student-generated LO were not readily mapped to the standardized scoring system, and in these instances the rater designated their best attempt at an appropriate scoring relative to similar standardized verbs; for example, “simulate” score as 5 due to its semantic similarity to the standardized 5 of “roleplay”. After each individual verb was scored, the mean verb scores associated with each individual session were calculated to determine an average LO score for the individual IQ session. In addition, the average LO score was multiplied by the number of verbs in the individual session to determine a total LO value for each session in order to measure verb quantity as well as quality. The average word length for each LO was also calculated. Content Variables—Are We Covering Enough Material? Content variables illustrate the factual medical knowledge discussed in the class session. At the conclusion of each IQ, faculty facilitators mapped session content to a list of 10 competencies and 23 unique sub- competencies students required for graduation.21 Session content was also mapped to a list of 222 unique core content areas associated with USMLE core content.22 Session content was also mapped to the UP50. The UP50 consists of 50 core topics critical to “pre- clinical” medical education such as headache, abdominal pain, and stroke derived from iterative consultation with multiple curriculum innovators in the US and Canada.23,24 Thus given our 12 month first-year coursework comparing to a typical 18 to 24 month “pre-clinical” curriculum, we would expect half to two-thirds of the UP50 should be covered as a baseline. Data Analysis To assess LO changes throughout the year, we conducted a quantitative analysis for each process variable individually to determine LO growth and change over time. The same was done for each content variable to determine if our student-guided sessions were on track from a content coverage perspective. Data for each variable in the first half of the academic year was compared to data from the second half of the academic year with a 2-sample t-test assuming unequal variances. For pattern analysis, trendlines of best fit were calculated along with R2 values using individual values for each variable, weekly averages for each variable, and monthly averages for each variable. The total UP50 count and interleaving topic count for the year were used to compare to expert-driven hypotheses about medical school pre-clinical curricular content coverage. The overall percentage of “high quality” (scoring 4, 5, or 6) and “low quality” (scoring 1, 2, or 3 on Bloom’s taxonomic list) LO was compared to baseline samples from the medical and nursing literature.20,25-27 Year-long content variable counts were also compared with those mapped from the PSCOM Hershey Campus (HC) as a convenience sample. Results Over the course of the year there were 67 IQ sessions. Thirty-eight sessions took place during the fall of the academic year and 29 during the spring. From these 67 sessions, there were a total of 261 LO containing 425 verbs for analysis. Process Variables The average LO score per session decreased significantly (p=0.0073) from fall to spring. The average verb scores most approximated a second-degree polynomial line-of- best-fit with R2=0.25 (Figure 2). The number of learning objective verbs per session increased significantly throughout the year (p<0.001). Verbs per objective increased significantly as the year progressed. Journal of Regional Medical Campuses, Vol. 4, Issue 2 Original Report Figure 2: UPRC Average LO Score Per Session Total LO value per session was calculated by multiplying average verb scores for the session by the number of verbs in that session. The increase in total LO value per session significantly increased (p=0.0081; two-tailed t- test) from fall to spring. While the average LO score followed a decreasing polynomial curve, the number of LO increased to compensate, leading to an increase in the total LO value through the year. Due to high intersession variability, weekly and monthly data were also plotted, showing linear trendlines with R2 values of 0.37 and 0.69 respectively (Figure 3). Figure 3: UPRC Monthly Total LO Value Per Session The average LO word count per session increased significantly from fall to spring (p<0.001; two-tailed t- test). Content Variables Sub-competency count per session increased significantly from fall to spring semester (p=0.010). Due to high intersession variability, weekly and monthly data were also plotted, showing linear trendlines with R2 values of 0.23 and 0.53, respectively. The increase in core content count per session was significant (p=0.0094) when comparing the first and second halves of the year. Given high intersession variability, weekly and monthly data were plotted with linear trendlines showing R2 values of 0.22 and 0.57 respectively. The UP50 Count per session was not significant (p=0.70) when comparing the 2 semesters. Generalizability To explore whether our students were writing LO of significantly lower quality than other curricula, we compared our data with 3 studies utilizing similar coding systems based on Bloom’s taxonomy.25-27 In these studies, course outcomes objectives and faculty session questioning—the closest surrogates for LO available in the literature—were faculty-generated rather than student-generated. Results from these studies showed that 79% of course outcome objectives in the medical school study and 69% and 91% of faculty session questions in the nursing school studies were of lower quality—scoring 1, 2, or 3 out of 6 on their Bloom’s scales. By comparison, only 54% of our student- generated learning objectives rated as lower quality. The 67 IQ sessions covered 40 unique UP50 topics, 23 of which were also addressed at least twice. Thus, 80% of the most fundamental “pre-clinical” content areas were addressed during the first year using student-generated LO. More than half of those topics had an interleaving opportunity during a follow-up session. To demonstrate proof of concept and comparison, we used the same sub-competency and core content mapping variables to compare our data with that of the HC. A 2-sample T-test revealed the UPRC monthly sub-competency average (p<0.0001) and the UPRC monthly core content (p=0.00011) were significantly higher than the more faculty-guided HC. The UPRC sample shows increases over the year in sub-competency and core content counts—linear-fit R2 values of 0.53 and 0.57, respectively—unlike the relatively flat HC comparisons (Figure 4). y = -0.0084x + 3.5796 R² = 0.0481 y = 4E-05x3 - 0.0052x2 + 0.1606x + 2.3318 R² = 0.2545 0 1 2 3 4 5 0 20 40 60 80 Average LO Score Per Session y = 1.0154x + 15.213 R² = 0.6929 0 10 20 30 0 2 4 6 8 10 12 Monthly Total LO Value Per Session Journal of Regional Medical Campuses, Vol. 4, Issue 2 Original Report Figure 4: Sub-competency and Core Content Per Session by Month for UPRC vs. HC Comparator Discussion This study reviews outcomes from the first-year of a new curriculum emphasizing the use of student-generated LO based on immersive clinical experiences. We hypothesized that our content variables would score similarly if not favorably compared to our more faculty- guided main campus and that our process variables would show elements of growth and change in student learning throughout the year. Our results largely validate the principles of process improvement and content comparability. As expected, student-generated LO changed significantly over the course of the academic year. While the average LO scores decreased on our Bloom’s-based rating scale, the number of objective verbs per session, the word length of objectives, and the total value of objectives per session improved significantly. The quadratic trend of average LO scores shows that our students initially improved in the taxonomic quality of verbs before ultimately shifting towards a preference for quantity over quality. This suggests students may be taking ownership of their learning process, concurrently adapting the LO process to maximize clarity and efficiency in terms of course preparation both for the sake of the IQ group and its broader week of independent studying. Process Variables With regard to Bloom’s taxonomy, average LO verb scores decreased through the year. At the beginning of the year, there was limited a priori training for students in terms of selecting LO verbs. While students regularly engaged with Bloom’s taxonomy as part of the LO creation process, there was no explicit curricular goal to maximize verb quality. The mid-year peak in LO score may be due to relative inexperience with LO creation early on, followed by improved student engagement mid-year towards a perceived goal of maximizing Bloom’s hierarchy. This later shifted perhaps to a goal of designing LO for effective study and learning rather than linguistic embellishment. As the year progressed, students gained comfort and confidence generating LO with multiple verbs to clarify key concepts and maximize studying preparation for class discussions. Over time LO value scores increased, which could be explained by a priority shift among the students who eventually focused less on the taxonomic quality of individual verbs and more on the quantity and fit of each verb to clarify and facilitate their independent studies. For example, compare the fall diarrhea clinical science LO “Distinguish between the most common types of diarrhea” with the spring eating disorder clinical science LO “Define the DSM V criteria for Anorexia Nervosa, Bulimia Nervosa, Binge Eating Disorder, and Eating Disorder Not Otherwise Specified. Identify risk factors for each disorder, clinical workups (including physical examination findings), and management for patients (focusing on healthy weight gain and including referral criteria)”. This spring LO is neither pithy nor does it contain high quality verbs—“define” and “identify” score 1 and 2 out of 6, respectively. It does, however, paint a specific roadmap for students to study in their topics of group interest. Given that verb score didn’t significantly increase while verb value did, this reveals a powerful increase in verb number per case over the year, indirectly implying that students over time more readily engaged with the LO development process and likely with their overarching studies. The evolution of average verb score and value, including the polynomial transition throughout the year in verb score, illustrate potential scaffolding for learners with regard to their perceptions of self-generated LO and the underlying content they are seeking to understand. More broadly, our results question the specific goal of Bloom’s taxonomy in scientific thinking. Whereas Bloom’s is generally discussed as a hierarchy with the goal of maximizing higher-order questions and thoughts, Journal of Regional Medical Campuses, Vol. 4, Issue 2 Original Report perhaps a blending of verb order scoring, or even something more complex like the polynomial findings of our study, could be beneficial for the ultimate goal of learning.28 Similarly, the increase in LO word counts reveal possible changes in LO purpose. As an example, the early year LO “Compose 2 potential open-ended questions to build shared decision making in our case of abdominal pain” scores highly, fulfills operationality, and works beautifully for packaged lectures. Such an LO does not, however, provide clear guidance for students to thoughtfully prepare for an upcoming IQ session. LO that work to summarize and focus a 50-minute didactic lecture do little to help students prepare for a robust 4-hour inquiry-based discussion and hours more of independent study. For our example above, consider the potential addition, “…demonstrating an understanding of PQRST pain assessment, acute abdomen definitions, and medical as well as surgical treatment options”. In such settings, longer wording with higher degrees of clarity and nuance prove to be an asset. The interplay between the polynomial shift in average LO score alongside the increases in LO value and word length suggest a transformation in student mental models of their content learning and the curricular preparation mechanisms driving that learning. This can be seen with objectives like the spring medical humanities LO during a diabetic ketoacidosis case, “Revisit biases by outlining how this patient’s care may have changed given his label of ‘prisoner.’ What are some strategies that we can use (or promote for others) to avoid making these assumptions and ultimately providing this patient with equal care?” While this LO had relatively low-scoring verbs—"revisit”, “outline”, and “what” rate as 2, 1, and 1 out of 6, respectively—the LO is quite long at 41 words and elicits numerous streams of potential academic exploration, from a textbook review of cognitive biases to a literature review of strategies for working with vulnerable patient populations to a potential introspective dive into the ethics and realities of working with incarcerated folks as a healthcare worker. LO like these show learners designing multi-level and multi-modality objectives far more nuanced than they themselves did earlier in their training and likely more engaging and nuanced for them in that moment than their faculty could have written for them. Content Variables From a content mapping standpoint, sub-competency and core content counts per session increased from fall to spring. Since the content variable mapping task was completed at the end of each session by clinical faculty rather than the students themselves, changes through the year would not be due to operator input skill. The increase in sub-competency count more likely reflects skill changes of students as they created their LO. Early in the year, simpler topics covering a smaller number of sub-competencies fully sated student appetites for study and IQ preparation. As the year progressed, students wrote better LO—exhibited by process variables data— and were better able to weave in a wider swath of content, representing more robust integration of content domains. Interestingly, UP50 counts per session did not increase significantly, likely due to the individual case- based focus of each IQ session which tended to correspond to a single case-content theme. Further, the concurrent increase in clinical responsibilities of our trainees throughout their first year may be driving some of the increases in these content-related variables; the symbiosis between clinical and classroom experience may have impacted the LO they were primed to develop. Generalizability Our process variable data shows that while students did not generate higher scoring LO as the academic year progressed, they did improve with regard to the number, length, and total LO value over time. Our comparator data from the medical and nursing education literature show that despite decreasing through the year, our student-generated LO verb quality still has a much higher percentage of higher-order Bloom’s verbs than faculty-derived objectives from these external samples.25- 28 This suggests that our student-generated LO were at least as rigorous as those in peer curricula. Respecting the concern that a student-guided approach to learning could miss higher yield topics and leave students underprepared for clerkships, our UP50 data showed that there was at least as much critical content exposure (80%) as we had envisioned (50-66%). Our UPRC vs. HC sub-competency and core content comparisons revealed that while the starting point for the 2 campuses was relatively similar on both variables, UPRC sub-competency and core content counts increased over time whereas HC counts decreased over time (Figure 4). While student bodies of the 2 campuses were not explicitly matched pairs, they compare similarly demographically and these findings are more likely due to the student-generated LO process at UPRC. These results give further reassurance that our student-based Journal of Regional Medical Campuses, Vol. 4, Issue 2 Original Report approach to LO generation during the MS1 year is not inferior to other curricula, and may instead be of benefit. Broader Takeaways LO created for faculty lectures tend to focus on knowledge transfer. As such, these LO benefit from concise, high-quality objectives and shape how educators are taught to think about writing LO in general. Our results highlight the importance of context in the LO creation process. Students may pivot away from the traditional view of LO to develop their own brand of longer, more varied objectives to guide independent explorations. Our comparator studies demonstrate that while our students did not progressively improve verb quality, the student- generated LO were still of higher quality than external faculty-generated samples. As with many educational processes, there is the pervasive need to carefully balance quality with quantity when it comes to creating effective LO.25-27 Our students designed LO at the outset of each week to guide their research and course preparation for the week to come. As students grew in experience and confidence as the year progressed, they crafted LO to fit their interests and their cognitive load capacity. Students alter linguistic hierarchy to favor detailed comparisons, differential diagnoses, and illness scripts. Students begin to think more like clinicians, using the learning objectives as a tool for multifaceted, specific, and clinically- applicable preparation rather than a more general session goal. For example, the late spring LO, “Demonstrate how to administer an NIH and Cincinnati stroke scale, explain how you would interpret the results, and defend how the results should inform treatment” not only meets the operationality of any LO, but also guides hours of independent study and gears learners towards practical knowledge for their upcoming clerkships. While process growth is a main target of our curriculum, it falls flat if content delivery is inadequate. As such, our goal of being similar if not favorable in terms of content coverage was met. The UP50 count over the year met our expectations. Internal measures of sub-competency and core content progressed significantly, and our HC content comparison showed favorably. Overall, our data suggest that the UPRC curricular innovations resulted in process as well as content growth. There are several study limitations where future research could guide further conversation. Qualitative survey data from the student and faculty perspective suggesting that there was intentionality with our student-guided process and content quantitative changes would be beneficial. Having information about the perceived usefulness of each LO after IQ sessions would also help identify which process or content variables were most related. Further, our UPRC vs. HC and literature-derived verb quality comparisons are imperfect proxies rather than pure control groups, and a robust experimental study could strengthen our conclusions. Our curriculum was created using the principles of design thinking, an intentional and iterative process of meeting with stakeholders, constructing and trialing products, and obtaining feedback to repeat the cycle.29 Positive outcomes for the IQ process and early immersive clinical experiences might be of benefit for other programs interested in implementing similar curricular dynamics. As we all find ourselves in the midst of pandemic-related curricular transition and a newfound focus on virtual learning, programs may find themselves looking for curricular adaptation. We believe pursuing student-generated LO may be one such curricular innovation that can spark engagement and learning in these unprecedented times. Further, with the upcoming changes to a pass/fail USMLE Step 1 exam for our students, we believe the such student created LOs allows for a more learning-centered orientation which aligns with a pass-fail exam. In our experience, student- generated LO addressed sufficient and appropriate content. Promoting contextual, learner-driven session framing may help to support retention and transfer of knowledge and skills to the clinical environment. References 1. Mager RF. Preparing Instructional Objectives. 3rd ed. London, UK: Atlantic Books; 1998. 2. Liaison Committee on Medical Education. LCME Functions and structure of a medical school: standards and accreditation of medical education programs leading to the MD degree 2021-2022. LCME. https://lcme.org/publications/#Standards. Published March 2020. Accessed August 2, 2020. 3. Accreditation Council for Graduate Medical Education. Milestones by Specialty. ACGME. https://www.acgme.org/What-We- Do/Accreditation/Milestones/Milestones-by- Specialty. Published November, 2020. Accessed December 3, 2020. Journal of Regional Medical Campuses, Vol. 4, Issue 2 Original Report 4. Accreditation Council for Graduate Medical Education. ACGME Outcomes Project Toolbox of Assessment Methods. ACGME. https://www.slideshare.net/pedgishih/toolbox- of-acgme-assessment-methods. Published September 2000. Accessed August 2, 2020. 5. Norman GR, Schmidt HG. The psychological basis of problem-based learning: a review of the evidence. Academic Medicine. 1992;67(9):557- 565. 6. Fincham AG, Shuler CF. The changing face of dental education: the impact of PBL. J Dent Educ. 2001;65(5):406–21. 7. Barrows HS. The essentials of problem-based learning. J Dent Educ. 1998;62(9):630–633. 8. Abdul Ghaffar Al-Shaibani TA, Sachs-Robertson A, Al Shazali HO, Sequeira RP, Hamdy H, Al- Roomi, K. Student generated learning objectives: extent of congruence with faculty set objectives and factors influencing their generation. Education for Health. 2003;16(2):189– 197. 9. Dolmans DH, Gijselaers, WH, Schmidt HG, Van der Meer SB. Problem effectiveness in a course using problem-based learning. Academic Medicine. 1993;68:207-213. 10. Kennedy SW, Wilkerson K. Student’s perception of problem-based learning. Academic Medicine. 1993;68:S31-S33. 11. Swanwick T, Forrest K, O’Brien B. Understanding Medical Education: Evidence, Theory, and Practice. 3rd ed. Hoboken, NJ: Wiley-Blackwell; 2018. 12. Alluri RK, Tsing P, Lee E, Napolitano J. A randomized controlled trial of high-fidelity simulation versus lecture-based education in preclinical medical students. Medical Teacher. 2016;38:404-409. 13. Brown P, Roedinger III HL, McDaniel M. Make It Stick. Cambridge, MA: Harvard University Press; 2014. 14. Goldberg DE, Somerville M, Whitney C. A Whole New Engineer. Douglas, IL: ThreeJoy Associates; 2014. 15. Clements DH, Sarama J. Learning trajectories in mathematics education. Mathematical Thinking and Learning. 2004;6(2):81-89. 16. Lave J, Wenger E. Situated Learning: Legitimate Peripheral Participation. Cambridge, UK: University of Cambridge Press; 1991. 17. Neville AJ. Problem-based learning and medical education forty years on: a review of its effects on knowledge and clinical performance. Medical Principles and Practice. 2009;18:1-9. 18. Tange HJ, Hasman A, de Vries Robbe PF, Schouten HC. Medical narratives in electronic medical records. International Journal of Medical Informatics. 1997;46:7-29. 19. Penn State College of Medicine. Our vision. Penn State College of Medicine. https://med.psu.edu/md. Accessed August 2, 2020. 20. Utica College. Blooms taxonomy of measurable verbs. Utica College. https://www.utica.edu/academic/Assessment/ne w/Blooms%20Taxonomy%20-%20Best.pdf. Accessed August 2, 2020. 21. Penn State College of Medicine. Competencies and subcompetencies for graduation. Penn State College of Medicine. https://students.med.psu.edu/md- students/medical-student-competencies-and- subcompetencies-for-graduation/. Published August 24, 2017. Accessed August 2, 2020. 22. Penn State College of Medicine Core competency list [Internal Communication]. 23. Penn State College of Medicine UP50 [Internal Communication]. 24. Phillips R, Randhawa AP, Randhawa V. Blackbook. 12th ed. Calgary, CAN: University of Calgary; 2020. 25. Yeo S. An analysis of verbs used in the course outcomes of outcome-based integrated courses at a medical school based on the taxonomy of educational objectives. Korean J Med Educ. 2019;31(3):261-269. 26. Saeed T, Khan S, Ahmed A, Gul R, Cassum S, Parpio Y. Development of students’ critical thinking: the educators’ ability to use questioning skills in the baccalaureate programmes in nursing in Pakistan. Journal of Pakistan Medical Association. 2012;63(3):200- 203. 27. Sellappah S, Hussey T, Blackmore AM, McMurray A. The use of questioning strategies by clinical teachers. Journal of Advanced Nursing. 1998;28(1):142-148. 28. Lord T, Baviskar, S. Moving students from information recitation to information understanding: exploiting Bloom’s taxonomy in Journal of Regional Medical Campuses, Vol. 4, Issue 2 Original Report creating science questions. Journal of College Science Teaching. 2007;36(5), 40-44. 29. Thoring K, Muller RM. Understanding design thinking: a process model based on method engineering. International Conference on Engineering and Product Design Education. 2011;493-498.