ORIGINAL ARTICLE Nov 2016. Christian Journal for Global Health, 3(2): 38-49. Outcomes of a faculty development conference in Mongolia Deborah A Kroeker a , Anne Walling b a MD, Associate Professor, Department of Pediatrics, University of Kansas School of Medicine, USA b MB, ChB, FFPHM, Associate Dean for Faculty Affairs, KU School of Medicine, Wichita, Professor of Family and Community Medicine, USA Abstract Background: Medical Education International (MEI), an organization that provides faculty development to medical educators in developing countries, wanted information on the program effectiveness of its faculty development conferences. Objectives: To assess the outcomes of an MEI faculty development conference in Mongolia on the knowledge, confidence in applying new skills, and attitudes of participants. Methods: A retrospective pretest survey of participants was used to assess the outcomes of a 3-day faculty development conference given twice at the Mongolian National University of Medical Sciences. The survey assessed participant views on their ability to meet the objectives of the conference, the perceived overall value of the conference, and their suggestions for improvements in future MEI conferences. Results: Twenty participants (65%) completed surveys. Participants reported significant changes in agreement with their ability to meet the objectives of the conference in all of the pre-post measures (pre-post p<0.001). The value of attending the conference was ranked at a mean score of 4.05 on a Likert scale from 1 to 5, with 1 indicating “Strongly Disagree” and 5 “Strongly Agree.” Conference attendees indicated interest in additional training on more advanced topics. Conclusion: Overall, the findings indicate that conference attendees gained knowledge and confidence in applying new skills and valued the training received from a faculty development conference led by physicians from the USA. Further research is needed to determine long-term impact on residency education in Mongolia. 39 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. Introduction The last three decades have brought a significant increase in the number and type of professional development programs for medical school faculty members, but the outcomes of many of those programs are not well studied. 1 This leaves faculty members and educational leaders without strong evidence to guide decisions about optimal faculty development activities for institutions and individuals. Increasing faculty and university time and financial constraints require that resources are used in the most productive manner possible. Medical Education International (MEI, online at https://cmda.org/missions/detail/mei) is an organization that provides conferences on request to faculty members in developing countries on clinical or faculty development topics. The Dean of the Mongolian National University of Medical Sciences in Mongolia, a developing country in central Asia, requested a conference on faculty development. After discussion about content and logistics, MEI agreed to provide this university with two identical faculty development conferences, each lasting three days, on residency education. The outcomes of MEI conferences have not been previously studied, and this project aimed to assess conference outcomes. Mongolian medical education has experienced significant change in recent years. Mongolia was a communist country until the dissolution of the USSR in 1990, with centralized control over education and clinical services. Since then, the country has been rapidly advancing technologically and in the depth and breadth of its clinical services. Many new, high-technology hospitals have opened. To keep up with these new opportunities and challenges, higher education has expanded and the number of faculty members in Mongolian universities doubled in ten years between 1997 and 2007 to 6,818 full-time faculty. 2 Mongolia has a long history of having medical schools. However, the first residency training programs were not started until the mid- 1990s. The duration of residency training for most specialties has increased from 18 to 24 months. Fellowship programs have also been started, although they can be as short as 6 months. Because of the rapid expansion in medical education, many new faculty members have little experience in the faculty role. All faculty members, regardless of country, require faculty development to meet the needs of the changing medical and educational environ- ments. 3 For Mongolia, the dramatic changes and growth make faculty development for residency program directors and other educational leaders imperative. In general, active learning strategies are less commonly used in Asian schools, and courses are often lecture-based. 4 Formal faculty develop- ment in teaching and programs to promote strong educational leadership are also more limited in Asia than in westernized countries. The leaders of the Mongolian National University of Medical Sciences (MNUMS) have identified the improvement of residency training programs as a goal. Faculty development is essential to achieve this priority. To date, faculty development at MNUMS has been provided through sabbaticals, grants, awards, mentoring, visits to observe in other countries, and workshops. 2 Staff and faculty members of MNUMS report that many international visitors from diverse groups travel to Mongolia to deliver conferences on clinical topics but none have addressed faculty development in education. Previous studies have addressed the outcomes of faculty development conferences, predominantly in westernized countries. Outcomes have been assessed mainly by changes in faculty attitudes towards teaching, increases in knowledge, and improvement in teaching skills and behaviors. 5 The outcomes of one conference were assessed at a higher level on Kirkpatrick’s levels of training evaluation by identifying the percentage of participants who successfully implemented faculty development projects at their home institutions. 6 Little information is available on the outcomes of faculty development conferences in developing countries, especially those using a https://cmda.org/missions/detail/mei 40 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. program conducted by individuals from Western countries. A PubMed literature search identified no information on the utility of such faculty develop- ment conferences in Mongolia. One study by Wong and Fang describes the outcomes of a Western-based faculty development seminar in a developing country in Asia. 7 This study provided the model for the assessment of this Mongolian MEI conference. Further research on this topic will provide useful information for the development of future MEI programs in developing countries and provide information for other educational organizations that offer similar conferences. This study seeks to answer the broad research question, “what are the outcomes of the MEI faculty development conferences on the knowledge, confidence in applying new skills, and changes in attitudes of the faculty participants?” The study of conference attendees identifies the extent to which they agree conference objectives were met, their perceived value of the conference, and their suggested areas for improvement. Faculty development can be defined as “a planned program designed to prepare institutions and faculty members for their various roles and to improve an individual’s knowledge and skills in the areas of teaching, research [and] administration.” 8 As learner requirements and innovative techniques for teaching, evaluation, and assessment evolve, faculty members must be well-prepared in both the theory and application of appropriate methods to effectively guide and evaluate learners. 3 According to the Accreditation Council of Graduate Medical Education (ACGME), program evaluation is defined as the ‘‘systematic collection and analysis of information related to the design, implementation, and outcomes of a resident education program, for the purpose of monitoring and improving the quality and effectiveness of the program.’’ 9 The hypothesis is that this three-day, MEI faculty development conference will enable participants to gain knowledge and confidence in developing residency training programs, enhance positive attitudes towards educational techniques, and report that time at the conference was valuable. Methods The conference presenters were volunteers selected by MEI from a list of board-eligible or board-certified physicians from the USA who had indicated interest in presenting at conferences and whose areas of expertise corresponded to the needs identified by the host organization (MNUMS). The team presenters for this Mongolian conference included an academic otolaryngology physician who served as an international residency program director, an academic psychiatrist who was previously a residency program director, an associate program director of a pediatric residency program, a retired academic general surgeon with expertise in teaching in mission hospitals, and a hepatobiliary surgeon who was a recent fellowship graduate. The conference took place in the capital city of Mongolia, Ulaanbaatar, at the MNUMS. Faculty attendees were chosen by the deans of the university and consisted of heads of departments in the university and directors of the residency programs in several specialties and subspecialties, including general surgery, otolaryngology, urology, orthopedics, internal medicine, ophthalmology, pediatric gastroenterology, pulmonology, and hematology/oncology. The 3-day conference was given twice in one week to accommodate schedules for two different groups of faculty. In total, about 18 faculty members participated in the first conference and 13 in the second conference. Participants occasionally left and returned to the conference during the day as needed based upon their other professional responsibilities. The conference format (see Table 1) was composed of lectures and small-group workshops in which faculty members worked together to discuss and expand on different conference topics. Following each small group breakout session, the large group reconvened and discussed the ideas from the small groups as a large group. All participants who were 41 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. present at the end of each of the conferences were offered a paper survey to complete anonymously prior to leaving the conference room. Because all faculty participants were offered a survey, the entire conference population was studied; sampling was not used. Table 1: Conference Schedule Time Topic Day 1 1:00pm Introductions and Conference Outline 1:30pm History of Residency Training in Mongolia 2:00pm History of Residency Training in the USA 2:30pm Life as a Resident in the USA 3:00pm Break 3:15pm Lecture on Needs Assessment for Developing a Residency Program 3:45pm Small Group Discussions on the Ideal Residency Program (Duration, assessments, etc.) 5:30pm Report by Small Groups 5:45pm Wrap up and Discussion 6:00pm Adjourn Day 2 1:00pm Lecture on Competencies 1:45pm Small Group Discussions on Competencies 2:15pm Lecture on How to Implement Competencies 3:00pm Small Group Discussion on the Implementation of Competencies in Mongolia 3:30pm Break 3:45pm Lecture on the Administration of a Residency Program/Block Schedules 4:30pm Small Group Work Developing Ideal Block Schedules 5:00pm Lecture on How to Develop Program Goals and Objectives 5:30pm Small Group Development of Program Goals and Objectives 6:00pm Adjourn Day 3 1:00pm Lecture on How to Develop a Competency-Based Curriculum 1:30pm Small Group Development of Rotation Goals and Objectives 2:00pm Lecture on Educational Activities within Residency 2:30pm Small Group Development of an Educational Schedule for Program 3:15pm Break 3:30pm Lecture on Assessments (Formative vs summative, end of training assessments, giving feedback, etc.) 4:30pm Small Group Discussion on giving Feedback in Current Residency Program 5:30pm Report by Small Groups 5:45pm Wrap-up and Evaluation of Conference 6:00pm Adjourn A mixed-methods survey was developed in conjunction with the faculty participants on the team. The data were obtained concurrently and each type of data was given equal weight. The interpreter reviewed the survey and provided feedback prior to use. The survey was written in English, while participants were able to respond in English or Mongolian. The cross-sectional study was based upon the results of a retrospective pretest method that used both a Likert-type scale to provide quantitative information and short answer questions to provide qualitative data. A retrospective pretest consists of a survey given to conference participants at the end of the conference. It asks the participants to rate themselves on how much they knew about specific topics before the conference began and then to rate 42 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. themselves again based upon how much they know at the end of the conference. The difference between the two responses indicates knowledge gains during the conference. 10 This method of assessment can improve the accuracy of the data obtained by a traditional pretest/posttest format. Participants who rate their knowledge using a traditional pretest may be unaware of what they don’t know, and be unable to accurately assess their knowledge level before the conference. In contrast, at the end of the conference, participants can better judge how knowledgeable they were before the conference. 10 Validity of the qualitative results was ensured by relaying the qualitative results in rich, descript- ive language to ensure a realistic portrayal of the situation. Additionally, two reviewers, one with no connection to MEI or the conference, reviewed the data independently. The reviewers analyzed the data until they came to mutual agreement on the themes and results. The principal investigator was open to finding areas upon which to improve for future conferences and open to negative opinions and findings. This provided an opportunity for voices with negative opinions to expand the available data. Grounded theory, a research design in which a theory is generated from the perspectives of participants, was used for analysis of the qualitative data, and reliability of the qualitative data was ensured. 11 The survey reports were rechecked for errors during analysis. Themes were identified and coded. Any disagreements between coders were discussed until consensus was reached. Validity of the quantitative data was also protected. The survey used was developed specifically for this project using the retrospective pretest methodology that has been used for similar studies. The survey questions were previewed by the Mongolian interpreter who provided feedback on the instrument prior to implementation. The survey was also reviewed by multiple physicians with suggestions incorporated into the survey prior to use. The survey (see Appendix A) was printed on a double-sided sheet of paper and given to the participants at the end of each of the three-day conferences. The purpose of the survey was described so the attendees were aware that the sole purpose was to improve future conferences for other attendees; no additional incentives were offered for completion of the survey. The surveys were written in English, but the participants could complete the surveys in English or Mongolian, according to their preferences. The interpreter was available for participant questions or clarification on the meaning of any of the English words or phrases. No identifying information was collected or requested on the surveys to ensure anonymity. Completion of the survey implied consent for inclusion in the study. The attendees left the surveys in the room for collection after they had completed them. The anonymous surveys were placed in a manila envelope for return to the United States for analysis. All answers in Mongolian were translated into English by the conference interpreter who was fluent in both English and Mongolian. The surveys were analyzed by the University of Kansas School of Medicine Office of Research members of the study team. This study was done with the approval of both the MEI director and the leader of the conference. Conference attendees were invited to participate, but participation was optional. While the par- ticipants completed the surveys without benefiting immediately, they may benefit if they attend any future MEI conferences that are improved in response to their feedback. Additionally, the survey results will benefit other future MEI conference attendees in Mongolia and other developing countries. No harmful or identifying information was collected, and the individual surveys will not be shared with those in authority over the participants. Because this was a program evalu- ation rather than a research project, it received IRB exemption status. The study team analyzed the results for statistical differences between the “before” and 43 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. “after” responses of the participants using the Wilcoxon Signed Rank test. All tests were 2-tailed and alpha was set at 0.05. The short-answer data was reviewed for emergent themes using grounded theory, a theory in which meaning is gleaned from the information provided by the participants through their responses on the surveys. 11 Themes were identified independently by two reviewers and final themes determined by consensus. The results of the qualitative and quantitative sections were analyzed in a convergent manner to determine how the results relate to one another. The data provided information on the outcomes of the MEI conferences on the knowledge, skills, and attitudes of faculty participants in Mongolia. Results Eighteen faculty leaders participated in the first conference, and thirteen participated in the second conference. Of the 31 participants, 20 com- pleted the survey for a response rate of 65%. All faculty members who were present at the end of the conferences completed the survey. Analysis of the survey results indicated significant changes in the faculty level of agreement in all of the pre-post measures with a p<0.001 (see Table 2.) The most significant change was noted in familiarity with competency-based medical edu- cation, with an average pre-conference score of 2.5 and an average post-conference score of 4.5. The areas in which faculty most strongly indicated agreement at the end of the conference were those of being able to build a needs assessment (mean 4.7) and being able to develop goals and objectives for a rotation or class (mean 4.7). The area that faculty ranked as the weakest at the end of the conference was that of being able to give effective summative assessments (mean 4.3). Even this, however, is still a significant increase from a mean pre-conference response of 3.1. Table 2: Attendee self-reported knowledge before and after conference (1-5 scale) Mean (SD) Before After I am familiar with competency-based medical education 2.5 (0.8) 4.5 (0.5) I can develop a Needs Assessment for a course in residency 2.8 (1.1) 4.7 (0.5) I can write overall goals for a residency program 3.0 (1.0) 4.7 (0.5) I can develop goals and objectives for a rotation or class 3.0 (0.9) 4.6 (0.5) I can create a block schedule for resident rotations 3.0 (1.2) 4.6 (0.6) I can give effective feedback to a resident 2.9 (1.1) 4.6 (0.5) I can give effective summative assessments 3.0 (0.9) 4.3 (0.6) I value giving feedback to residents 3.1 (0.9) 4.6 (0.6) Note: All p values are significant at <0.001 Participants ranked the value of attending the conference at a mean score of 4.05 (see Table 3). They ranked the helpfulness of the small group discussions with colleagues between 4.26 (discussion on how to implement competencies into the medical education system) and 4.63 (discussion on the ideal structure of residency training). Participants rated the helpfulness of the small group discussions higher than they rated the value of attending the conferences. Table 3: Impact of conference on attendees (1-5 scale) N Mean (SD) I value attending this conference 20 4.1 (0.9) Small group discussions with colleagues on the ideal structure of residency training were useful 19 4.6 (0.8) 44 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. Small group discussions on how to implement competencies in our medical education system were helpful 19 4.3 (0.7) Small group discussions on how to give feedback will change how I give feedback in the future 18 4.5 (0.5) Small group discussions on how to give assessments will change how I use assessments in the future 19 4.5 (0.5) Respondents indicated that they had received between 0 and 10 days of faculty development on similar topics in the past, with an average of 2.3 days of previous training (see Table 4). There was a wide range of responses when asked about the percentage of material that was new to participants. Two (10%) respondents indicated that 20-30% of the material was new, while four (20%) indicated that 80-100% of the material was new to them (see Table 5). On average (mean 57%, [range 20- 100%]) about half of the conference material was reported as new to the attendees. Table 4: Number of days of similar faculty development training Number of Participants 0 Days 5 1-2 Days 1 3-6 Days 4 7-10 Days 1 Table 5: Percent of material that was new Number of Participants 20-30% 2 40-50% 1 60-70% 4 80-100% 4 The most common participant response to the question asking which part of the conference was most helpful was the response “all” (see Appendix B). While participants were reluctant to report anything negative, one response was that some parts of the conference were not sufficiently advanced for the group; the other respondents indicated that there were no negative outcomes. Participants suggested the conference could be more useful by using a translator who was familiar with residency education, by including more details and examples in the presentations, and by having more small- group sessions. They suggested the conference could be more culturally appropriate by including “more education and practice.” Elsewhere, par- ticipants reported that more examples would be helpful; this could mean that participants would like examples from their local universities and contexts. Respondents also suggested that the provision of continuing education for participants at different levels would be desirable. When asked how MEI could support the faculty in further development of their residency programs, responses included access to all the curriculum and documents used during the conference, and more training. When asked about specific topics for additional training, participants indicated topics specific to their disciplines, education methods for residency programs, and learner assessments. Discussion We used a retrospective pretest survey method with additional qualitative questions to evaluate the outcomes of a faculty development conference for medical educators in Mongolia. The qualitative questions also provide information to inform future faculty development conferences, as the responses have information on what was perceived as helpful or not, and suggestions on how to improve similar conferences in the future. Overall, we found that participants valued and learned from the conference. The most common respondent answer to the question, “What part of the conference was most helpful?” was “all.” Participants had received minimal training in residency education (mean of 2.3 days) in the past, indicating a gap in previous faculty development 45 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. training. This was consistent with their relatively low self-rating on the “before” section of the survey. Many participants also requested more training in the future, and participants requested the handouts and presentations used in this training for future reference. While one person wrote that “we have so many needs and issues,” others indicated that they would like more advanced training, including more “detailed information on different educational methods.” This, as well as the comment that participants would like education specific to participants of different levels of experience, indicates that many Mongolian faculty members already have a basic level of knowledge and would like to expand that knowledge. It also shows their passion for professional growth and development. The only negative outcome of the conference identified was one person’s response that “some part[s were] not advance[d]” enough. Requests for additional training included “more detailed information and examples on training,” and “practice from another country.” Prior to planning any future faculty development conference, conference leaders would benefit from surveying potential participants on their level of training and comfort using various skills. The assessment of conferences in non- Westernized developing countries requires an awareness of the cultural context and how this might impact both the conference and the validity and reliability of an assessment. In the Confucian belief system, which is common in Mongolia, hierarchy is respected, and a strong power differential exists between teacher and learner. In contrast, Westernized cultures have a much more individualistic paradigm, with power differentials playing a much smaller role in organizations. 12 These cultural differences have many implica- tions for the educational environment of a conference. In Westernized cultures, questioning a professor is not only acceptable; it is often viewed positively as a demonstration of critical thinking. For a student from an Eastern culture, questioning an authority figure is disrespectful; students are less likely to ask questions since that might insinuate that the teacher had not adequately explained the topic. Presenters with a Western mindset may assume that all learners understand the material if no questions are asked, whereas the learners may have many questions that they do not ask out of respect for the Westerners. If presenters from the West are not aware of and responsive to these different perspectives, learning can be impacted. To modulate this challenge, the conference was arranged with frequent workshops after a presentation (see Table 1). During the workshops, the participants worked together to discuss and apply the material that had just been presented. Leaders of the conference were available to provide feedback to attendees during those sessions, and participants provided feedback to each other as well. This provided a non-threatening environment in which participants asked many questions. One respondent commented on the im- portance of a having a content-knowledgeable interpreter for the conference. A highly-trained physician was the interpreter for the first conference, and she facilitated clear com- munication. The first day of the second conference, we were given two interpreters, and neither was as familiar with residency education; they occasionally asked for clarification of terms. On the second and third day of the second conference, another interpreter completed the interpretation for us, and this was more effective. The presenters adapted to these challenges by speaking more slowly and using descriptors to assist the interpreter with technical terms. This study had several limitations, confound- ing factors and barriers. The first is that the study population was only those faculty members who participated during the entire three days of the conference and were still present at the end of the third day. This could have led to a sampling error biased positively towards the utility of the conference since those who attended the entire conference were likely to be those who perceived it 46 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. as most beneficial. This could also have impacted the number of surveys completed since some participants left early without completing a survey. Additionally, the study population could have been depleted of the busiest faculty members who needed to attend to other responsibilities during parts of the conference. Language was another barrier. The survey was written in English, while Mongolian is the participants’ primary language. The participants indicated that they read in English and declined having the surveys translated into Mongolian. Participants were informed that they could answer the survey in Mongolian; Mongolian responses were translated into English by a Mongolian physician who is fluent in both English and Mongolian. While this should result in an accurate translation, there is a possibility of a loss of some of the nuances between the Mongolian and English languages. Also, as discussed above, a content- knowledgeable interpreter is imperative for optimal communication. Providing a conference for participants from a different culture is challenging. One response indicated that the system in Mongolia is significantly different from the Western system with which the presenters were accustomed, stating, “in my opinion our philosophy and goals is different from western countries.” This can have implications on the transferability of the infor- mation presented on the USA’s residency education system. Additionally, several comments mentioned the need for administrative leaders to participate in the conferences and to make changes in the Mongolian residency education system. The assessment of conferences in non- Westernized developing countries requires an awareness of the cultural context and how this might impact both the conference and the validity and reliability of an assessment. Since learners from Eastern cultures are careful not to shame their instructors, they may have withheld criticism and felt obliged to indicate that they learned a lot and appreciated the conference, regardless of their true opinions. To ensure that research is reliable, it is imperative that researchers work within the cultural belief system in a manner that can obtain accurate information. Using a neutral third-party to obtain feedback may improve the accuracy of data obtained from faculty participants. Strengths of this study included the ability of attendees to provide written, anonymous, qualitative data without having to give feedback verbally to presenters. Additionally, the survey could be completed in English or Mongolian, based upon participant preference. Finally, all particip- ants who were present at the end of the conference participated in completing the surveys. As a result of this project, MEI faculty development conferences have introduced surveys to evaluate the outcomes of conferences. This affirms that MEI leadership values the information provided by the survey and has taken action to evaluate the outcomes of conferences. Based upon feedback from the surveys, MEI has changed how conferences are led by making sessions more interactive. The compilation of feedback from additional conferences will provide information that is more valid and generalizable. It might be helpful if teams providing a faculty development conference survey participants in advance of the conference to obtain information on the current level of participant knowledge and specific areas of need. This would help presenters tailor the conference to the needs, level and interests of the participants. It could also serve to stimulate interest in and provide realistic expectations for the conference. Conclusions Overall, our findings indicate that conference attendees valued the faculty development conference led by physicians from a Westernized country. Our findings also indicate the importance of including administrative leadership from the host country in faculty development. This will facilitate discussions regarding changes in the training system at the structural level and foster a 47 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. conference environment that is culturally relevant. Further research needs to be done to determine long-term effectiveness of the conferences through changes in residency education and patient outcomes. This survey can be modified and used to evaluate the outcomes of other faculty development conferences in developing countries. References 1. Newman L, Pelletier S, Lown B. Measuring the impact of longitudinal faculty development: a study of academic achievement. Acad Med, Advance Online publication. 2015. http://dx.doi.org/10.1097/ACM.0000000000001016 2. Baasandorj D. Faculty development program needs at Mongolian State Universities: content and strategies [Doctoral Dissertation]. 2010. Retrieved from ProQuest (DAI-A 71/09) 3. Hegde P. Faculty development trends in medical education: a review. SE Asian J Med Educ. 2013;7(2),11-6. 4. Majumder M. Issues and priorities of medical education research in Asia. Ann Acad Med Singap. 2004;33(2),257-63. 5. Steinert Y, Mann K, Anderson B, Barrnett BM, Centeno A, Naismith L, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach. 2016;38(8),769-86. http://dx.doi.org/10.1080/0142159X.2016.1181851 6. Houston T, Ferenchick G, Clark J, Bowen J, Branch W, Alguire P, et al. Faculty development needs. J Gen Intern Med. 2004;19(4), 375-9. http://dx.doi.org/10.1111/j.1525-1497.2004.30619.x 7. Wong J, Fang Y. Improving Clinical Teaching in China: initial report of a multihospital pilot faculty development effort. TeachLearnMed. 2012;24(4),355-60. http://dx.doi.org/10.1080/10401334.2012.719801 8. Bland CJ, Schmitz CC, Stritter FF, Henry RC, Aluise JJ. Successful faculty in academic medicine: Essential skills and how to acquire them. New York: Springer-Verlag; 1990. 9. ACGME. 2010a. Accreditation council for graduate medical education: glossary of terms. Accreditation Council for Graduate Medical Education. [Updated 2013 July 1; cited on 2016 Sept 21.] Available from: https://www.acgme.org/Portals/0/PDFs/ab_ACGM Eglossary.pdf 10. Lamb T, Tschillar R. Evaluating learning in professional development workshops: Using the retrospective pretest. Journal of Research in Professional Learning. 2005;1,1-9. 11. Cresswell, J. Research Design: Qualitative, Quantitative and Mixed Methods Approaches. Los Angeles: SAGE. 2014. 12. Kikukawa M, Stalmeijer, RE, Emura S, Roff S Scherpbier AJ. An instrument for evaluating clinical teaching in Japan: Content validity and cultural sensitivity. BMC Med Educ. 2014;14,179-86. http://dx.doi.org/10.1186/1472-6920-14-179 Appendix A: Retrospective Survey on Faculty Development Conference Thank you for completing the following survey, which measures how useful and effective you feel this conference was. Your responses will be anonymous. The information will be used to improve future Medical Education International conferences both in Mongolia and in other countries. We appreciate honest responses since they will help us improve. This survey is optional. You do not have to answer it if you do not want to. If any of the questions make you feel uncomfortable, you may skip them. The data from the surveys may be presented in a paper, a poster, or another form of dissemination. Participation in the survey indicates agreement with this. Please select your level of agreement with each of the following statements on a scale from 1 to 5, with 1 being “strongly disagree,” 2 being “disagree,” 3 being “neutral,” 4 being “agree,” and 5 being “strongly agree.” With each statement, please select your level of agreement with the statement before the conference began and after the conference was over. 1.) I am familiar with competency-based medical education. Before: 1 2 3 4 5 http://dx.doi.org/10.1097/ACM.0000000000001016 http://dx.doi.org/10.1080/0142159X.2016.1181851 http://dx.doi.org/10.1111/j.1525-1497.2004.30619.x http://dx.doi.org/10.1080/10401334.2012.719801 https://www.acgme.org/Portals/0/PDFs/ab_ACGMEglossary.pdf https://www.acgme.org/Portals/0/PDFs/ab_ACGMEglossary.pdf http://dx.doi.org/10.1186/1472-6920-14-179 48 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. After: 1 2 3 4 5 2.) I can develop a Needs Assessment for a course in residency. Before: 1 2 3 4 5 After: 1 2 3 4 5 3.) I can write overall goals for a residency program. Before: 1 2 3 4 5 After: 1 2 3 4 5 4.) I can develop goals and objectives for a rotation or class. Before: 1 2 3 4 5 After: 1 2 3 4 5 5.) I can create a block schedule for resident rotations. Before: 1 2 3 4 5 After: 1 2 3 4 5 6.) I can give effective feedback to a resident. Before: 1 2 3 4 5 After: 1 2 3 4 5 7.) I can give effective summative assessments. Before: 1 2 3 4 5 After: 1 2 3 4 5 8.) I value giving feedback to residents. Before: 1 2 3 4 5 After: 1 2 3 4 5 9.) I value attending this conference. Before: 1 2 3 4 5 After: 1 2 3 4 5 10.) Small group discussions with colleagues on the ideal structure of residency training were useful. 1 2 3 4 5 11.) Small group discussions on how to implement competencies into our medical education system were helpful. 1 2 3 4 5 12.) Small group discussions on how to give feedback will change how I give feedback in the future. 1 2 3 4 5 13.) Small group discussions on how to give assessments will change how I use assessments in the future. 1 2 3 4 5 14.) Please answer the following questions in English or Mongolian:  What part of the conference was most helpful?  Were there any negative outcomes of the conference, and if so, what were they?  What could be done differently to make future conferences more useful?  How could the conference be more culturally appropriate?  How could we support you in further development of your residency programs?  What faculty development topic would you like further training on in the future?  How many days in the past have you spent studying similar themes in other training programs or conferences?  Approximately what percentage of the material covered in this conference was new to you? Appendix B: Qualitative Data: Themes from Survey Responses 1. What part of the conference was most helpful? a. Entire conference b. Lectures c. How to write program goals and objectives 2. Were there any negative outcomes of the conference, and if so, what were they? a. None b. Some parts weren’t advanced enough. 49 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. 3. What could be done differently to make future conferences more useful? a. More small group study b. Utilize a content-knowledgeable translator c. Provide more details and examples d. Have more administrators attend 4. How could the conference be more culturally appropriate? a. Education and practice b. Provide continuous education at different levels 5. How could we support you in further development of your residency program? a. Curriculum materials/documents/milestones/lectures for the participants b. Continued training, some in collaboration with other countries c. Research work d. Participation of president and others e. Our philosophy and goals are different from western countries 6. What faculty development topic would you like further training on in the future? a. Topic specific: e.g. family medicine, Internal medicine b. Education methods for residency programs (with block education mentioned once) c. Assessment, evaluation Peer Reviewed Competing Interests: None declared. Acknowledgments: Cari Ahlers-Schmidt, PhD, Associate Research Professor, Office of Research, University of Kansas School of Medicine-Wichita contributed to the design of the work and was instrumental in the analysis of the data, and Matt Engel, MPH, Research Associate, University of Kansas School of Medicine-Wichita for data analysis. Correspondence: Dr. Deborah Kroeker, University of Kansas School of Medicine, USA deborah.kroeker@wesleymc.com Cite this article as: Kroeker DA, Walling A. Outcomes of a Faculty Development Conference in Mongolia. Christian Journal for Global Health (Nov 2016), 3(2):38-48. © Kroeker DA, Walling A. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly cited. To view a copy of the license, visit http://creativecommons.org/licenses/by/4.0/ www.cjgh.org mailto:deborah.kroeker@wesleymc.com http://creativecommons.org/licenses/by/4.0/ http://creativecommons.org/licenses/by/4.0/