Research 148 September 2017, Vol. 9, No. 3 AJHPE Background The mini-clinical-evaluation exercise (mini-CEX) is a way of assessing the clinical performance of medical students.[1] The mini-CEX consists of three basic elements: a direct observation of the clinical performance of students; an assessment of clinical performance, based on components of competence; and a feedback session immediately after the observation.[1] Assessment based on direct observation of clinical performance, with feedback, has a strong impact on student behaviour and learning processes.[2,3] However, studies have found that the feedback given to students is inadequate.[4] Factors that have influenced the lack of provision of feedback are the ability of the teacher to provide feedback, the format of the assessment form (for written feedback) and the process of assessing and giving feedback.[3,5,6] Since the original emergence of the mini-CEX assessment, the form has included spaces to write feedback and an action plan for further learning.[1] The space for writing feedback is divided into two sections: areas that are positive (strengths), and areas that need to be improved (weaknesses). These spaces are found to be mostly unutilised.[7] The provision of written feedback and an action plan on the mini-CEX form is not satisfactory owing to the use of unspecific language, the format of the assessment form and uncertainty around the assessment process.[5] Moreover, the assessment process provides little ‘dialogue’ space between the teacher and the student. Efforts to use a modified mini-CEX form were therefore made to stimulate the provision of written feedback and reflection.[5] Unwritten or unspecific written feedback and action plan on the mini- CEX form leaves unanswered questions as to how the feedback session went, and whether it really benefited the student’s learning process. The use of written feedback is important for evaluating the progress of a student based on written records/documents, and in the preparation of his/her portfolio.[8] Against this background, we tested whether a modified mini-CEX form, with the addition of specific spaces on separate sheets for feedback and action plan, and of a new section for written reflection, would improve the quantity and quality of the written feedback, action plan and student reflection in the mini-CEX assessment. Ethical approval for this study was obtained from the Medical Health Research and Ethics Committee (MHREC) at Gadjah Mada University (ref. no. KE/FK/953/EC). Methods Design This was a single-group pre-test-post-test quantitative study, comparing the data from previous mini-CEX forms and the data collected over 3 months after the introduction of the modified mini-CEX forms. The analysis was done by means of comparing the quantity and quality of the written feedback, action plan, and student reflection before and after the use of a modified form. Background. Mini-clinical-evaluation exercise (mini-CEX) assessment forms that have been modified with the addition of specific spaces on separate sheets are expected to improve the quantity and quality of written feedback and the action plan for further learning which is agreed upon, and to encourage written reflection. Objective. To test whether the modified mini-CEX assessment forms improve the quantity and quality of the written feedback, action plan and student reflection in the mini-CEX assessment process. Methods. Data collection was conducted over 3 months. Data analysis used the c2 test to compare the quantity and quality of written feedback, action plans, and reflections before and after the introduction of a modified mini-CEX assessment form. Results. Twenty-four clinical teachers and 44 clinical students participated in this study. The percentage of written feedback increased by 39%, and the specificity increased by 30.1% (p=0.001). The percentage of written action plans increased by 37%, and the specificity increased by 17.7% (p=0.001). The percentage of written reflection in the new spaces on the modified forms reached 73%, and 49.2% were classified as ‘specific’. Conclusion. The use of modified mini-CEX assessment forms, with the addition of extra spaces on separate sheets, improved the quantity and quality of written feedback and action plans, and encouraged written reflection. Afr J Health Professions Educ 2017;9(3):148-152. DOI:10.7196/AJHPE.2017.v9i3.804 Quantity and quality of written feedback, action plans, and student reflections before and after the introduction of a modified mini-CEX assessment form R M Djajadi,1 MD, MMedEd; M Claramita,2 MD, MHPE, PhD; G R Rahayu,2 MD, MMedEd, PhD 1 Medical Education Unit, Faculty of Medicine and Health Sciences, Warmadewa University, Bali, Indonesia 2 Department of Medical Education, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia Corresponding author: R M Djajadi (robinmartilo@gmail.com) This open-access article is distributed under Creative Commons licence CC-BY-NC 4.0. Research September 2017, Vol. 9, No. 3 AJHPE 149 Participants The participants were 24 clinical teachers and 44 undergraduate clerkship students who were on duty between 16 September and 15 December 2014 in the Departments of Obstetrics and Gynecology, Surgery, Internal Medicine and Paediatrics in Sanjiwani Hospital, Indonesia. In order to gain as much collectable data in the form of the mini-CEX sheets as possible, we purposely chose participants from those departments which had the longest period of rotation (3 months), and the highest number of students assigned at the time. Instrument The instrument used for collecting data was a modified mini-CEX form.[7] The original unmodified form was the regularly used mini-CEX form adapted from Norcini’s.[1] We then modified the mini-CEX form by adding specific spaces for feedback and reflection on each component of competence that was assessed. This modification was adapted from Pelgrim et al.’s[5] work on the mini-CEX form. The components of competence assessed in the mini-CEX form are as follows: (i) history taking; (ii) physical examination; (iii) professionalism; (iv) clinical judgement/diagnosis; (v) patient management; (vi) communication skills; (vii) organisation/ efficiency; and (viii) overall clinical care. Our modification was to include separate sheets for writing feedback, an action plan and student reflection. Sheet A, which refers to the components of competence assessed, is used for assessing student clinical performance. Sheet B, which contains specific feedback spaces for each component of competence, is used for writing feedback, and sheet C, which contains specific reflection spaces on each component of competence and space for an action plan, is a student reflection sheet. The modified mini-CEX form was then validated by two experts in medical education. The process of using the modified mini-CEX form was conducted based on the ‘reflective-feedback dialogue’ principle, as follows:[9] (i)  students undertake clinical performances for the purposes of the mini-CEX, accompanied by a teacher; (ii) the teacher directly observes and assesses the student’s performance using sheet A; (iii) during the observation of the student’s performance, the teacher can write a short and specific feedback section on sheet B; (iv) after finishing the clinical performance, the student is given the opportunity to write a brief reflection on sheet C; (v) the teacher asks the student to present his/her reflection; (vi) the teacher provides oral feedback based on the feedback that has been written and based on student reflection; (vii) the teacher and student agree on an action plan based on learning goals, and the student writes this down briefly and specifically in the action plan space on sheet C; (viii) a follow-up should be planned to evaluate the achievements of the action plan at the next meeting; and (ix) the form is collected and placed in the portfolio document. Procedures Students are required to undergo mini-CEX assessments as a prerequisite for their being permitted to take the final examination on each departmental rotation. Each departmental rotation has a different schedule and different requirements about the number of mini-CEX assessments which must be performed. At the Department of Obstetrics and Gynecology, students must do two mini-CEX assessments, in weeks 5 and 7. The Department of Surgery requires three mini-CEX assessments, in weeks 3, 5 and 7. One mini-CEX is required in the rotation of the Department of Paediatrics, in the last week before the final examination (week 12). The rotation in the Department of Internal Medicine also requires one mini-CEX assessment, which is conducted between weeks 8 and 10. The teachers who assess the mini-CEXes are decided on according to a fixed schedule, and cannot be specifically requested by students. Before data collection was carried out, we met all clinical teachers and undergraduate clerkship students to introduce the modified mini-CEX form. The length of the introduction session was ~20 - 30 minutes. It consisted of a brief description of the modified mini-CEX forms, and the steps to be taken to use them. We collected all unmodified mini-CEX forms from the mini-CEX assessments that had been done in the 3 months before the study began. Then we collected all of the modified mini-CEX forms used over the next 3 months. Once all the forms had been collected, the data transcription was carried out. Data analysis The amounts of written feedback, action plan and reflection were calculated by using the number of filled spaces on the form for each category as a percentage of the number of mini-CEX forms collected. The quality of the written feedback, action plan and reflection was determined according to specificity, where each variable was classified as belonging to one of three categories: specific, quite specific, or not specific. The specificity criterion for written feedback and written reflection was defined using the classification of Pelgrim et al.[5] The written material was classified ‘specific’ if it was clear which component was being referred to, which aspects of it were already positive v. which needed improvement, and why they were either considered good or needed to be improved. It was ‘quite specific’ if it only indicated which component was being referred to and what was already good about it v. what still needed to be improved. It was ‘not specific’ if only general aspects of the overall student clinical performance were covered, without indicating which components were really referred to in terms of what was already good and what needed to be improved. Similarly, the written action plans were classified using Pelgrim et al.’s [5] criteria as ‘specific’ if they showed explicitly which aspects of their clinical performance should be studied further; ‘quite specific’ if they only listed the components that should be corrected without giving further details; and ‘not specific’ when only general statements were made. Inter-rater reliability was tested with three other raters, who were students of a Master’s programme in medical education, using Cohen’s kappa test to ensure the validity of the classification made by the researcher. The raters were chosen based on their knowledge of mini-CEX assessments and feedback. The Cohen kappa-test results for the specificity classification of the written feedback, action plan and reflection between researchers indicated that there was agreement on the analysed variables (kappa values >0.6). P-values were calculated using the c2 test. The test was considered significant when p<0.05. Results The total number of unmodified mini-CEX forms collected between 16 June 2014 and 15 September 2014 was 78. The total number of the modified mini-CEX forms collected between 16 September and 15 Decem- ber 2014 was 63. Research 150 September 2017, Vol. 9, No. 3 AJHPE Written feedback The amount of written feedback increased by 39.0%, and the amount of specific feedback by 30.1% (p=0.001), when the modified forms were used (Table 1). On the modified mini-CEX form, there are feedback spaces that are specific to each component assessed. If there was only one feedback comment on the assessed component that corresponded to the ‘specific’ criteria, then the overall feedback was classified as ‘specific’. An example of specific written feedback on the modified mini-CEX form is: The format of the interview already met/Preparation of the patient prior to the interview should be considered; ‘Generally good/observe patient reaction during examination; Already greets the patient and introduced himself/Maintain friendliness with the patient; Already good/(blank); (blank)/Need to understand again about termination of pregnancy; Already good/(blank); (blank)/ (blank); (blank)/(blank). (Sheet OB1B1).’ In this example, statements on the left of the ‘/’ indicate aspects of the clinical performance, such as history taking, physical examination, professionalism, patient management, communication skills, that are listed as strengths, while those to the right are aspects still considered deficient. It also considers what is good v. what is still lacking, and/or why it is good or still lacking, in ‘interview format (already met), the patient's reaction (during examination), (already) greets the patient, (maintain) friendliness, (need to understand again about) termination of pregnancy’. Action plans The percentage of written action plans increased on the modified forms by 37.0%, and specific action plans increased by 17.7% (p=0.001) (Table 2). In the example of a specific action plan, below, it is made explicit which aspects of clinical performance should be studied further: ‘Review the reason for induction in cases of oligohydramnion, the plan of putting a patient on a drip in the case of oligohydramnion; re-evaluate how to perform a physical examination of the Leopold manoeuver and VT [vaginal toucher] and how to calculate FHR [fetal heart rate].’ (Sheet OB6B3) Written reflection A total of 73% of the written reflection sheets had been used by the students to write reflections, while specific reflections made up 49.2% of the written reflection on the modified forms (Table 3). An example of a specific written reflection, again with strengths listed on the left and weaknesses on the right, is as follows: Detailed history/(blank); (blank)/Leopold examination, VT examination; greetings/(blank); (blank)/Indications for administration of oxytocin drip; Comprehensive/(blank); Good/(blank); Good/(blank); Good/(blank). (Sheet OB6B3) This example states which components are already good, and which are still deficient, such as history taking, physical examination, profession alism, patient management, communication skills. It also shows what is good v. lacking, and/or why this is so, in ‘detailed (history), Leopold (examination), VT (examination), greetings, indication for giving oxytocin drip’. Discussion The results of this study showed that the use of a modified mini-CEX form may increase the quantity and quality of teachers’ written feedback. The modified mini-CEX forms have spaces next to each component of competence to write feedback. These spaces can be used to capture written feedback, either specific or not. In the written feedback it can be assumed that the more specific the feedback, the more concerned the teacher was about this component. The mini-CEX was constructed to take into account the limited time that clinical teachers have while teaching and assessing students at the clinical bedside.[1] At first, after the introduction, most of the clinical teachers who participated in this study were reluctant to engage in the intervention because of the impression that it would give them more tasks and take up more of their time. However, we tried to reassure them that based on the process of using the modified mini-CEX form, the writing of feedback would be part of the student performance observation, which required no additional amount of time. The teacher would write Table 1. The amount and quality of written feedback Quality of feedback Mini-CEX form type Unmodified (N=78), Modified (N=63), n (%) n (%) Blank 49 (62.8) 15 (23.8) Filled 29 (37.2) 48 (76.2) Not specific 21 (27) 14 (22.2) Quite specific 3 (3.8) 11 (17.5) Specific 5 (6.4) 23 (36.5) p=0.001, significant if p<0.05. Table 2. Action plan specificity in modified v. unmodified forms Specificity Mini-CEX form Unmodified (N=78), Modified (N=63), n (%) n (%) Blank 71 (91) 34 (54) Filled 7 (9) 29 (46) Not specific 1 (1.3) 5 (8) Quite specific 5 (6.4) 12 (19) Specific 1 (1.3) 12 (19) p=0.001, significant if p<0.05. Table 3. The specificity of written reflection Reflection Modified mini-CEX form (N=63), n (%) Blank 17 (27) Filled 46 (73) Not specific 3 (4.8) Quite specific 12 (19) Specific 31 (49.2) Research September 2017, Vol. 9, No. 3 AJHPE 151 the feedback while doing the observation as part of usual procedure. Therefore, the modified mini-CEX form was not meant to be a burden, but should help the teachers remind themselves of their feedback statements that would be delivered at the end of the assessment. This written feedback could benefit the teacher as a reminder, if they had no time available to deliver the feedback right away. They could do the feedback session later based on the written feedback note. The results of this study also showed that the use of the modified mini- CEX form increased the quantity and quality of the written action plans. These results are in accord with those of Haffling et al.,[6] who used the reflective-feedback dialogue principle in a structured assessment form. This study adapted the steps appropriate to the reflective-feedback dialogue principle, which require teacher and student to interact in order to reach conclusions, and thereby develop the action plan. The existence of this written action plan showed that there was interaction between teacher and student, resulting in agreement. Therefore, it can be concluded that the increase in the number of action plans developed that was seen in this study occurred because the steps used on the modified mini-CEX form can encourage teacher and student to interact better and to create a more specific action plan. This result proved that the use of the reflective-feedback dialogue principle on the mini-CEX assessment process was beneficial for encouraging interaction and thus improving the quality of the action plans developed. The modified mini-CEX form used in this study also includes a separate student reflection sheet that contains spaces to reflect on each component of competence. This reflection sheet serves to remind students to reflect after the clinical performance is complete. In addition, it also serves to remind them that there are things that must be communicated between student and teacher. The filled spaces for written reflections were used in 73% (n=63) of the forms, and 67% of these written reflections were classified as specific (31/46). This result is satisfactory, considering that the writing of reflections on the mini-CEX form was a first-time experience for the clinical students in this study. Note that reflection takes practice to make it a habit, which the use of these forms encourages. There was a possibility that the reflection data collected were written after performing the steps, i.e. outside of the mini-CEX duration, or after receiving feedback. This may have affected the result. However, according to Sargeant et al.,[10] reflection after receiving feedback can help a person in the process of deciding to accept or to reject the feedback. Therefore, despite this possibility of having written the reflections outside of the mini-CEX duration, it can still be concluded that in this study the use of the modified form encouraged and trained students to write reflections on their mini- CEX assessment form. In the process of formative assessment, there are two aspects that may be reflected on: the performance itself, and the feedback provided after the performance.[11,12] Either reflection on the performance[13] or reflection on the feedback given[10] can be beneficial to the learning process. Therefore, it should be a point of consideration in the future in determining the use of the reflection sheet: that is to say, whether the reflection should be made only after receiving feedback, or before or after receiving feedback. The nature of the reflection that happened can be seen from the contents of the written reflections, which are able to demonstrate the strengths (aspects already good) and the weaknesses (things that need to be improved) of students’ performance, and possible things that can be improved after the mini-CEX assessment. The existence of this reflection on experience indicates that there has been a deep learning process.[12,14] We suggest that future research should elaborate on this issue of reflection. In this study, the use of a modified form led to an increase in the quantity and quality of the feedback and the action plans. However, it still can be questioned why the results have not been closer to 100%. There are several factors that affect the quantity and quality of the feedback and the action plans: factors involving the research subjects (communication skills and understanding of the feedback and the action plan) and factors related to conditions in the field (especially the duration of the mini-CEX). The research-subject factors play a more important role in determining the quantity and quality of the feedback and the action plan.[5] The mini-CEX duration factor also affects the quantity and quality of the feedback and the action plan. Presumably, the longer the duration of the mini-CEX assessment and feedback session, the greater the likelihood of having better quantity and quality of feedback and action plan. In this study, the mini-CEX duration factor was not controlled. Data were captured in real conditions that occur in the field, in terms of variations in the duration of the mini-CEX. This study may have been affected by the Hawthorne effect bias, in that the research subjects felt that they were being observed, inducing them to improve their performance. However, data on some subjects that undertook mini-CEX assessments several times within the 3 months showed that there were variations in the quantity and quality of the feedback. Some of the subjects remained consistent in providing written feedback, and some did not. This shows that the effect of an improvement in performance due to being observed was minimal. The long data collection period (3 months) also minimises this effect of being observed. This study used a small sample, but if we consider the proportion of the sample to the total population of teachers and students in the hospital, the sample is large. The number of research subjects in the clinical-teacher sample was 24 people, or 63% of the population (38 people). The number of research subjects in the clinical student sample was 44 people, or 38% of the population (115 people). The sample of clinical students in this study was not randomised, because we used the convenient sampling method, but considering that the distribution of the clinical rotation schedule for every department was already done randomly, this implies that the clinical students had already undergone a randomisation process. This study only considered four departments/laboratories, namely, obstetrics and gynaecology, surgery, internal medicine and paediatrics. This may affect the generalisability of the result to the whole undergraduate clerkship. The number of mini-CEX assessments performed in each rotation also varies, so that the amount of data collected from each research subject differed. Efforts to control the number of mini-CEX assessments conducted for research subjects in order for them to each have the same number is technically not feasible, and therefore we selected a period for comparison, i.e. 3 months before and after the intervention. We suggest that further research on these modified mini-CEX forms be carried out in different locations and settings. Conclusions From these results it can be concluded that the use of the modified mini- CEX form, with the addition of specific spaces on separate sheets, improved the quantity and quality of written feedback and written action plans. It could also encourage and train students to write reflections on their clinical performance. Educational institutions could apply the use of the modified Research 152 September 2017, Vol. 9, No. 3 AJHPE mini-CEX form to support the learning process of mini-CEX assessments. Data from the modified mini-CEX forms could be used for student portfolios, to monitor the development of the clinical-student learning process. Future studies are needed to assess user perceptions of using the modified mini-CEX form, and its relationship to the results of the next mini-CEX assessment or examination. Acknowledgements. The authors would like to thank the following individuals for their feedback on the article: Dr Efrayim Suryadi, Dr Yoyo Suhoyo and Dr Ova Emilia. Author contributions. RMD contributed to the design and implementation of the research, data collection, to the analysis and interpretation of the results and to the writing of the manuscript. Both MR and GRR supervised the project, gave feedback to the design and implementation of the research, interpretation of results and contributed to the final version of the manuscript. All authors provided critical feedback and helped shape the research, analysis and manuscript. Funding. This research received no grant from any funding agency in the public, commercial, or not-for-profit sectors. Conflicts of interest. The authors report no conflict of interest. The authors alone are responsible for the content and writing of the article. 1. Norcini J. The mini clinical evaluation exercise. Clin Teach 2005;2:25-30. https://doi.org/10.1111/j.1743- 498X.2005.00060.x 2. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007;29:855-871. https://doi.org/10.1080/01421590701775453 3. Burch VC, Seggie JL, Gary NE. Formative assessment promotes learning in undergraduate clinical clerkships. S Afr Med J 2006;96:430-433. 4. Daelmans HEM, Overmeer RM, van der Hem-Stokroos HH, et al. In-training assessment: Qualitative study of effects on supervision and feedback in an undergraduate clinical rotation. Med Educ 2006;40:51-58. https://doi.org/ 10.1111/j.1365-2929.2005.02358.x 5. Pelgrim EAM, Kramer AWM, Mokkink HGA, Van der Vleuten CPM. Quality of written narrative feedback and reflection in a modified mini-clinical evaluation exercise: An observational study. BMC Med Educ 2012;12:97. https://doi.org/ 10.1186/1472-6920-12-97 6. Haffling AC, Beckman A, Edgren G. Structured feedback to undergraduate medical student: 3 years’ experience of an assessment tool. Med Educ 2011;33:e349-e357. https://doi.org/10.3109/0142159x.2011.577466 7. Djajadi RM. Jumlah dan Kualitas Umpan Balik dan Rencana Aksi Secara Tertulis Sebelum dan Sesudah Penggunaan Lembar Penilaian Mini-CEX dengan Penambahan Ruang Spesifik Umpan Balik dan Refleksi. Master’s thesis. Yogyakarta: Universitas Gadjah Mada, 2015. 8. Haffling AC, Beckman A, Pahlmblad A, Edgren G. Students’ reflections in a portfolio pilot: Highlighting professional issues. Med Teach 2010;32:e532-e540. https://doi.org/ 10.3109/0142159X.2010.509420 9. Cantillon P, Sargeant J. Giving feedback in clinical settings. BMJ 2008;337:1292-1294. https://doi.org/ 10.1136/bmj.a1961 10. Sargeant JM, Mann KV, van der Vleuten, CP, Metsemakers JF. Reflection: A link between receiving and using assessment feedback. Adv in Health Sci Educ 2009;14:399-410. https://doi.org/ 10.1007/s10459-008-9124-4 11. Archer JC. State of the science in health professional education: Effective feedback. Med Educ 2010;44:101-108. https://doi.org/ 10.1111/j.1365-2923.2009.03546.x 12. Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: A systematic review. Adv Health Sci Educ 2009;14:595-621. https://doi.org/ 10.1007/s10459-007-9090-2 13. Pelgrim EAM, Kramer AWM, Mokkink HGA, et al. Reflection as a component of formative assessment appears to be instrumental in promoting the use of feedback: An observational study. Med Teach 2013;35:772-778. https://doi. org/10.3109/0142159X.2013.801939 14. Loughran J. Effective reflective practice: In search of meaning in learning about teaching. J Teach Educ 2002; 53:33-43. https://doi.org/10.1177/0022487102053001004 Accepted 5 March 2017. https://doi.org/10.1111/j.1743-498X.2005.00060.x https://doi.org/10.1111/j.1743-498X.2005.00060.x https://doi.org/10.1080/01421590701775453 https://doi.org/ 10.1111/j.1365-2929.2005.02358.x https://doi.org/ 10.1111/j.1365-2929.2005.02358.x https://doi.org/ 10.1186/1472-6920-12-97 https://doi.org/10.3109/0142159x.2011.577466 https://doi.org/10.3109/0142159X.2010.509420 https://doi.org/ 10.1136/bmj.a1961 https://doi.org/ 10.1007/s10459-008-9124-4 https://doi.org/ 10.1111/j.1365-2923.2009.03546.x https://doi.org/ 10.1007/s10459-007-9090-2 https://doi.org/10.3109/0142159X.2013.801939 https://doi.org/10.3109/0142159X.2013.801939 https://doi.org/10.1177/0022487102053001004