178 October 2016, Vol. 8, No. 2 AJHPE Research Bringle and Hatcher[1] described service-learning as a ‘course-based, credit- bearing education experience in which students participate in an organized service activity that meets identified community needs, and reflect on the service activity in such a way as to gain further understanding of course content, a broader appreciation for the discipline, and an enhanced sense of civic responsibility’. Eyler et al.[2] defined reflection as ‘the hyphen between service-learning’. However, the way in which reflection is conducted should be carefully considered, as it may not necessarily result in deep learning or may lead to learning the incorrect information. It is therefore important to ensure good-quality reflection on experiences where students articulately express the essence of their learning.[3] These authors also found a correlation between meeting deeper learning outcomes and how rigorous the reflection opportunity is presented, and the quantity and quality of the reflections. As students learn from and through experience in service-learning modules, reflection becomes an integral part of the learning process. In this article, we first describe reflection and its role in service-learning. Subsequently, an overview of previous research on the measurement of reflection in service-learning is provided, and, lastly, the use of rubrics in the measurement of reflection is discussed. Reflection and its role in service-learning Reflection allows one to think critically about successes and failures, develop concepts that are based on ideas and information from multiple sources, and apply such knowledge in future.[4] Currently, reflection is regarded as an integral part of the learning process. There are various types of reflection journals for service learning,[5] including key-phrase journals, double-entry journals, critical incident journals, directed writings and three-part journals. The last was applicable to this research, as students had to respond to guided questions addressing three separate aspects: (i) to describe what happened in the service expe- rience with regard to what was successful or not; (ii) to provide possible reasons for this and to analyse how the course content relates to the service experience; and (iii) to apply the service experience to how they see the world (e.g. goals, values, attitudes, beliefs and philosophy). The benefits of reflection are numerous: it allows one to make better choices or to take more appropriate actions in future, which result in greater effectiveness;[6] students develop a deeper and more sustainable knowledge of curricular content, skills, and increased understanding of self, others and the community;[7] and they develop a critical understanding of their theoretical modules and foster an ability to consider their own progress, values and goals. Reflection activities must allow students to discover the value of dialogue, embrace the importance of perpetuity in the learning process, and develop the ability to consider the meaning of personal experience. Reflection should start on a contextual level (a holistic view of the situation) and move towards a dialectical level (to question the value of knowledge systems, moral and ethical issues).[8] Effective reflection should firstly link the service experience to the course content and be structured. It should also ensure regular provision of feedback from the instructor so that students learn how to improve their reflective practices, and provide the opportunity for students to discover, refine or adjust their values and opinions.[1] Facilitators need to help students to link their experiences to the course material and to challenge their beliefs to deepen their learning.[3] One of the challenges of facilitators in service-learning is to determine how reflection contributes to the students’ professional and personal growth. According to Bender et al.[9] it is essential that the outcomes of service- learning activities be assessed to measure learning and growth. Students do not encounter reflection similarly and also differ in how quickly they mature in their ability to learn from reflection.[1] Background. During a service-learning module, the focus is on the development of reflective competence, which is part of professional competence. The students have to reflect in a structured manner on the service-learning experience to understand and appreciate not only the module and discipline, but also their sense of personal value and social responsibility. By providing structured opportunities for reflection, deeper learning can be facilitated, which enhances competence. Do students benefit from the process of reflection and how should it be measured? Objective. To determine the usefulness of an assessment tool. Methods. A documented review of reflection journals made use of a rubric to score the structured reflection of students at a particular service-learning site. Descriptive statistics were used to analyse the data. Results. The results showed positive changes in terms of analysis, critical thinking, emotive aspects, social responsibility and self-confidence. Conclusion. Specific factors were identified that could have affected the reflections, and recommendations are made to increase the effectiveness of the assessment tool and the process of reflection. Afr J Health Professions Educ 2016;8(2):178-183. DOI:10.7196/AJHPE.2016.v8i2.586 The usefulness of a tool to assess reflection in a service-learning experience A-M Wium, BA Logopaedics, M Communnication Pathology, DPhil, PGCHE, PGD (Ethics); S du Plessis, BA Logopaedics, M Communnication Pathology, DPhil, PGCHE Discipline of Speech-Language Pathology and Audiology, School of Health Care Sciences, Sefako Makgatho Health Sciences University, Pretoria, South Africa Corresponding author: A-M Wium (anna-marie.wium@smu.ac.za) This open-access article is distributed under Creative Commons licence CC-BY-NC 4.0. October 2016, Vol. 8, No. 2 AJHPE 179 Research Measuring reflection: previous studies Most previous assessments of reflection relied on self-reported measures of outcomes, but did not assess student learning.[10] There seems to be a gap in the literature on how to determine professional and personal growth, or how the amount and type of reflection activity relate to student outcomes.[1] This can most probably be ascribed to limited assessment procedures or tools. Bender et al.[9] recommended that rubrics be created as scoring tools of reflection, in which the specific expectations and acceptable and unacceptable levels of performance are stipulated. Critical reflection is the process that transforms the service to the ideas and understanding of the learning experience. This study developed a rubric to assess the reflective competence of students. The objective was to determine the usefulness of an assessment tool that measures change in students’ personal and professional growth. Methods This was a retrospective study using document review.[11] Once ethical clearance had been obtained (MREC/H/170/2012:IR), qualitative data were collected from reflection journals that were previously completed by each student. The data were analysed qualitatively and quantita- tively. The journal entries were obtained from three different occasions over a period of 1 year and were scored retrospectively; the scores were compared to measure change.[10] These reflections were scored using a rubric to increase reliability and validity. A sampling design was not used, as all the journals compiled by an entire group of eight students at a particular service-learning site were analysed. The reflection journals consisted of structured questions. As a rule, the journal entries were made on a weekly basis for 30 minutes during the service-learning programme. Students reflected on their experiences without consulting peers and completed their reflection sheets on site after the service had been provided. At the time of the reflection, the students congregated in a communal meeting room in the presence of a 4th-year student (programme manager) and their two facilitators. The individual reflection was then followed by a discussion led by the programme manager, during which students shared with each other their experiences during the day in terms of planning, implementation and future planning, and their feelings. This procedure was repeated on a weekly basis during the academic calendar. Because the research focused on the individual reflection of each student, as documented in the reflection jour- nals, it indicates the changes in professional and personal competence (including knowledge, skills and attitude/emotive components) over the course of the year. The reflection journals consisted of 10 questions to guide students and were graded according to a coding scheme using a 3-point scale (1 – no reflection; 2 – inadequate response; 3 – adequate/ positive response with rationale). The journal entries documented throughout a period of 1 year were obtained; three specific entries made at the beginning of the year, during the middle of the year, and towards the end of the year were analysed and compared to determine whether any change had occurred in the students’ thinking (knowledge), skills, and attitudes (including confidence) over time. The rubric was pilot tested before use. Two researchers independently reviewed the data and obtained 80% agreement with regard to coding to increase trustworthiness. Marks assigned to each criterion were automatically calculated as percentages in Excel and an average of total scores was calculated from the three measurements taken over the year. Descriptive statistics were used to describe, summarise, and interpret the data.[12] The linear regression facility of the analysis tool pack in Excel (Microsoft, USA) was used for the analyses to measure growth across time. Growth was depicted by the slope in the graph and displayed by a change in colours ranging from red (poor), yellow (average/limited) to green (growth) on the Excel spreadsheet where the data were entered. The statistical results were supported by specific verbatim quotes. The use of a rubric allowed the data to be compared across time and among various participants. Results Change related to professional growth The results in Fig. 1 show that the reflections documented at the beginning of the academic year differed from those obtained towards the middle and end of the year. The graph depicts three measurements. Linear regression was used to fit a curve to the experimental data. A slope of the regression line signifies the degree of improvement from one phase to the next. In this case, the statistical analysis indicates that the slope of the regression line is >0% at a confidence level of 95%, i.e. even in this limited sample size a steady improvement has been noted. With reference to the structured questions in the reflection journal depicted in Section A of Table 1, results show a change over time in critical thinking and emotive (attitude) aspects in terms of all the questions in the analyses. The limited change observed in problem- solving skills addressed in Questions 9 and 10 in Section A could be attributed to these questions being the last two to be completed in the jour- nals. Time constraints in the completion of the reflection could influence the results. The results show an increase in students’ sense of social responsibility. An awareness of the needs of the community increased from 42% at the beginning of the year to 100% towards the end of the year. At the beginning of the year, only 38% of the students indicated that they would stay in the A ve ra g e sc o re fo r ea ch d at e, % 100 90 95 80 85 70 75 60 65 50 55 1 2 3 4 5 6 7 8 9 10 Question Feb. 2012 Oct. 2012 June 2012 Fig. 1. Average score per question. 180 October 2016, Vol. 8, No. 2 AJHPE Research Ta bl e 1. M at ri x fo r re fl ec ti on Se ct io ns Le ve l o f r ef le ct io n St ud en t 1 St ud en t 2 St ud en t 3 St ud en t 4 St ud en t 5 St ud en t 6 St ud en t 7 St ud en t 8 Fe br ua ry , % Ju ne , % O ct ob er , % G ro w th Se ct io n A Le ve l 1 ‘A na ly se s’: ‘W ha t? ’ W ha t k in d of ac tiv ity d id y ou en ga ge in to da y? 3 2 3 3 2 3 2 3 88 10 0 96 1 W ha t d id y ou le ar n to da y? 3 1 2 3 2 2 2 2 71 92 88 1 Le ve l 2 C ri tic al th in ki ng ‘S o w ha t? ’ W ha t d id y ou d o th at w as s uc ce ss fu l? W hy w as it su cc es sf ul ? 3 3 3 3 2 3 2 2 88 83 10 0 1 W ha t d id y ou do th at w as n ot su cc es sf ul ? W hy w as it n ot ? 3 3 3 2 2 3 2 2 83 83 92 1 W ha t d id y ou d o to co m pe ns at e fo r th e ab ov e? 2 3 3 2 2 2 2 2 75 79 83 1 D id th e co m pe ns at or y st ra te gy w or k? W hy ? 2 3 3 2 2 2 2 2 75 83 83 1 Le ve l 3 Em ot iv e as pe ct s W ha t a re y ou r fe el in gs r e th e se rv ic e- le ar ni ng ex pe ri en ce ? 2 3 2 3 2 2 3 2 79 88 10 0 1 W ha t a re y ou r fe el in gs r e yo ur se rv ic e de liv er y? 3 3 2 3 2 3 3 1 83 88 10 0 1 Le ve l 4 Pr ob le m so lv in g: ‘N ow w ha t? ’ W ha t w ill y ou d o di ff er en tly n ex t tim e? 3 3 3 2 2 2 2 2 79 92 79 2 W ha t i s th e re le va nc e of th e in te nd ed a ct io n pl an ne d? 3 3 2 2 2 2 2 2 75 92 79 1 Fe br ua ry , % 90 90 87 83 67 80 73 67 Ju ne , % 10 0 97 90 90 10 0 83 63 80 O ct ob er , % 97 63 10 0 87 10 0 97 83 93 G ro w th 1 3 1 1 1 1 1 1 co nt in ue d … October 2016, Vol. 8, No. 2 AJHPE 181 Research Ta bl e 1. (c on ti nu ed ) M at ri x fo r re fl ec ti on Se ct io ns Le ve l o f r ef le ct io n St ud en t 1 St ud en t 2 St ud en t 3 St ud en t 4 St ud en t 5 St ud en t 6 St ud en t 7 St ud en t 8 Fe br ua ry , % Ju ne , % O ct ob er , % G ro w th Se ct io n B D ev el op m en t of s oc ia l re sp on si bi lit y D oe s st ud en t s ho w an a w ar en es s fo r th e ne ed s of th e co m m un ity ? 1 3 1 1 1 1 1 1 42 33 10 0 1 D oe s st ud en t s ho w / ex pr es s a ne ed to st ay in vo lv ed in th e co m m un ity ? 1 2 1 1 1 1 1 1 38 38 96 1 D oe s st ud en t s ho w an u nd er st an di ng o f hi s/ he r co nt ri bu tio n to s oc ia l c ha ng e? 2 3 1 2 1 2 1 1 54 46 10 0 1 Fe br ua ry , % 43 90 33 43 33 43 33 33 Ju ne , % 67 33 33 43 33 33 33 33 O ct ob er , % 90 10 0 10 0 10 0 10 0 10 0 10 0 10 0 G ro w th 1 1 1 1 1 1 1 1 Se ct io n C Su m m ar y of p er ce iv ed co m pe te nc e Pe rc ei ve d un de rs ta nd in g of w ha t h e/ sh e ha s be en d oi ng m at ch es th e cu rr ic ul um ou tc om es 1 1 1 1 1 2 1 2 42 50 88 1 Pe rc ei ve d su cc es s w ith im pl em en ta tio n 3 2 2 2 2 2 2 2 71 83 10 0 1 St ud en t’s a tt itu de to s er vi ce -l ea rn in g ex pe ri en ce 3 3 2 2 2 3 3 2 83 79 10 0 1 Fe br ua ry , % 77 67 57 57 57 77 67 67 Ju ne , % 90 10 0 67 77 57 43 67 67 O ct ob er , % 77 10 0 10 0 10 0 10 0 90 10 0 10 0 G ro w th 2 1 1 1 1 1 1 1 182 October 2016, Vol. 8, No. 2 AJHPE Research community, but their views changed over the course of the year, as 96% were positive after working in this community for the entire year. Their work in the community for an extended period also made them understand how they could contribute to social change, as shown in the results that changed from 54% to 100% towards the end of the year. Considering that community awareness is an integral part of service- learning, it is important that facilitators guide their students in this regard.[3] In this study professional services were provided to young learners at a rural preschool, and although the circumstances in which they worked were dire, not all sessions were particularly aimed at increasing community awareness. An increase in social responsibility could only be noted once the students were escorted walking through the village to visit clients in their homes. In these cases the specific opportunity allowed them to look beyond the boundaries of providing their professional services and to see the needs of the wider community. Only then did they comprehend how they fit into the bigger picture and could they challenge their own beliefs and value systems. The goal of reflection is to think about the larger social issues behind the needs for which the service is provided. These issues include the social, cultural, economic, and political context of the needs to be addressed.[13] The traditional roles of lecturer and student are challenged in service learning, as the lecturer is no longer the sole expert contributing to the training, but fellow students and community members (even if uneducated) can contrib- ute to the students’ education and knowledge acquisition.[8] The following quotes emphasise how the students have developed a sense of social respon- sibility and valued their service-learning experience: ‘It is important to be a helper of people and as a person try to help where you can because as a community we all stand together and there are a lot of people that are not identified that need help.’ ‘This experience has taught me to strive to make a difference.’ ‘It is important to stay involved in this community – because the need for education and special services are very high and I know I can make a difference by helping these children.’ The results obtained in Section C of Table 1 indicate the change in the students’ perceived self-confidence. All students were of the opinion that they have become clinically competent in their service provision at the site, as these scores increased from 71% to 100%. The following demonstrates their attitudes at the end of the year: ‘I will be more able to communicate with others who can’t speak my language and collaborate with teachers to make a change. I can work better in a team when helping a client. I have more self-confidence when doing therapy and working with children. It makes me more positive about the future.’ Some students did not understand how their service-learning experience related to the curriculum outcomes. Despite having received learning guides in which the relevance is explained, students tend not to read them. Although change did occur for the group (42 - 88%) in terms of whether their actions matched the curriculum outcomes, it is possible that some degree of uncertainty remained. It is therefore important that facilitators continuously make students aware of their curriculum outcomes at the service-learning site. Overall, the students were positive about their experience at the end of the year compared with the beginning of the year. The results show that the students were pushed beyond superficial interpretations and that their reflective competence contributed to personal growth, civic responsiveness, and critical thinking. Evaluating the usefulness of the assessment tool The strength of the assessment tool was that it provided answers for programme evaluation (quality control). The tool indicates change and was relatively quick and easy to use once the scorers were familiar with the procedure. Reliability therefore depends on how familiar the scorer is with the rubric, which is determined by the frequency of the measurements. Once the scorers had familiarised themselves with the tool, there was an increase in reliability. The 3-point scoring matrix used in this study was adequate for providing a general indication of whether change had occurred. Future research may opt for a 5-point scale to obtain a better understanding of a student’s reflective competence (e.g. surface to moderately deep reflection to depict a student not looking beyond the particular experience, and a moderate to deep reflection for the student who looks at how the process has helped him/ her to plan for the near future).[14] Although this study focused on the use of the rubric as a tool to assess whether there was any professional and personal growth, the particular reflection practice used at this site also came under scrutiny. Criticism from the literature clearly indicates that individual student reflection may be inadequate and one-sided, because students’ established thoughts are not challenged and they are not stimulated to think beyond their original viewpoints. Reflective journal notes in individual reflection may, however, be judgemental as opposed to a wider understanding and a rethinking of ideas, as evident in reflective interaction with other students.[8] When reflection takes place in small groups, ideas may be generated by the sharing of different perspectives. At our specific site, the individual reflection is followed by small-group reflection, with verbal feedback by the facilitators to the entire group. The research emphasised the need for facilitators to also provide individual feedback on the students’ written reflections to further guide them on how to critically reflect. It is therefore suggested that such feedback be provided during supervisor feedback sessions when students are assessed on their clinical skills. The use of the rubric for multiple measurements over time to assess written journal entries was time consuming for the facilitators. Dalal et al.[14] found the use of a rubric in peer assessment of e-portfolios to be effective, which is a possibility that should be investigated for future use. This implies that journal entries will be completed after the service-learning experience when the students access their computers. It also implies that students would have had the opportunity to reflect as a group before they are allowed to reflect individually, which may contribute to their growth. Factors affecting reflection and completion of the journals Specific factors emerged from analysing the journal entries that could have affected the manner in which the students interpreted their practical experiences or documented their impressions. Some students were succinct when writing their reflections, while others expanded and expressed themselves more eloquently. For some students the writing exercises were easy, while others wrote more laboriously and required more time to formulate their thoughts.[8] Language proficiency (English is not the students’ first language, but an additional language) may therefore also be a factor to consider when analysing reflection. The structure provided by the 10 questions, however, made the writing experience easier. The October 2016, Vol. 8, No. 2 AJHPE 183 Research questions made them describe their experiences, consider the needs of the community, and relate the experiences to themselves. Guided reflection therefore contributed to their professional and personal growth. It became evident from reading the journal entries that diversity is related to the manner in which students reflect. Students of different cultures, race, and socioeconomic backgrounds work together at practical sites, which has an effect on how they construct meaning from their practical experi- ences.[8] From scrutinising and relating the reflections to the individual students, it appears as if those from disadvantaged backgrounds were less affected by poverty in this specific community than their peers from more affluent cir- cumstances, who were in the same group. From this limited sample it seems as if the background of the students and the context in which the service is provided influence the manner in which the students reflect. The individual reflections portrayed how the students felt directly after their experience. The following quotes show how their feelings were largely influenced by how well they were prepared for providing the service, as those who were well prepared reflected a more positive attitude, and vice versa: ‘Be more prepared and not look like a fool again. Need to pick-up my sox. I can do better than this.’ ‘Not being totally prepared is a waste of time to yourself and others that you are helping. Life is knocking.’ Whenever students embarked on a new task, they often felt unsure and anxious, which reflected in their journals. At the beginning of the year most 2nd-year students have not acquired the clinical skills to meet the learning outcomes, but they developed these over the course of the year. Furthermore, their reflections were related to real-life experiences, which is why they experienced some days to be better than others. Their reflections when describing the activity therefore changed as they became more competent and gained more clinical skills. Further limitations were time-related factors, which led to no response to some of the journal questions. Students were allowed 30 minutes for their reflection, but as they were dependent on communal transport leaving at a specific time, those who started late tended to omit certain questions. To increase the response rate it is important that all students simultaneously and strictly keep to the time restrictions. The data showed a discrepancy between the different students, as some performed better than others. Not all students develop at the same pace and development can therefore be placed on a continuum. This is probably because reflection skills develop slowly over time from adolescence to adulthood.[15] When a task was new or difficult to perform (e.g. assessment), students responded less positively in their reflection, and some reflected negatively because of their incompetence at the time. Once they became more skilled in that task, they became more confident, which was reflected in their journal entries. Unfortunately, the nature of the service-learning experience allowed for some tasks to be performed only at the onset of the year, when students still felt unsure about their competence. It is therefore recommended that the researchers compare results obtained at the start of the year over several years and not only across time for a specific year to obtain more reliable results. Recommendations The two facilitators refrained from allocating marks for the reflections to avoid the Hawthorne effect[11] (i.e. students trying to impress the supervisor). However, it is recommended that such reflections be used by facilitators in their feedback to students, which could ultimately affect the final mark for practical work, and could also contribute towards formative assessment. Assessment of reflection in supervisor feedback should be conducted on a regular basis.[10] Although this rubric shows potential for use in determining professional growth at a service-learning site, it should be further developed. It is recommended that a larger sample be selected, and that the study be implemented at various sites to increase the dependability of the research. It is suggested that it should also be compared with other tools and across different cohorts. References 1. Bringle RG, Hatcher JA. Introduction to Service-Learning Toolkit: Reading and Resources for Faculty. Providence, RI: Campus Compact, 2003. 2. Eyler J, Giles DE, Schmiede AA. A Practitioner’s Guide to Reflection in Service-Learning. Nashville, TN: Vanderbilt University, 1996. 3. Ash SL, Clayton PH. The articulated learning: An approach to guided reflection and assessment. Innovative Higher Educ 2004;29(2):137-154. DOI:10.1023/b:ihie.0000048795.84634.4a  4. Hopkins D. Improving the quality of teaching and learning. Support for Learning 1997;12(4):162-165. DOI:10.1111/1467-9604.00038  5. Hatcher JA, Bringle RG. Reflections: Bridging the gap between service and learning. J Coll Teach 1997;45:153. DOI:10.1080/87567559709596221 6. Rogers R. Reflection in higher education: A concept analysis. Innovative Higher Educ 2001;26(1):37-57. DOI:10.1023/A:1010986404527 7. Rice K. Engaging all partners in reflection: Designing and implementing integrative reflection opporunities, 2010. suu.edu/servelearn/pdf/RiceReflectionPacket.pdf (accessed 11 July 2016). 8. Petersen N, Osman R. An introduction to service learning in South Africa. In: Osman R, Petersen N, eds. Service Learning in South Africa. Cape Town: Oxford, 2013. 9. Bender G, Daniels P, Lazarus J, Naude L, Kalawathie S. Service-Learning in the Curriculum: A Resource for Higher Education Institutions. Pretoria: Council on Higher Education, 2006. 10. Molee LM, Henry ME, Sessa VI, McKinney-Prupis ER. Assessing learning in service-learning courses through critical reflection. J Experiential Educ 2010;33(3):239-257. DOI:10.5193/jee33.3.239 11. Leedy PD, Ormrod JE. Practical Research: Planning and Design. 9th ed. Boston: Pearson, 2010. 12. Johnson RB, Christensen LB. Educational Research: Quantitative, Qualitative, and Mixed Approaches. 2nd ed. Boston, MA: Allyn and Bacon, 2004. 13. Tsang E. Use assessment to develop service-learning reflection course materials. 32nd ASEE/IEEE Frontiers in Eduation Conference, Boston, MA, 2002. DOI:10.1109/FIE.2002.1158150 14. Dalal DK, Hakel MT, Sliter MT, Kirkendall SR. Analysis of a rubric for assessing depth of classroom reflections. Int J ePortfolio 2012;2(1):75-85. www.theijep.com/pdf/IJEP115.pdf (accessed 11 July 2016). 15. King PM, Kitchener KS. Reflective judgment: Theory and research on the development of epistemic assumptions through adulthood. Educ Psychol 2004;39(1):5-18. DOI:10.1207/s15326985ep3901_2 http://dx.doi.org/10.1023/b:ihie.0000048795.84634.4a http://dx.doi.org/10.1111/1467-9604.00038 http://dx.doi.org/10.1080/87567559709596221 http://dx.doi.org/10.1023/A:1010986404527 http://dx.doi.org/10.5193/jee33.3.239 http://dx.doi.org/10.1109/FIE.2002.1158150 www.theijep.com/pdf/IJEP115.pdf http://dx.doi.org/10.1207/s15326985ep3901_2