Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 1 The Journal of Teaching and Learning for Graduate Employability ISSN: 1838-3815 (online) Journal Homepage: https://ojs.deakin.edu.au/index.php/jtlge/ Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes Julie Dunne1 Corresponding author: Julie Dunne Julie.dunne@dit.ie 1Technological University Dublin, ORCID ID: 0000-0002-5858-2694 Abstract This study investigated the effect of activities to promote awareness of specific prioritised graduate attributes on the quality of reflection displayed in work-placement reflective blog assessments for Pharmacy-technician students. Previous related research showed an increased tendency to frame placement experiences in terms of graduate attributes when they are made more explicit in the curriculum. Now, a thematic analysis of reflective writing from control and research groups, using a priori codes of ‘reflection’ and ‘graduate attributes’ explored the impact on the depth of critical reflection, and its relationship to explicit discussion of graduate attributes. The findings show an improvement in the quality of reflection, with significantly higher levels of critical reflection in the research group (37%), compared to the control group (20%), and lower levels of the non-reflective categories of thinking such as thoughtful action. Furthermore, in 62% of research group cases, there is overlap between critical reflection references and explicit graduate attributes references. Overall, the quality of reflective writing has improved through framing experiences in terms both of graduate attributes, as well as core theoretical knowledge, leading to increased focus on personal development and consequences for future action. Keywords: graduate attributes, work- placement, Pharmacy Technician, reflective assessmen, thematic analysis, employabil ity, critical reflection, NVivo Introduction This paper discusses the results of a research investigation into the effect of emphasising graduate attributes within a curriculum on the quality of reflection in Pharmacy Technician student work- placement assessments compared to previous years. Like many higher education institutions seeking to focus on graduate employability, in 2013 Dublin Institute of Technology (DIT) (now Technological University Dublin) identified a suite of skills aligned to its academic mission as an industry focused Institution that delivers job-ready graduates (DIT, 2013). DIT requested that these be integrated into curricula, and the current research explores an aspect of this policy by analysing its impact on a work placement reflective assessment. The research in this paper builds on a recent report that demonstrated that activities that emphasise prioritised graduate attributes within a curriculum in https://ojs.deakin.edu.au/index.php/jtlge/ mailto:Julie.dunne@dit.ie Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 2 advance of entering an internship or work-placement can help students by providing a focal point to frame their work-placement experiences in their reflective assessments (Dunne, 2017). Work-placement is a period of planned work-based experiential learning, where the learning outcomes are part of a program of study and should be associated with a formal assessment (European-Commission, 2009a, 2009b). There is increasing emergent consensus that being reflective and developing reflective practice is crucial in becoming an effective healthcare professional, adding to and enhancing everyday professional values, knowledge and skills (Neville, 2018). Indeed K. Smith, Clegg, Lawrence, and Todd (2007) argue that the academic benefits of work-based experiences depend largely on the extent to which students reflect on them and relate them back to knowledge gained in theoretical modules. Students have reported that having appropriate experiences were seen to be a key part of developing reflective skills, and that staff who champion the development of reflective skills and mentor students were perceived to play key roles in aiding the development of reflective skills (Vivekananda-Schmidt et al., 2011). A comprehensive review of integrating graduate attributes into the curriculum in Australia identified challenges in setting standards, and in determining to what extent these standards are being met by students (Hughes & Barrie, 2010). It emphasised that graduate attribute assessment plans will not accomplish their aim if students are not made aware of their objectives and if students are not actively involved as partners in the assessment process. There have been many literature reports in various fields wherein attempts have been made to make more explicit within the curriculum the development and assessment of graduate attributes (Treleaven and Voola (2008) Sin and McGuigan (2013); Nicol (2010)). The involvement of peers has been especially highlighted by Stracke and Kumar (2014), who advocate the use of peer support groups as a bottom-up effort to enhance the top-down embedding of graduate attributes for research students. Of key importance is the reflective process that underpins students’ understanding and development of graduate attributes (Hughes & Barrie, 2010). Reflective practice is also recognised as an integrating pedagogical tool to support the clarification and acquisition of Threshold Concepts along with Graduate Attributes (Hyde, Flatau, & Wilson, 2018). The earlier research by the author showed that students had an increased tendency to frame their placement experiences in terms of graduate attributes, as well as core skills developed in the curriculum, when graduate attributes were made more explicit in the curriculum (Dunne, 2017). This consequently broadened their reflection and advanced their potential employability as defined by Hillage and Pollard (1998). This current paper will return to this research and consider whether the activities to emphasise graduate attributes also had an impact on the depth or level of reflection. To investigate this, it is necessary to gain an understanding of the various levels of reflection. In their seminal work, Boud, Keogh, and Walker (1985) have broken the reflective process into three stages, as outlined in Table 1. Table 1: Reflective Processes Stages of reflection Actions involved Returning to the experience Remembering it Documenting it without judgement Attending to feelings Utilising the positives Dealing with the negative ones to remove obstructing feelings e.g. laughing at embarrassment, crying, anger Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 3 Re-evaluating the experience Associating it with what is already known Integrating it by seeking relationships Validation of the authenticity of the ideas or feelings which have resulted Appropriation, that is, making the knowledge one’s own. New perspectives, changes in behaviour, and willingness to act on the new ideas follow the outcomes of these processes. In the case of work placement experiential learning, it can be argued that it is the students’ professional behaviour and actions that change, which naturally can be related to their graduate attributes. However, for students, the process of reflecting on work-based learning is not trivial. K. Smith et al. (2007) have discussed the difficulties students face when conceptualising work in terms of academic knowledge. The reverse process of transforming experiential/tacit knowledge from the workplace into a form they can verbalise is also challenging for students. They require support to enable them to achieve successful reflection. Different types of reflective skills have been described for professional judgement, which students on placement should develop, namely, reflection-in-action (during experiences) and reflection-on-action (reflection on past-action) and includes reflection-for-action (considering future action). Research shows well-designed activities and scaffolded assessment tasks involving explicit instructions, relevant support materials identified and clear assessment criteria appear to encourage higher quality reflective practice, specifically reflection-for-action (Stupans, March, & Owen, 2013). An example of supports to improve the quality of student reflection in work-placement journals has been reported by Hume (2009). he has described the use of ‘Shulman’s framework for good science teachers’ as a focal point for reflection. She found that in addition to a framework, additional pre- placement workshops and activities, along with time to practice reflective writing, use of exemplars, and clear assessment criteria were important to help with reflection while on placement. Naturally the same basic principles of reflection also apply for reflective assignments in an online environment. Teaching innovations in this area include online technologies (e.g. blogs) which provide a virtual space for reflection which can be accessed by peers and tutors alike (Chretien, Goldman, & Faselis, 2008). A review of the supports for student reflection in technology-enhanced learning identified that pre-defined guidance, human-intervention guidance and a mechanism to promote students to focus on critical points can act as a positive support for high-quality reflection (Kori, Pedaste, Leijen, & Mäeots, 2014). This aligns well to Hume’s (2009) approach. The consideration of a framework to allow students to focus on critical points for reflection is central to this current research. It asks if emphasising graduate attributes will act as a critical point for students to focus upon for their personal development while on work placement, and will this influence the depth of reflection shown in their reflective assessments? This requires a means to judge if levels of reflection have improved, such that evidence for moving from non-critical reflection to deeper critical reflection can be found. The current study based this judgement on previous work by Wallman, Kettis Lindblad, Hall, Lundmark, and Ring (2008), and has been explained further in the theoretical framework section below. The Study Theoretical framework In the earlier part of this study described in Dunne (2017), a Participatory Action Research (PAR) methodology was implemented in order to create a learning environment that connects theory and practice with action and reflection as an outcome of participating with others (Reason & Bradbury, Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 4 2008). This facilitated the students, through their participation and interactions with the tutor and each other, to develop their own understanding of the concept of graduate attributes, and become aware of and involved in the conversation. The current paper follows from this work and evaluates the impact of the PAR project on student reflection. A constructivist ontological position allowed for concepts about how graduate attributes are perceived and developed by the students to be produced through social dialogue and reflection. Given that research into how each individual student perceives graduate attributes must take into account the differences in individuals, an interpretation of their reflective assessments was required. Therefore, an interpretivist epistemological position was assumed. In their project to test the reliability, feasibility, and responsiveness of a categorization scheme for assessing Pharmacy students’ levels of reflection during internships, Wallman et al. (2008) have used a categorisation scheme that is outlined in Table 2. In the scheme, the categories build upon each other as the students reach higher levels, increasing from Habitual action to Premise reflection. Levels 4 to 6 are non-reflective, while levels 1 to 3 are reflective. See Appendix 1 for further explanation of Wallman et al. (2008) categories. Table 2: Categorisation Scheme for Assessing Pharmacy Students’ Reflection Categorisation of reflection Premise reflection Process reflection Reflective Content reflection Introspection Thoughtful action Non-reflective Habitual action Bell, Kelton, McDonagh, Mladenovic, and Morrison (2010) have further evaluated this model in the context of business students’ reflective writing, and they found it reliable. Wong, Kember, Chung, and Yan (1995) have also investigated assessment of students’ reflective journals. They have similarly developed methods of coding reflection at different levels, one more advanced than the other. They align to the Boud et al. (1985) description of reflection as described in Table 1. At a more basic level, they found that it was reliable to distinguish between the three stages of reflection. However, they found it was not possible to reliably distinguish between the actions involved within each stage. Based on this they proposed that it is possible and reliable to sort students into three categories, namely non-reflector, reflector and critical reflector. Critical reflector describes a student who attained the actions of validation and appropriation, whereas reflection is associated with the actions of integration and association. Indeed, it could be argued that both the Wallman and Wong approaches and models amount to the same thing, in that both categorise the levels of reflection into a higher and lower order of reflection. The lower order focuses on reflecting on the content or knowledge related aspects of the experience and how it relates to prior knowledge, and also to the immediate response to the experience. Meanwhile the higher level of reflection relates more to the personal development of the reflector and the capacity to change future actions based on reflecting on the experience. Consequently, no attempt was made to code levels of reflection beyond the main levels of ‘reflection’ and ‘critical reflection’, as was proved by the previously mentioned studies to be reliably reproduced (Bell et al. (2010); Wong et al. (1995)). Nonetheless, is a recognised challenge for assessing reflection to have grader consistency (M. Smith & Trede, 2013). For rigour, an independent double blind coding of a random sample of assessments was carried out. Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 5 The method chosen for the interpretation of the reflective assessments was through template analysis for thematic coding (Baldwin, 2012). Analysis of the assessments of a control group (2013/14) and research group (2014/15) was based on a hypothesis driven confirmatory approach using the a priori codes of ‘Reflection’ and ‘Graduate Attributes’ (King, 2014). This was selected to accelerate the coding process to specifically address the research question which sought to investigate any relationships between the two themes. Participants in the study The control group (CG) and research group (RG) involved in this study were all the students registered in the final year of the DIT Higher Certificate in Pharmacy Technician Studies in the two consecutive years of 2013/14 for the control group, and 2014/15 for the research group. There were no incidents of students belonging to both groups – each group was entirely independent of the other. Other than the activities included in the study described by this paper that were implemented for the research group in order to address the inclusion of Graduate Attributes more explicitly in the curriculum in accordance with DIT policy, there were no other significant changes to the curriculum, nor those teaching it, during the years involved. Both groups also had the same workshop to support reflective writing, as previously reported (Dunne & Ryan, 2016). Implementation The implementation of this Participatory Action Research project has been described in the earlier study (Dunne, 2017). The research involved: Knowledge development: The students’ baseline knowledge and confidence about the DIT graduate attributes (DIT, 2014b) was examined. Their ability to articulate or define the various attributes was developed through group exercises to create definitions for each attribute. To assist with this, the students were each furnished with several relevant examples of Graduate Attributes definitions. These included information from the University of Sheffield careers service (Careers-Department, 2014), the HE STEM Employability Skills Review (Toland, 2011), the Open University careers service resources, and resources from the University of Aberdeen Centre for Academic Development (Aberdeen, 2014). Graduate attributes prioritisation: A stakeholder survey was used to prioritise the graduate attributes. Further details about the survey design, which was based on the Australian Graduate Employability Indicators survey are available (Dunne, 2015). The prioritised attributes are outlined in Table 2.3. Graduate attributes development: The Assuring Graduate Capabilities website (Oliver, 2011) was used to identify suitable learning activities to develop graduate attributes. A series of online self-evaluation exercises to allow students to determine their strengths and weaknesses in skills to date. The links to self-assessments from the University of Kent ("Analyse your Employability Skills," 2015; "Employability Skills Exercise,") were sent to the students to complete in their own time. A link to communications resources from the University of Kent ("Communication Skills: Speaking and Listening," 2015). The link to the survey was sent to the students to carry out in their own time. An ethics debate centred on a short ethical case study relevant to the role of the pharmacy-technician, organised based on a modified version of the ‘think-pair-share’ format reviewed in Kennedy (2007). A separate description and evaluation of our introduction of this debate in the curriculum has been reported (Hogan & Dunne, 2018). Reflective assessment analysis: Coding of reflective blogs to determine any differences in breadth and depth of reflection. The coding supported by NVivo is explained below. Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 6 Analysis of data using NVivo While qualitative research is not given to mathematical analysis, it is nonetheless systematic in its approach to data collection and analysis. In this research, the process involved breaking down the data into discrete ‘units’ and coding them to categories according to pre-defined a priori codes. Coding of the 2013/14 and 2014/15 reflective blogs was carried out using NVivo software to support the qualitative analysis. While the researcher carried out the hermeneutic task, NVivo was used as a tool for efficiency and transparency of coding, and to query the data during the analysis. The coding in relation to the results in this paper sought to look for evidence of increased emphasis on graduate attributes in student reflection, as well as level or depth of reflection. There is some numerical aspect to the analysis, and some statistical analysis of the results using a Chi Square test to examine association between the level of reflection and the class group has been carried out. However, the coding has not been analysed with regard to word count being an indicator of level of discussion in relation to any of the given topics. The data was instead used to uncover broad trends in the assessments, to gain an understanding of any changes that may have occurred in the blog writings of the research group compared to the control group. This is appropriate owing to the qualitative nature of the study, as well as for robustness given the delimitation that only a random sample of the data has been double blind coded. The blog sources were individually coded, initially for the a priori codes of graduate attributes and reflection. Any reference to either code, whether explicit or implicit and at any level was coded using the NVivo software. Many references were coded to both categories, whilst many more were coded to one category only.The phases of the analysis were as follows: Phase 1: Extracting the data from individual blog posts from the research and control groups into individual Microsoft Word documents and uploading these as sources in NVivo as ‘case nodes’. The control group comprised of 26 student case nodes and 78 blog sources, while the research group consisted of 29 student case nodes and 87 blog sources. For anonymity when using extracts from their blogs, the students have been assigned the codes CG1-CG26 and RG1-RG29 respectively. Phase 2: ‘Open coding’ each blog post case node into the primary NVivo ‘nodes’ of Graduate Attributes and Reflection, according to the NVivo Codebook ( Table ). Phase 3: ‘Coding on’ the primary nodes into sub-nodes, according to the NVivo Codebook. Note, during the early analysis phase, a further coding-on was carried out to reduce the explicit references to graduate attributes into ‘explicit but unnamed’, and ‘explicit and named’. Phase 4: Running NVivo matrix queries on the coded data to establish trends and relationships. Phase 5: Comparison of sample double coded assessments for agreement of coding. Table 3: Section of NVivo Codebook for Thematic Analysis of Control and Research Blogs with Respect to Depth of Reflection and Graduate Attributes Statement Type Open code Sub-code Code description Reflection All references to reflection at any level Level of Reflection: Thoughtful action or Introspection References to reflective comments that are better described as thoughtful action or Introspection (Wallman, 2008) Non-critical reflection References to reflective comments that are non-critical in nature, as per 'content reflection' (Wallman, 2008) Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 7 Critical reflection Higher order reflection, Process reflection and premise reflection (Wallman, 2008) Graduate attributes All references to graduate attributes explicit and implicit Statement type: Implicit References to graduate attributes that have been interpreted by the researcher rather than explicitly labelled by the blog author. Explicit References to graduate attributes that have been explicitly acknowledged by the blog author, either by using the Graduate attribute term itself or clearly alluding to the skill as has been defined for this project. Interpretation by the researcher was not required. Limitations, delimitations and ethics This research was delimited to the students in DT425/2 Higher Certificate in Pharmacy Technician Studies (DIT), 2013/14 as the control group for comparison. The study is limited to the cohort of students registered to the module TFIP2001 Pharmacy Work-placement DIT (N=29). Owing to the nature of the action research based methodology, there is an inherent deficit in the study because the research group and control group were not sampled from the same cohort, but rather individual cohorts in their final year of the same course in consecutive years. The study was delimited to a single primary coder, namely the author of the study, with a representative sample of double coding by an academic unrelated to the project for rigour. The level of coding and analysis was delimited to that which is considered, through previous studies discussed in the literature review, to be reliably reproducible (Wallman et al. (2008); Bell et al. (2010); Wong et al. (1995)). The project has received ethical clearance from the DIT Research Ethics committee (Ref no. 14-68). Findings and discussion The main research question of this study was to establish the effect of activities to promote awareness of graduate attributes development on the quality of reflection displayed in student pharmacy- technicians’ reflective assessment blogs compared to previous years. Impact on reflection (1): Range of graduate attributes discussed The aforementioned previous report by the author, focusing on the quality in terms of the breadth of reflection on prioritised graduate attributes, revealed that the research group showed a considerably greater likelihood to reflect on a wider range of graduate attributes. The report also demonstrated that this reflection was also much more likely to refer to the development of the attribute or skill in an explicit manner. This improves the students’ employability by providing them with concrete examples of transferable skills that they can clearly articulate to prospective employers (Dunne, 2017). These results have been summarised in Table 4. Table 4: Analysis of most Commonly Referenced Graduate Attributes for Explicit vs Implicit statements by Research vs Control group Graduate Attribute Control Group Research Group Communication Explicit 11 30 Implicit 56 36 Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 8 Emotional intelligence Explicit 8 13 Implicit 42 34 Motivation Explicit 2 20 Implicit 6 12 Work related learner Explicit 7 18 Implicit 5 7 For context, the following blog examples illustrate the difference between implicit and explicit references to the ‘communications’ graduate attributes. Typical implicit references to communication are shown below, and really reflect the student retelling of the day to day events of the pharmacy which involve their interaction with customers, and other members of the pharmacy team: Just to make sure that it would be suitable with the antibiotic I double checked what I was giving to the girl by telling the pharmacist what the problem was and what I suggested for her to take [Control Group no. 4]. I told him that a generic version of a medicine may be a different size or shape, have different colours, flavours or combinations of non-active ingredients compared to the original product, but that none of these differences affect the way the medicine works [Control Group no.1]. Whilst characteristic explicit references include the following, which fall into named and unnamed categories respectively: While looking back on the situation I realise how important communication skills are especially when dealing with the elderly as things need to be explained clearly and examples must be shown if necessary [Research Group no.3]. I am strongly of the opinion that as healthcare professionals, we need to know the audience to whom we are delivering advice to. It is important to alter the way you deliver information to each individual [Research Group no.4]. Impact on reflection (2): Level of critical reflection This current report will now focus on investigating the more complex question of whether this increase in awareness and articulation of graduate attributes is associated with an increased level of critical reflection. In the next section of findings, the coding for levels of reflection will be considered generally, and following that any influence of discussion of graduate attributes linked to level of reflection will be explored. In total, 364 references were coded to the primary ‘Reflection’ code. To gain an overall appreciation of the level of reflection shown in the entire set of blogs analysed for this study, these references have been grouped according to the categories Thoughtful Action (non-reflection), Reflection, and Critical Reflection in Table 5. Table 5: Breakdown of Reflective References (Research and Control Group Combined) by Category Category No. References Critical reflection 106 Non-critical reflection 190 Thoughtful action or Introspection 68 Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 9 As can be seen, there is a higher number of non-critical reflective references, compared to critical reflection. In the next phase of analysis, an NVivo matrix query was run to determine if there was any difference in level of reflection between the control and research groups, and the results are shown in Table 6. Table 6: Breakdown of Reflection by Category between Research and Control Group Control Group references (% of total CG references) Research Group references (% of total RG references) Critical reflection 33 (20%) 73 (37%) Non-critical reflection 90 (55%) 100 (50%) Thoughtful action 42 (25%) 26 (13%) A chi-square test showed a strongly significant association (χ2= 8.06E-13, df = 2, P=0.103*) between the distribution of references across the categories of critical reflection, non-critical reflection and thoughtful action, and the class group (control or research). These results clearly show the trend in this association is in the direction of improvement in the quality of reflection, with higher levels of critical reflection in the research group, compared to the control group, and lower levels of thoughtful action. Hillage and Pollard (1998) explains that students’ communicative skills in reflection can be developed through group discussions, manipulation of materials in response to a stimulus, simple role-plays or simulated interviews. These can refine reflective skills in a supportive environment, and can lead (amongst other outcomes) to more sophisticated written reflective essays (Ryan, 2011). It is possible that the research group have become more forthcoming in their reflections through participation in the earlier cycles of action research, including discussions to define the graduate attributes, and participation in the oral debate (Hogan & Dunne, 2018). Impact on reflection (3): Relationship between explicit graduate attributes and level of reflection The main research question was to establish the effect of activities to promote awareness of graduate attributes development on the quality of reflection displayed in student pharmacy-technicians’ reflective assessments compared to previous years. It is interesting at this point, therefore, to investigate if the apparent increase in levels of critical reflection in the research group (Table 6) can be linked to the increased levels of explicit discussion of the graduate attributes shown earlier in the findings to be the case for this group (Table 4) (Dunne, 2017). The result from an NVivo matrix query to determine the level of overlap between the critical reflection references and the graduate attributes statement references is shown in Figure 1. This shows that of the 73 critical reflection references made by the research group, 45 (62%) were also coded to explicit graduate attributes. A further 13 (18%) were also coded to the implicit graduate attributes category. The breakdown of this by attribute is shown in Figure 1. Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 10 Figure 1: Matrix Query of Critical Reflection References: Relationship between Explicit and Implicit Reference Statements by Control Group and Research Group An example of this overlap is shown in the following blog extract: Reflecting back on this situation, I can see why I felt stressed and my levels of concentration were low as I was put under pressure to perform a task in a short amount of time in a busy and crowded dispensary, knowing the lady is waiting. Since this incident, I have learned to manage my time no matter what the situation in order to prevent errors such as this one and ask for help if I feel under pressure completing a task. It also shows the necessity for accuracy and organisation in order to avoid error. I also wish I had used better communication skills, explaining to the lady that she should collect her medication the following day which would of gave me time, space and concentration to prepare it correctly. Due to this incident causing a lot of stress and disappointment, I have learnt a lot from it and my mistakes. This extract shows critical reflection that has been related to personal development, and describes how future action can be altered based on reflection on the experience and on the graduate attributes, as well as competencies, required to adequately complete the task in the workplace. It also shows the importance of work-placement in the curriculum, as this scenario simply could not be replicated in a college environment. This extract is typical of many in this category, whereby the students have been able to enhance and deepen the reflection of a workplace experience or event by framing the experience in terms of graduate attributes, as opposed to knowledge or competencies in isolation. This aligns to the explanation of critical reflection by Kalantzis and Cope (2008) in Dunne (2017), who discuss it in the context of a socio-cognitive process, involving experiencing new ideas, contexts or behaviours and making sense of them in light of what is already known or experienced, by identifying with them in relation to our existing schemas or foundation knowledge; analysing their underlying features and how they sit within the broader context (for example social, cultural, professional), so that this new knowledge can be applied appropriately in different contexts. Arguably, this change in emphasis and level of critical reflection in student assessments has been brought about by the action research project that has served to raise awareness and improve student articulation of skills, as well as highlighting to them the importance of these skills for their employability. Furthermore, the combination of activities mentioned in the Implementation section that were designed to highlight graduate attributes, as well as a means for students to articulate and find evidence for their development will improve the quality assurance around graduate attribute type learning outcomes, as advocated by Ryan (2011). 0 10 20 30 40 50 Explicit Implicit Control Group Research Group Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 11 Conclusions and future work Overall, this action research project has been successful in demonstrating that activities to promote awareness and develop graduate attributes can improve the quality of reflection displayed in work- placement assessments. The analysis of the reflective blog assessments shows that there has been a change in the nature of reflection compared to previous years. There has been a substantial increase in the breadth of attributes being reflected upon, with communication, emotional intelligence, work- related learner, and motivation discussed most often. Additionally, these have been explicitly identified considerably more frequently as part of the reflective process compared to previous years. This has served to provide students with suitable examples and experience in articulating their skills, which should enhance their employability. On the whole, the quality of reflective writing has improved. Non-critical lower order reflection has given way to a significant increase in deeper critical reflection in the research group cohort, characterised by relating experiences to personal development and consequences for future action. This has often arisen because the students have been able to enhance and deepen the reflection of a workplace experience or event by framing the experience in terms of transferable skills, as well as core technical skills and knowledge. Arguably, this change has been brought about by the action research project. The nature of the blog assessment allows students to share experiences and application of knowledge through reading and commenting on each other’s blogs, however the research group have had additional opportunity to discuss and debate with their peers the development of graduate attributes and learn from sharing suitable examples that demonstrate these. It is anticipated that an overall curriculum review is now required, to include opportunities to develop, be assessed and document experiences relating to graduate attributes from the earliest stages of the program, for example in a Professional Development ePortfolio module. Acknowledgements The author wishes to thank the students of the DIT Pharmacy Technician Studies Program and the DIT Learning, Teaching and Technology Centre for their contributions to this research. References Aberdeen. (2014). Graduate Attributes. Retrieved from http://www.abdn.ac.uk/graduateattributes/ - attributes Analyse your Employability Skills. (2015). Skills Inventory. Retrieved from http://www.kent.ac.uk/careers/sk/skillsinventory.html Baldwin, M. (2012). Participatory action research. London: SAGE. Bell, A., Kelton, J., McDonagh, N., Mladenovic, R., & Morrison, K. (2010). A critical evaluation of the usefulness of a coding scheme to categorise levels of reflective thinking. Assessment & Evaluation in Higher Education, 36(7), 797-815. doi:10.1080/02602938.2010.488795 Boud, D. J., Keogh, R., & Walker, D. (1985). Reflection: Turning Experience Into Learning: Kogan Page. Careers-Department. (2014). Description of Skills. Retrieved from https://www.careers.dept.shef.ac.uk/pdf/skillslist.pdf Chretien, K., Goldman, E., & Faselis, C. (2008). The reflective writing class blog: using technology to promote reflection and professional development. J Gen Intern Med, 23(12), 2066-2070. doi:10.1007/s11606-008-0796-5 Communication Skills: Speaking and Listening. (2015). Retrieved from http://www.kent.ac.uk/careers/sk/communicating.htm DIT. (2013). DIT Graduate Attributes. Retrieved from http://www.dit.ie/lttc/aboutthelttc/lttcspotlight/ditgraduateattributes/. http://www.abdn.ac.uk/graduateattributes/#attributes http://www.abdn.ac.uk/graduateattributes/#attributes http://www.kent.ac.uk/careers/sk/skillsinventory.html https://www.careers.dept.shef.ac.uk/pdf/skillslist.pdf http://www.kent.ac.uk/careers/sk/communicating.htm http://www.dit.ie/lttc/aboutthelttc/lttcspotlight/ditgraduateattributes/ Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 12 DIT. (2014a). DIT Ethical Guidelines. Retrieved from http://www.dit.ie/researchandenterprise/researchatdit/ethicsindit/content/guidelines/ DIT. (2014b). DIT Graduate Attributes. Retrieved from http://www.dit.ie/teaching/graduateattributes/ Dunne, J. (2015). Participatory Action Research: Effect of Emphasising Graduate Attributes on Work- Placement Reflection. (MA in Higher Education), Dublin Institute of Technology Dublin. Dunne, J. (2017). Work placement reflective assessments and employability enhanced through highlighting graduate attributes. Journal of Teaching and Learning for Graduate Employability, 8(1), 40-59. Dunne, J., & Ryan, S. M. (2016). Enhancing Professional Development and Supporting Students on Work- Placement by Peer-Peer Learning Using an Online Reflective Blog Assessment. Irish Journal of Academic Practice, 5(1), 1-25. Employability Skills Exercise. Retrieved from http://www.kent.ac.uk/careers/sk/skillstest.html European-Commission. (2009a). Bologna Process information site. Retrieved from http://ec.europa.eu/education/policy/higher-education/bologna-process_en.htm European-Commission. (2009b). ECTS Users Guide. Retrieved from doi:10.2766/88064 Hillage, J., & Pollard, E. (1998). Employability: developing a framework for policy analysis. Retrieved from https://www.academia.edu/23029127/Employability_developing_a_framework_for_policy_analysis Hogan, S., & Dunne, J. (2018). Evaluating the Effectiveness of a Focused Debate on the Development of Ethical Reasoning Skills in Pharmacy Technician Students. American Journal of Pharmaceutical Education, 82(6), Article 6280. Hughes, C., & Barrie, S. C. (2010). Influences on the assessment of graduate attributes in higher education. Assessment & Evaluation in Higher Education, 35, 325-334. Hume, A. (2009). Promoting higher levels of reflective writing in student journals. Higher Education Research & Development, 28(3), 247-260. doi:doi: 10.1080/07294360902839859 Hyde, S., Flatau, A., & Wilson, D. (2018). Integrating threshold concepts with reflective practice: Discussing a theory‐based approach for curriculum refinement in dental education. European Journal of Dental Education, 22(4), 687-697. doi:10.1111/eje.12380 Kalantzis, M., & Cope, B. (2008). New learning: Elements of a science of education. Cambridge: Cambridge University Press. Kennedy, R. (2007). In-Class Debates: Fertile Ground for Active Learning and the Cultivation of Critical Thinking and Oral Communication Skills. International Journal of Teaching and Learning in Higher Education 19(2), 183-190. King, N. (2014). Template Analysis. Retrieved from http://www.hud.ac.uk/hhs/research/template- analysis/ Kori, K., Pedaste, M., Leijen, A., & Mäeots, M. (2014). Supporting reflection in technology-enhanced learning. Educational Research Review, 11, 45-55. Neville, P. (2018). Introducing dental students to reflective practice: a dental educator’s reflections. Reflective Practice, 19(2), 278-290. doi:10.1080/14623943.2018.1437400 Nicol, D. (2010). The foundation for Graduate Attributes: developing self-regulation through self and peer assessment. Retrieved from Quality Assurance Agency for Higher Education: http://www.enhancementthemes.ac.uk/themes/21stCGraduates/outcomes.asp Oliver, B. (2011). Assuring Graduate Outcomes. Retrieved from Strawberry Hills, Sydney: Reason, P., & Bradbury, H. (2008). The Sage Handbook of Action Research: Participative Inquiry and Practice (2 ed.). London: Sage. Ryan, M. (2011). Conceptualising and teaching discursive and performative reflection in higher education. Studies in Continuing Education, 34(2), 207-223. doi:10.1080/0158037X.2011.611799 Sin, S., & McGuigan, N. (2013). Fit for Purpose: A Framework for Developing and Assessing Complex Graduate Attributes in a Changing Higher Education Environment. Accounting Education, 22(6), 522- 543. doi:10.1080/09639284.2013.847320 Smith, K., Clegg, S., Lawrence, E., & Todd, M. J. (2007). The challenges of reflection: students learning from work placements. Innovations in Education and Teaching International, 44(2), 131-141. Smith, M., & Trede, F. (2013). Practical concerns of educators assessing reflections of physiotherapy students. Physical Therapy Reviews, 18(6), 445-451. doi:10.1179/1743288X13Y.0000000099 http://www.dit.ie/researchandenterprise/researchatdit/ethicsindit/content/guidelines/ http://www.dit.ie/teaching/graduateattributes/ http://www.kent.ac.uk/careers/sk/skillstest.html http://ec.europa.eu/education/policy/higher-education/bologna-process_en.htm http://www.hud.ac.uk/hhs/research/template-analysis/ http://www.hud.ac.uk/hhs/research/template-analysis/ http://www.enhancementthemes.ac.uk/themes/21stCGraduates/outcomes.asp Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 13 Stracke, E., & Kumar, V. (2014). Realising graduate attributes in the research degree: the role of peer support groups. Teaching in Higher Education, 19(6), 616-629. doi:10.1080/13562517.2014.901955 Stupans, I., March, G., & Owen, S. (2013). Enhancing learning in clinical placements: reflective practice, self-assessment, rubrics and scaffolding. Assessment & Evaluation in Higher Education, 38(5), 507– 519. doi:10.1080/02602938.2012.658017 Toland, A. (2011). HE STEM Employability Skills Review. Retrieved from file:///C:/Users/Study/AppData/Local/Temp/he-stem-employability-skills-review.pdf Treleaven, L., & Voola, R. (2008). Integrating the development of graduate attributes through constructive alignment. Journal of Marketing Education, 30(2), 160-173. Vivekananda-Schmidt, P., Marshall, M., Stark, P., Mckendree, J., Sandars, J., & Smithson, S. (2011). Lessons from medical students’ perceptions of learning reflective skills: A multi-institutional study. Medical Teacher, 33(10), 846–850. doi:10.3109/0142159X.2011.577120 Wallman, A., Kettis Lindblad, A., Hall, S., Lundmark, A., & Ring, L. (2008). A Categorization Scheme for Assessing Pharmacy Students’ Levels of Reflection During Internships. American Journal of Pharmaceutical Education, 72(1), Article 05. Wong, F. K. Y., Kember, D., Chung, L. Y. F., & Yan, L. (1995). Assessing the level of student reflection from reflective journals. Journal of Advanced Nursing, 22(1), 48-57. doi:10.1111/1365-2648.ep8542692 Dunne, J. (2019). Improved levels of critical reflection in Pharmacy Technician student work-placement assessments through emphasising graduate attributes. Journal of Teaching and Learning for Graduate Employability, 10(2), 1-14. 14 Appendix 1: A Categorization Scheme for Assessing Pharmacy Students’ Levels of Reflection During Internships (Wallman et al., 2008) 6. Non-reflection: Habitual action. Habitual action is an unconscious act that takes place without thought and can be performed at the same time as another act. A description of an act performed without thought or having to focus could be, for example, writing using a keyboard (for a skilled typist). A description of the course of events can be categorized as habitual action. 5. Thoughtful action: Thoughtful action draws upon existing knowledge. The starting point lies in previously existing knowledge, and choices between different alternatives regarding how to perform the task are made either unconsciously or not at all. Why a certain choice is made is not questioned and no interpretation is made. No thought is given to the consequences of the act except according to the previously learned action. 4. Introspection: Introspection refers to thoughts about oneself, one’s own thoughts or feelings about performing a task. There is no comparison between the actual task and/or one’s previous experiences, nor are there any thoughts as to why these feelings occur or what they might lead to. Reflection. The definition of reflection as it is used below, is that a situation is identified in relation to an actual experience. This problem must somehow be analysed in order for the task to be executable. Previous knowledge is used in the specific situation and is questioned and criticized when necessary. 3. Content reflection: Content reflection pertains to what one perceives, thinks, or feels, or how one acts when doing a task. There should be a questioning or an interpretation of behaviour in order to be categorized as reflection, otherwise it is most often categorized as ‘5. Thoughtful action’. Content reflection, on the other hand, is based on a person’s previous knowledge or a previous experience and the person consciously thinks of what he/she does in order to solve the actual problem. They do not, however, reflect upon why the action taken works or how their own behaviour developed. What effect the thought, feeling, or act may have should be discussed. 2. Process reflection: Process reflection refers to how one performs the functions of perceiving, thinking, feeling, or acting, and to an assessment of how effective the performance is. There should be a proposal for, or an interpretation of, behaviour for a categorization as process reflection. For example, a person smiles to solve the problem, but also thinks further on how he or she thinks it might work out. They also consider how a kind reception from another person can reduce their own irritation. Reflection of process can also contain reflection of how they feel and act themselves when they meet the angry customer, and how this is considered as a problem, as well as how they handle their own feelings. One’s thoughts and beliefs about how the thought, feeling, or act has an effect should be discussed in addition to how others apprehend the act. 1. Premise reflection (Theoretical reflection): Premise reflection relates to why one apprehends, thinks, feels, or acts the way one does and the consequences of that existing knowledge sets the framework for how one acts in different situations. This should include an analysis of the whole situation/problem; ‘what’ and ‘how’ should be put into context. Consequences should be considered so that they can be included in a deeper understanding or reinterpretation of the problem. Alternative methods should also be considered, often leading to questioning of prejudice based on a theoretical reasoning. This could lead to a reinterpretation of the situation so that the starting point is different the next time thes same kind of problem occurs, and thus the action becomes different. This can be very hard to identify in written essays; the behaviour must be controlled the next time it happens.