Case Studies Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 7, 2013 1 Student collaboration in developing an on-line self assessment tool, to enhance development for student and newly qualified professionals Julie Laxton, University of Leeds; Paul Dagg, University of Huddersfield; Janet Hargreaves, University of Huddersfield; Duane Laverick, University of Huddersfield; and Carrie Mitchell, University of Huddersfield Abstract A “Competence in Practice” (CiPA) self assessment tool was developed initially as a research instrument as part of the Centre for Excellence in Teaching and Learning; Assessment and Learning in Practice Settings. The tool asks a series of questions against which participants self-rate their preparedness for practice. A working group then created software from this that is used as a formative exercise to rate confidence and perceived competence in practice. This freely available package (http://cipa.hud.ac.uk ) responds with feedback to individual users on their self evaluation, facilitating reflection and guided supervision. The development involved an innovative, collaborative partnership with academics and students, who undertook the work as a paid project as well as actively participating in workshops and conferences. Working collaboratively was a powerful experience for all involved. This case study presents the development of the tool, focusing on the collaborative aspects of the work. Working collaboratively with students in this manner enabled many positive outcomes and lessons for the future. Introduction The purpose of this case study is to describe the methods used to develop an existing research tool into an easily accessible self-assessment tool for developing or recently qualified professionals within health and social care, and to share our experiences of student collaboration in that project. The Assessment and Learning in Practice Settings - Centre for Excellence in Teaching and Learning (ALPS-CETL) involves five UK universities (the universities of Bradford, Huddersfield, Leeds, Leeds Metropolitan, and York St John) and its National Health Service (NHS) and Social Care partners comprising the clinical and social care networks, professional and statutory regulatory bodies, and 16 health and social care professional groups. The ALPS-CETL programme set out to nurture innovation and change in higher education learning and teaching, rewarding departments that could identify a strong programme of development. It was successful in developing a range of assessment tools and other artefacts, and the network of universities and clinical organisations involved with ALPS strived to create collaborative projects that could enable this. The regional Strategic Health Authority (for Yorkshire and the Humber) offered follow-on funding for one year after the end of the ALPS programme, which was used to incentivise projects that maintained and developed the ALPS outputs and collaborative work. One of the unique features of the ALPS programme was that the developments involved collaboration across 16 different health and social care professions and therefore combined http://cipa.hud.ac.uk/ Case Studies Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 7, 2013 2 innovative learning and teaching with many different practices and professional requirements. The ALPS team set out to explore students’ understanding of their competence to practice. Whilst some authors question the ability of students to self –assess (see for example Baxter et al 2011; Kruger et al 2002) the Competency in Practice Assessment (CiPA) tool was primarily developed as a research instrument. Initially it was piloted as a self-rating tool with new graduates with the intention of extending its use by employers of new graduates. The tool offered a way of establishing some baseline measures of confidence and competence, where these did not currently exist. It also enabled comparison of confidence and competence levels in and between different professions. It focused on the agreed ALPS common competences such as communication, team-working and ethical and professional practice, (Holt 2010). The CiPA tool was subsequently rigorously calibrated to ensure it was reliable and valid across a multi-professional group of undergraduate health and social care students, as part of the original ALPS CiPA work. The re-use of the CiPA research instrument as the basis for a reflective self assessment tool was the focus of a collaborative follow-on project, which in turn has been subject to initial evaluation. Figure One records the timeline for these developments. 2007 The need for a ‘base line’ assessment of competency is identified by the ALPS team 2008 The CiPA research instrument is designed and tested 2008 -10 Longitudinal data is gathered from final year health and social work students just before graduation and at 6 and 12 month post qualifying. 2010 -2011 The CiPA self assessment tool is designed and launched 2011-2012 An evaluation survey is added to the CiPA website Figure One: CiPA Timeline Having decided to conduct the project with student involvement, their collaboration became particularly significant. Students as collaborators A strategic decision was made to have significant involvement of students as collaborators in this project. The authors are aware of a number of examples of successful student collaboration in research and development projects in addition to their own. For example the ‘Student as Partners’ initiative launched as part of the CETL programmes has been embedded within established schemes involving Birmingham City University, Northumbria University, and the University of Manchester. This is now sustained as the “Student Learning and Teaching Network” (2012). Similar schemes involving student collaborators were conducted in the ALPS programme. Millard and Hargreaves (in press 2013) have argued theoretically that student collaboration is a positive and a sustainable model for funded academic developments. However little seems to have been published that specifically evaluates students engaging in project work as collaborators with academics, an area we hope to investigate in the future. Case Studies Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 7, 2013 3 A literature review using Ovid Medline and the key words of “student” AND “engagement” OR “involvement” AND research” revealed only 3 published articles, one of which was an editorial. (Harpin 2012, Vessey 2008, Kramer 2005) even with these articles the emphasis is on educating the students to take part in research, rather than actual examples of their collaboration activity with academics. Collaboration as a key word revealed further papers on interprofessional education, and some on students as activists or change agents, rather than the type of collaboration this case study is concerned with. We therefore thought that looking more widely at student collaboration within an educational setting might offer some insights. There is, for example, a wealth of research exploring the use of ‘group work’ in its many guises. Two papers explicitly refer to this as a ‘collaborative’ activity. Conway-Gòmez & Palacios (2011), evaluated students from the USA and Chile collaborating on a sustainable development project via non synchronous discussion online, and Osman et al (2011) conducted in depth interviews with 10 students to explore their experiences of collaborative or group-based learning. Both, in different ways, focused on the collaborative aspect of our project. In both cases students reported pleasure in learning from and about others, and the Osman et al (2011) study supported the notion that collaborative learning is rated highly as an engaging and enjoyable experience. However in both studies, structure, facilitation and careful managing of the relationships within the collaboration were important and students did not necessarily report the positive outcomes that the researchers would have preferred. In particular more seemed to be gained from their interaction with others, than from any pedagogic advancement of their knowledge or critical understanding of the topics involved. An evidence based report (HEA 2012) from the Higher Education Academy reinforces the power of creating a culture of involvement throughout the whole student journey. This is reflected in their personal success, engagement and enhanced university experience Problem-based learning could be regarded as a similar approach where students learn through facilitated problem – solving. In this process the tutor facilitates the work of the students rather than delivering a lecture. Evidence suggests that this enables students to develop flexible knowledge, effective problem-solving skills, self-directed learning skills, effective collaboration skills, and intrinsic motivation (Hmelo-Silver 2004). We agree with this work and used this approach. The students responded with their ideas and enthusiasm for the work required to complete this tool. Method Despite limited evidence to support collaboration in project development and research, the ALPS experience led to this collaborative, interdisciplinary approach to the management of the follow on project. The project set out to convert the CiPA research instrument into a self assessment tool. We theorised that, as the questions had been validated they might form a successful basis for a self assessment. The students could then complete the questions via a web based piece of software, and receive feedback on their scores. Rather than a researcher analysing the data Case Studies Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 7, 2013 4 gathered, each student could use their own result to reflect on and develop their own practice. The project group consisted of academic staff from several universities, computing students on a full time sandwich year bursary working in the learning Technology Support Unit of a School of Human and Health Sciences, and student collaborators who responded to an advertisement though the university jobshop for part time work. The inclusion criterion for the work was that they had some experience of being on a placement (work experience). 14 students responded to the advertisement. A series of workshops were undertaken where the project was explained to the student collaborators who completed the CiPA questions themselves. Working in groups they discussed and agreed feedback they considered useful, and gathered resources that they felt would aid development. By this process our student collaborators were able to strongly influence feedback so that it was relevant, helpful and student focused. The computing students worked with their supervisor to develop software to the specification requested. The academic team moderated the feedback to create a consistent format, and all the students offered iterative feedback on the stages of the tool as it was developed. The subsequent self assessment tool was launched at a number of seminars and conferences regionally and nationally, and remains openly available on the internet [see http://cipa.hud.ac.uk ] The aim of this project was to create a tool where the software was as simple and practical as possible. Having been given this brief, CiPA was developed as an online application that allows the person to assess how competent or ready they are for practice/placement/work. Through taking the test, the user answers a series of questions selecting either ‘not applicable’ or 0 to 4 on a likert scale. Once completed the assessment runs some calculations to generate a feedback report to help build on competence for practice/placement/work. It is this feedback that was developed through the student collaboration. CiPA is a freely available tool that anyone can use which has been received very positively by people at dissemination events. Indeed it has been adopted as a springboard for newly qualified staff joining a number of NHS Trusts in the region to commence their ‘preceptorship’ period of initial post qualifying employment, linked to the ‘Flying Start’ [http://www.flyingstart.scot.nhs.uk ] programme. The disadvantage of the approach was that the development team were left with a belief that the tool was well received, but gave no means of knowing who the tool was being used by, or if they were finding it helpful. Evaluating CiPA In order to address this deficit ethical approval was gained to add a survey to the website that users were invited to complete after they had used the tool. A simple survey using Bristol On Line (BOS) software was designed and developed by subsequent computing students and the website was updated to include the survey. The survey is intended to identify:  Which professions are using the tool http://cipa.hud.ac.uk/ http://www.flyingstart.scot.nhs.uk/ Case Studies Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 7, 2013 5  Whether they were using it individually or as part of a supervision relationship, and  Ease of use and usefulness for their practice. Figure two lists the professions and roles of those who responded to the tool. Whilst we were disappointed to only receive 11 responses within the time period, we were pleased that it represented a range of students and qualified professionals from 6 different disciplines. One respondent was mentoring a newly qualified person. All the results were generally favourable; all but one responded agreed or strongly agreed against all of the questions, this would suggest that the tool does have interprofessional application, and can work as a starting point for discussion and supervision, as well as individual reflection and development. Total number of respondents 11 Discipline areas 6 Students 4 Newly qualified professionals 2 Supervisor/mentor 1 Experienced professional 4 Figure 2: Evaluation Results – Professions and Roles More detailed feedback identified a number of points, including relevance to practice, different styles of communication and reflection. The evaluation also highlighted some of the more challenging aspects of developing the professional role, such as dealing with cultural issues, which students often state that they feel ill prepared for once they are no longer under supervision. Asked if there were any ‘surprises’ most respondents said no. One, however, picked up on the emphasis on service users as a separate and important group. Service user engagement had been an important feature of the ALPS programme, so it was good to see this reflected here: ‘It gave opportunity to separate communication between service users and colleagues which was useful, they are often thought of or reflected on together.’ Respondents were asked if it offered anything new or different that they could use. Most said yes regarding the structure it could give: ‘Suggested some useful strategies in the feedback. This is different to clinical reflections done in the past. Good to have some guidance on where you can go or activities to help develop’. This particular result indicates that the student involvement in developing the feedback and signposting to support and guidance may have positively influenced the quality of the tool. Whilst these are only limited initial results, the team feel that they offer justification for the strategy of student collaboration. Case Studies Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 7, 2013 6 Conclusion Working collaboratively did have a number of advantages for the project team: student insights and ideas for the generation of feedback offered a perspective that would not have been reached by academics working in isolation. The involvement of student technologists allowed for a flexibility of approach and the input of fresh ideas. In addition there were clear advantages for the students, all of whom enjoyed the involvement and gained personal confidence and employment skills, mirroring the aspirations we had for users of the CiPA tool. The learning and skills displayed by the students were similar to those identified by Osman et al (2011). More structured evaluation of the student experience is being built in to subsequent projects. Reflecting on this experience, it was the first time that the academics involved had engaged in a collaboration of this nature. There were a number of lessons learned with regard to this: Firstly, the interaction with the students during the development of the responses to the tool was significant. We wanted to learn from them, and for this to be reflected in a tool that had meaning and purpose to them, rather than us. We recommend this methodology to others: giving students time and a free rein to think creatively led to a richer outcome. The tool relates strongly to student and newly qualified professionals’ needs and concerns. Secondly, facilitation was important – from our perspective we learned that more short sessions over time would have been better than one big session at the beginning, as it would have given students more time to understand the project and get to know us and the other students Whilst there are gains in student collaboration, there is also possible dilution of purpose. Having given the students the framework, we had to go with their direction and could not predict or fully control the outcome. The project team therefore need the confidence, and flexibility of design to allow for this type of variation. Finally, we have created what we believe is a robust and very useful tool. However, wide ranging implementation has proved challenging, which we recognise by the small number of responses to the evaluation so far. We have strong anecdotal feedback that the tool is fit for purpose and value, which motivated us to create the evaluation. A limitation of this study is that we still do not know the full extent of its use. References Baxter, P. and Norman, G. (2011). Self-assessment or self deception? A lack of association between nursing students’ self-assessment and performance. Journal of Advanced Nursing. 67 (11), 2406-2413. Conway-Gòmez, K. and Palacios, F.A. (2011). Discussing the Geography of Sustainable Development through an International Online Collaboration with Students in Chile and the USA. Journal of Geography in Higher Education, 35(2), 265-279. Flying Start Programme, NHS Scotland. Availble at: http://www.flyingstart.scot.nhs.uk. (Accessed 13 August 2012). http://www.flyingstart.scot.nhs.uk/ Case Studies Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 7, 2013 7 Harpin, S. and Gilmore-Bykovoskyi, A.L. (2012). Bringing students to the table: the value of student involvement in Midwest nursing research society. Western Journal of Nursing Research. 34 (3), 287-8. Higher Education Academy (2012). What works? Student retention and success programme, Available at: http://www.heacademy.ac.uk/what-works-retention. (Accessed 7 January 2013). Hmelo-Silver, C.E. (2004). Problem-based learning; What and How do students learn? Educational Psychology Review, 16(2), 235-266. Holt, J.; Coates, C.; Cotterill, D., Eastburn, S.; Laxton, J.; Mistry, H.; and Young, C. (2010) Identifying common competences in health and social care: An example of multi-institutional and inter-professional working. Nurse Education Today. 30(3), 264-270. Kramer, K. (2005). Medical student involvement in research in the pre-clinical years. Hawaii Medical Journal. 64 (7), 190-1. Kruger, J. and Mueller, R.A. (2002). Unskilled, Unaware, or Both? The Better-Than-Average Heuristic and Statistical Regression Predict Errors in estimates of Own Performance. Journal of Personality and Social Psychology. 82 (2), 180–188. Millard, L. and Hargreaves, J. (2013) Creatively employing funding to support innovation Innovations in Education and Teaching International (in press). Osman, G; Duffy, T.M.; Chang, J.Y.; and Lee, J. (2011). Learning through collaboration: student perspectives. Asia Pacific Education Review, 12(4), 547–558. Student Learning and Teaching network (no date). Available at: http://www.northumbria.ac.uk/sd/academic/sches/programmes/lt/afl/cetl_afl/engagement/net work/studentnetwork (Accessed 13 August 2012). Vessey, J.R. and de Marco, R.F. (2008). The undergraduate research fellows program: a unique model to promote engagement in research. Journal of professional nursing. 24(6), 358-63. Author Biographies Julie Laxton MA, BSc, RD is Teaching Fellow, Inter Professional Education. She teaches non-clinical subjects to medical students and other professions. Her main research and teaching interests are interprofessional education. Contact j.c.laxton@leeds.ac.uk Paul Dagg and advises staff with the development of technology-based learning resources. He managed the technical assistants who worked on the CIPA tool as well as contributing to its development. Paul is interested in useability, interface design and social media, and is currently completing a Master’s in Technology Enhanced Learning.Contact p.w.dagg@hud.ac.uk Dr. Janet Hargreaves MA, BA, RGN is Associate Dean Learning & Teaching in the School of Human and Health Sciences, University of Huddersfield. She teaches ethics, research and reflection. Her research interests are professional education and nursing history. Contact j.hargreaves@hud.ac.uk Duane Laverick BSc ICT (Hons) Year 4, placement year student Learning Technology Assistant He develops web applications for the School of Human and Health Sciences and assists in supporting students and academics with our VLE. http://www.heacademy.ac.uk/what-works-retention http://www.northumbria.ac.uk/sd/academic/sches/programmes/lt/afl/cetl_afl/engagement/network/studentnetwork http://www.northumbria.ac.uk/sd/academic/sches/programmes/lt/afl/cetl_afl/engagement/network/studentnetwork mailto:j.c.laxton@leeds.ac.uk mailto:p.w.dagg@hud.ac.uk mailto:j.hargreaves@hud.ac.uk Case Studies Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 7, 2013 8 Carrie Mitchell BSc ICT (Hons) Year 4, Learning Technology Assistant and placement year student. She develops databases and other IT applications for the School of Human and Health Sciences and assists in supporting students and academics with our VLE