Literature Review Journal of Learning Development in Higher Education ISSN: 1759-667X Special Edition: Researching PDP Practice, November 2010 Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry Elizabeth Symonds University of Worcester, UK Abstract The benefits of using e-Portfolio tools to enhance the student learning experience have been widely reported. These benefits include opportunities for reflection, flexibility, connectivity and integrated learning. In recognition of these potential benefits to students, the University of Worcester adopted an institution-wide e-Portfolio system which is available to all staff and students at the university. In order to maximise the potential of this system, a study was carried out to elicit staff and student views as to what the benefits of using an e-Portfolio were. It is all too easy to focus on the negative aspects when struggling to implement a new initiative, particularly involving technology. With this in mind a conscious effort was made in the evaluation to value and learn from the positive experience of users. Feedback was sought from early adopters of the system in an Appreciative Inquiry framework in order to ascertain the benefits of using such a system, to both staff and students, and to inform future practice and recommendations. Findings of the inquiry indicated that there are three significant factors which can influence effective practice with e-Portfolios: tutor engagement, support and embeddedness. Whilst both staff and student users found the use of e-Portfolio beneficial for the teaching and learning process, the issues that were raised by each group were subtly different. This paper will discuss how staff and students perceive the benefits of using an e-Portfolio and the factors that can influence engagement. It will also consider the implications of using an Appreciative Inquiry approach in institutional research. Key words: e-Portfolio; evaluation; learning technology; staff and student views. Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry Background and rationale E-Portfolios have become increasingly popular in higher education over the last decade (JISC, 2008) as they offer a means of supporting students in a lifelong and life-wide capacity for recording formal, non-formal and informal learning (Chen, 2009). They also provide a flexible, personalised, student-centred electronic environment where learners can record, reflect on, and present their own learning stories to a variety of audiences. Definitions of e-Portfolios can be problematic as there are a number of ways of approaching the issue. According to Stefani et al. (2007), the term ‘e-Portfolio’ can be used to reference the tool, a particular presentation of material, or all of the content of a portfolio. Cambridge (2009) goes further in suggesting the concept of ‘portfolio learning’ which is concerned more about the process involved in constructing a portfolio than the tool or the output. At the University of Worcester the PebblePad e-Portfolio had already been chosen as the preferred tool. In addition, the groups of staff and students participating in the study had pre-defined outputs that were required to be produced to fulfil the learning outcomes of the modules. This study therefore addresses this process approach to e-Portfolio development by looking at the ways in which staff and students engaged with the tool, how this experience affected their learning, and what the factors are that influence successful e- Portfolio learning. The University of Worcester adopted the PebblePad e-Portfolio system in October 2008 and it was implemented and rolled out to staff and students in a phased project which aimed to provide a thorough introduction and grounding in the use of e-Portfolios through structured training workshops and supported pilot projects. In phase one staff were invited to take part in the initial implementation by using the e-Portfolio for their own continuing professional development as individuals. The aim of this phase was to allow staff to become familiar with the software in order to better equip them to using it with their students at a later stage. It was also intended to build a community of users who could support each other and share ideas for best practice. Phase two invited those members of staff who had taken part in phase one to then use the e-Portfolio with their students in a structured and supported way. Support was offered by Journal of Learning Development in Higher Education, Special Edition: November 2010 2 Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry the Learning and Teaching Technology Unit in terms of advice and guidance on using PebblePad to address some or all of the learning outcomes of the individual courses of study. In addition, practical workshops were provided for students to introduce them to the e-Portfolio tool and guide them through the process of producing the required output to meet the learning outcomes, and these were followed up by ongoing support and drop-in sessions throughout the period of study. In phase three of the roll-out, the e-Portfolio tool was made available to all staff and students through web-links and support materials, and generic workshops were offered throughout the year. This study focuses on the staff and student users who took part in phase two of the project. Methods It was recognised that there was a need to evaluate the use of the e-Portfolio at an early stage in the pilot in order to inform future developments. As the university had committed to using PebblePad for an initial three year period, there was a desire to avoid the traditional deficit model of evaluation often associated with academic research (Cousin, 2009) and new technologies in particular. Quantitative measures can be acquired from an institutional perspective, for example, number of active users or tools usage, and are often useful as a snapshot to the extent of usage of a particular tool. Lawton and Purnell (2010), however, point out that this data is relatively meaningless in the context of PDP and e-Portfolio development. In order to attain a richer, deeper understanding of e-Portfolio use, the types of quantitative measures we could use rely on the researchers recording data such as time spent and number of reflections/experiences recorded. The very nature of e-Portfolios as a personal learning system means that the information recorded in it is private unless the user chooses to share it with someone else. To request access to an individual’s personal e-Portfolio will inevitably affect the way in which the user engages with it knowing that they are being observed (the Hawthorne effect – Roethlisberger and Dickson, 1939) and therefore not yield a true reflection of engagement. Journal of Learning Development in Higher Education, Special Edition: November 2010 3 Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry Appreciative Inquiry is a solution oriented research method which seeks to uncover what brings life to a project or initiative with a view to changing it or making evaluative judgements. Historically it has been associated with organisational change, however it is being increasingly adopted in higher education research projects (Cousin, 2009). In its pure form, Appreciative Inquiry involves four stages – the four Ds – as illustrated below. Figure 1. Adapted from http://www.appreciative-inquiry.org/. Appreciative Inquiry can be regarded more as a spirit of inquiry rather than a pure methodology and as such can be adapted to suit the requirements of the individual investigation (Cousin, 2009). In this context, it was decided to use Appreciative Inquiry as the overall approach but focusing on the discovery phase only, to elicit the views of staff and students and use that as a departure point for future research. Interviews were conducted with four members of academic staff who had used the system both personally, and with groups of students for learning and teaching. A focus group with fifteen students who had used the system for assessment was also held to gather feedback from the student perspective. A thematic evaluation of the transcripts was undertaken to elicit common threads. In addition, naturally occurring data such as system statistics and anecdotal evidence were used to triangulate the information. Journal of Learning Development in Higher Education, Special Edition: November 2010 4 http://www.appreciative-inquiry.org/ Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry As a practitioner/researcher I was very aware that my role may well influence the outcomes of the research having been involved in both the implementation and the evaluation of the e-Portfolio. In order to minimise any bias in the interpretation of the data, I returned to the original staff participants in order to present my initial interpretation to ascertain some common understanding of the discussion. In doing this, I was co-creating meaning from the interviews with the interviewees and as Cousin (2009, p.171) points out, Appreciative Inquiry is not necessarily a truth yielding process but more a generative one. I was also mindful that the sample of people I spoke to were by necessity pilot users who were championing the new system and therefore likely to have a more positive outlook than someone who had chosen not to use it. It may therefore be useful in future investigations to speak to non-users about their reasons for not engaging with the tool. Findings and discussion The thematic analysis identified three major factors which influenced the success of e- Portfolio learning: tutor engagement; support; and embeddedness. These factors have some similarity to the findings of Tosh et al. (2005) who looked at student perspectives on e-Portfolio usage at four different universities and found that buy-in, motivation, assessment and technology were key factors from the student perspective. Tutor engagement JISC (2009, online) states that staff engagement is a key factor in e-Portfolio implementation: Staff engagement is key – role, support and attitude of learner support staff is critical. Staff who use an e-Portfolio for their PDP have a higher learner engagement. Each of the staff interviewed had experience of using the e-Portfolio for their own professional development and understood how the tool worked and the importance of making its use relevant to the learning outcomes. As such, they believed that the use of e- Journal of Learning Development in Higher Education, Special Edition: November 2010 5 Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry Portfolio had the potential to enhance the student learning experience and were motivated to encourage full student participation: I think the single most important thing is to use PebblePad as I would expect the course participants to use it, so in the first instance I have been developing my own personal portfolio. What that has provided is the opportunity to use and utilise all the facilities within PebblePad first hand so I have had the opportunity to experiment with a range of facilities. Staff who had used the e-Portfolio themselves were familiar with the possibilities and limitations of the system and were in a position to offer support to their students in addition to the support provided by the central service. Three of the tutors interviewed had also used their own e-Portfolio to develop module resources which they then shared with the students. This had the dual benefit of both delivering information to the students but also providing a working exemplar of the kind of output they were intended to produce: A facility where all the sessional information is held, particular dates when the module is delivered and a course blogging facility. One tutor went a step further in creating a template for their students as a basis for their assessed work: Well I gave them a template and the very first part of the template was just a very open reflection on how they used ICT, both in their personal lives and what they have seen in school. In addition to using their own portfolios for producing resources and exemplars, all the staff interviewed also encouraged students to submit their assessments through the e-Portfolio system and marked online. This was considered to be a huge benefit from the staff perspective: Journal of Learning Development in Higher Education, Special Edition: November 2010 6 Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry From my point of view I could look at what they have done anywhere and I don’t have to carry around 125 A4 ring binders. I felt it was very flexible and it was very quick and easy to mark the assignment because I just went on the computer, logged in, scrolled through and marked each one, made a few comments and I think that’s a far better way for the student because they can go back to them. Students were however a little more reticent about the online submission and felt quite insecure about sharing their work. Many felt that they needed more confirmation when something had been submitted: The only thing that I found was that when you got to the actual gateway and you have to submit, it just went and you are sat there going did it go? Did it not go? …so a bit of confirmation just so you know that it’s there would be good. I think you have to be very careful when you are sending stuff as well because I noticed on the very first summative we did I had to ring Dot because she had sent the whole summative to the whole gateway without realising. I just feel a bit paranoid. These comments highlight the fact that whilst tutor engagement is a key factor in securing student involvement, attention should be paid to students’ concerns, particularly where summative assessment is involved. One of the students summed it up as follows: Because we have used it for actual assignments, to me things are more relevant and more important whereas if you’re just using it as a guide or whatever then a lot of these issues wouldn’t even matter. So with engagement comes a responsibility for ensuring that sufficient support and guidance is built into the programme so that student fears are addressed and alleviated. Journal of Learning Development in Higher Education, Special Edition: November 2010 7 Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry Support Support was a critical issue for both the staff and students in this study although, unsurprisingly, students were more concerned about the technicalities of the system and being able to use the tool effectively, whereas staff were more focused on the outcomes and processes involved in the students’ learning. This reflects the findings of Fitch et al. (2008) who found that students in their study were particularly concerned about various technical aspects of the e-Portfolio. In order to address potential student concerns about operation and functionality of the e- Portfolio tool, a structured support system was built into the modules that took part in the study. To this end the support team were heavily involved in the module delivery by running in-class workshops for all students at the outset of the module which focused particularly on the required learning outcomes. Practice outputs were developed during the workshops which then provided the scaffolding to allow the students to work more independently on their assessed work. Further drop-in sessions were offered and attention was drawn to the online help available within the system. One tutor also developed their own support materials: …and I have been making those Camtasia videos about how to do different steps which I think has been quite helpful. In general, the students were very happy with the level of support provided: If it hadn’t been for all your support I think I would have thrown my toys out of the pram. I think the support that was given was really good. However, the fact that the work was assessed and formed part of their overall grade was a concern for some: I think it was worse for us because we are in level three and it was used for assignments and not something there to just help you, whereas in level one they have got time to learn. If I was doing this in level one I wouldn’t have felt as stressed about it as I was doing it now. Journal of Learning Development in Higher Education, Special Edition: November 2010 8 Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry The staff involved were already confident users of the system so were less concerned about the technical issues, however, they were mindful of the fact that their students were concerned about the use of a new technology: Initially some have that hesitation when they first come across a new tool but a lot of them, once we had gone through it, they have come back and said it is much easier than their initial impression and it was actually quite simple to use, so I think generally they are quite positive about it. ...initial stage of reticence, the usual references to not being ‘techie’,having an under-familiarity with technology, feeling a bit disadvantaged…The interesting thing is as soon as they find that the PebblePad system is very round edged, that it has a consistency of reliability…as they work incrementally through each of these outcomes and looking at prospective evidence, I think what starts to open up is that they are reminded of their own personal assets, not the assets within PebblePad but their own personal assets and particularly their prior learning and experience and how that seeks to inform these negotiated learning outcomes, so I think it is quite a remarkable experience and I am talking from a course tutor perspective where the penny starts dropping, that the learning didn’t begin at the start of this module and won’t end at its completion. Despite the apprehension and concern in the early stages of use, as students began to feel more comfortable with the tool, the benefits of using the e-Portfolio became more apparent: I think it helped in a sense because when I did the webfolio it helped put a mental image in my head... like the different approaches to learning that we went over – it kind of structured it in my head. Yes it was nice to learn in different ways. One of the key benefits identified by the students was the ability to be creative and work in different ways which they found liberating and motivating: Journal of Learning Development in Higher Education, Special Edition: November 2010 9 Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry ...it just enabled you to set up something that looked really professional with all your information and I actually really liked it. I like the concept of doing something different rather than doing just written work that we do all the time. It made it more fun doing the web page, I was excited about putting pictures up really. Using PebblePad, the quality of work you do produce in the web portfolio, you have much more of a sense of accomplishment than you do when you do an essay. Overall, the structured approach to supporting students seemed to be successful and the commitment and engagement of the tutors involved served to reinforce the benefits of using the e-portfolio: …by the end of it they recognise just how much it is enhancing their learning experience although to begin with I think it was hidden by something that was new and different and there was an element of ‘oh how do we do this?’ But there are very positive comments that people have said that they felt they had done something really good at the end of it which I think they have, and they showed me that they really understood. Embeddedness Embedding the use of e-Portfolios involves making the use of the tool explicit and mandatory. This was achieved by using the tool not only for students to post their own work, but also by providing resources and a means for communication both peer to peer and student to tutor. The creation of course resources within the system was discussed earlier and serves to both provide students with information and act as an exemplar. Further, by using the tool in this way, students have to engage with the system in the first instance in order to access these resources, and this was usually carried out in the workshop provided at the beginning of each module: Journal of Learning Development in Higher Education, Special Edition: November 2010 10 Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry We built it into the module so when to do things is all planned out in the module outline and we give them some support in the sessions in making it, and the case study materials are there to do in school so it’s all very much embedded and linked together because I’m not sure it would have worked that well if it wasn’t that way! The utilisation of PebblePad was deeply embedded, obviously embedded in the documentation which was a very intentional strategy in that we specified that PebblePad would be utilised in terms of webfolio development and utilising and regulating its facilities to create an evidentially driven work-based webfolio, so it was clear and mandatory. In each case there was also the provision of a module or course gateway within the system. Gateways act as a community area where users can share work with each other and with their tutors. This facility was used in different ways in different modules, however, one of the key benefits of gateways is that they allow students to post their items, which can then be viewed by the whole group, and feedback provided to each other. The tutors found this facility particularly useful in building a learning community amongst the students and for peer assessment: I think the thing I found really useful was the gateway and them being able to look at each others’ work. The opportunity to communicate with the participants regularly. This community element allowed the students to work in a more open and collaborative way, particularly on the formative assessments: The first case study they peer assess, they just look at someone else’s in the group just to give them some early sort of formative feedback on what they are doing and how they are doing it, and hopefully as well to sort of cross fertilise ideas with each other. Using PebblePad it’s easy to get them that instant feedback so it can be fed into that next piece of work which hopefully did happen. Journal of Learning Development in Higher Education, Special Edition: November 2010 11 Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry These comments also indicate the level of tutor commitment to the process of e-Portfolio development within their own courses of study. Gateway activity in courses where the use of the e-Portfolio was not embedded and peripheral to the main learning outcomes for the module attracted a much lower level of student engagement, with many not participating at all. The students also found the gateway a useful tool and seemed to like the community aspect this provided: With the gateway I liked how it was our own little area for this module and it’s different from Blackboard as it has a better feeling about it, I don’t know how to describe it, I think it’s a warmer feel than Blackboard. I didn’t comment on other people’s assignments as much as I should have but I did like the fact that I could have if that makes sense, but I thought it was nice how it was much more interactive and on a more personal scale. It is interesting to note at this point that in phase three of the roll-out of PebblePad which allowed all students access to their own e-Portfolio account, spontaneous, self-directed engagement was almost non-existent. This was evidenced by the poor attendance at generic workshops offered to all students where only three students took part. Conclusion This study has highlighted some of the factors that influence effective e-Portfolio learning, including the importance of tutor engagement, the need for rigorous support mechanisms for both technical and pedagogical support, and the necessity of embedding the use of the e-Portfolio in the curriculum. It is recognised that this is a small scale study which was carried out within an Appreciative Inquiry framework so although the findings reflect much of what is in the literature around e-Portfolio practice, further work needs to be done to explore these findings more thoroughly across a wider sample of both staff and students. As a pilot study, however, these outcomes have proved invaluable in advising staff on how to utilise the e-Portfolio most effectively for learning and teaching, and a fully supported structure is now in place to provide users with the guidance they need. Journal of Learning Development in Higher Education, Special Edition: November 2010 12 Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry Acknowledgement This paper is an outcome of the National Action Research Network on researching and evaluating Personal Development Planning and e-Portfolio practice project (2007-2010). The project was led by the University of Bolton in association with the University of Worcester and Centre for Recording Achievement, and in national collaboration with the University of Bedfordshire, Bournemouth University and University of Bradford. The project was funded by the Higher Education Academy, National Teaching Fellowship Project strand. More details about the project can be found at: http://www.recordingachievement.org/research/narn-tree.html. References Cambridge, D. (2009) ‘Two faces of integrative learning online’, in Cambridge, D., Cambridge, B. and Yancey, K. (eds.) Electronic portfolios 2.0: emergent research on implementation and impact. Sterling: Stylus Publishing, pp. 41-49. Chen, H. (2009) ‘Using e-portfolios to support lifelong and life-wide learning’, in Cambridge, D., Cambridge, B. and Yancey, K. (eds.) Electronic portfolios 2.0: emergent research on implementation and impact. Sterling: Stylus Publishing, pp. 29-35. Cousin, G. (2009) Researching learning in higher education: an introduction to contemporary methods and approaches. Abingdon: Routledge. Fitch, D., Peet, M., Reed, B.G. and Tolman, R. (2008) ‘The use of e-Portfolios in evaluating the curriculum and student learning’, Journal of Social Work Education, 44(3), pp. 37-53. JISC (2008) Effective practice with e-Portfolios: supporting 21st century learning. JISC/HEFCE. Available at: http://www.jisc.ac.uk/publications/programmerelated/2008/effectivepracticeeportfolio s.aspx (Accessed: 1 June 2010). Journal of Learning Development in Higher Education, Special Edition: November 2010 13 http://www.recordingachievement.org/research/narn-tree.html http://www.jisc.ac.uk/publications/programmerelated/2008/effectivepracticeeportfolios.aspx http://www.jisc.ac.uk/publications/programmerelated/2008/effectivepracticeeportfolios.aspx Symonds Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry JISC InfoNet (2009) E-Portfolios: emerging lessons. Available at: http://www.jiscinfonet.ac.uk/infokits/e-portfolios/emerging-lessons (Accessed: 1 June 2010). Lawton, M. and Purnell, E. (2010) ‘What evidence for which audience?’, Researching and Evaluating Personal Development Planning and E-Portfolio: 2nd International Residential Seminar. Nottingham 26-28 April. Roethlisberger, F.J. and Dickson,W.J. (1939) Management and the worker. Cambridge, Mass: Harvard University Press. Stefani, L., Pegler, C. and Mason, R. (2007) The educational potential of e-Portfolios: supporting personal development and reflective learning. Abingdon: Routledge. Tosh, D., Light, T.P., Fleming, K. and Haywood, J. (2005) ‘Engagement with electronic portfolios: challenges from the student perspective’, Canadian Journal of Learning and Technology, 31(3) [Online]. Available at: http://homepages.ed.ac.uk/jhaywood/papers/Engagement%20with%20Electronic%2 0Portfolios.pdf (Accessed: 1 June 2010) Author details Elizabeth Symonds is the Team Leader for the Learning and Teaching Technology Unit at the University of Worcester and has participated in the National Action Research Network on researching and evaluation PDP and e-Portfolio practice. Journal of Learning Development in Higher Education, Special Edition: November 2010 14 http://www.jiscinfonet.ac.uk/infokits/e-portfolios/emerging-lessons http://homepages.ed.ac.uk/jhaywood/papers/Engagement%20with%20Electronic%20Portfolios.pdf http://homepages.ed.ac.uk/jhaywood/papers/Engagement%20with%20Electronic%20Portfolios.pdf Evaluating an e-Portfolio implementation with early adopters using Appreciative Inquiry Abstract Background and rationale Methods Figure 1. Adapted from http://www.appreciative-inquiry.org/. Findings and discussion Tutor engagement Support Embeddedness Conclusion Acknowledgement References Author details