Australasian Journal of Educational Technology, 2017, 33(1). 32 Using PeerWise to develop a contributing student pedagogy for postgraduate psychology Wendy McKenzie and John Roodenburg Monash University The importance of the role of peer and self-assessment in developing formative and sustainable assessment practice in higher education is increasingly becoming evident. PeerWise is an online software tool that engages students in contributing to their own and others’ learning by authoring, answering and providing feedback on multiple choice questions. Using a mixed methods approach, 20 students responded to discussion questions and an online survey about their perceptions of using PeerWise compared to online quizzes as part of the blended delivery of a postgraduate psychology unit. Students considered both authoring and answering questions in PeerWise to equally benefit their learning. Answering questions in PeerWise was perceived to be more helpful for learning than questions on a Moodle quiz. This advantage was evident across more complex cognitive skills, understanding, applying, evaluating and creating information, although only significant for facilitating understanding. PeerWise and online quizzes were seen to be equally helpful in facilitating recall. Despite the perceived benefits of PeerWise, students preferred quiz questions to be set by an expert if used as recognition of progress. Introducing PeerWise was effective in promoting engagement with peers; however, refinements to the model should focus on increasing student confidence in their own and peers’ capabilities. Psychology is in line with other health professions in adopting an evidence-based practice (EBP) framework for improving client outcomes (Spring, 2007). Evidence-based practitioners base decisions about treatment on the critical appraisal of evidence from research and clinical practice, integrated with consideration of patient characteristics and values (Babione, 2010; Collins, Leffingwell, & Belar, 2007). The most challenging requirement of EBP can be keeping abreast of the best available evidence given the growing proliferation of research and increasing frequency of paradigm shifts (Lilienfeld, Ritschel, Lynn, Cautin, & Latzman, 2013). Developing evidence-based practitioners who are able to meet these demands requires emphasis on capability over competence (Bauer, 2007; Fraser & Greenhalgh, 2001). A capable graduate is someone who habitually practises critical scholarship, adapts competencies in response to paradigm shifts to be effective in unfamiliar contexts, and seeks continuous improvement (Fraser & Greenhalgh, 2001). The challenge for educators of the next generation of professional psychologists is to design effective and efficient courses to develop capability in EBP as a sustained lifelong habit. Thomas, Saroyan, and Dauphinee (2011) proposed a set of principles to guide best practice for teaching EBP: • build on existing understandings of EBP • emphasise the negotiation with peers in clinical decision making • facilitate learning contexts that encourage problem-solving, higher-order cognitive skills • foster collaborative learning to encourage self-monitoring and highlight different understandings • scaffold learning for students to attain goals just beyond the level of their current ability. As such, there is clear alignment between developing capabilities in EBP and values of lifelong learning that encompass resolution of knowledge variations and adaptation for new contexts (Belar, 2011; Boud, 2000; Luebbe, Radcliffe, Callands, Green, & Thorn, 2007). Developing graduates who are well prepared to contribute to a learning society requires teaching practices based on social constructivist approaches to learning that focus on process rather than content (Luxton-Reilly & Denny, 2010; Rolloff, 2010; Thomas et al., 2011). This process engages the learner in the active construction of knowledge driven by resolving cognitive conflicts between existing and new understandings (Palincsar, 1998). Social interaction plays a critical role in mediating the resolution of multiple realities towards a shared understanding (Kanuka & Anderson, 1999; Vygotsky, 1978/1997). Social constructivism is supported by approaches to teaching and learning that encourage reflective and self-directed learners who tolerate uncertainty and are receptive to feedback (Collins et al., 2007; Fraser & Greenhalgh, 2001). Australasian Journal of Educational Technology, 2017, 33(1). 33 Formative assessment involves the learner in monitoring and acting on feedback in order to move closer to achieving their learning goal (Boud, 2000; Nicol & Macfarlane-Dick, 2007). It is widely acknowledged that formative assessment practice is a critical part of developing lifelong learners (Boud & Falchikov, 2006; Yorke, 2005), and an integral part of effective teaching practice (Black & William, 1998; Taras, 2005). Opportunities for formal and informal formative assessment can be diminished in circumstances where resources for teaching and learning are stretched (Yorke, 2005). Although summative assessment tasks can serve the dual purpose of providing formative feedback and certification of achievement, there is concern that reliance on summative tasks is sending a message to students that grades are more important than feedback (Boud, 2000; Taras, 2005). Actively involving students in both the assessment and feedback practices reduces students’ learned dependence on grades (Yorke, 2005), separates out the business of feedback and assessment (Taras, 2002), and creates more sustainable practices (Boud, 2000; Boud & Molloy, 2013). Peer assessment has become an increasingly important component of formative and sustainable assessment practices in higher education (Boud, Cohen, & Sampson, 1999; Falchikov, 2007). The advantages afforded by having peers involved in this process extend beyond the value of helping others to learn, to recognising the contributions of peers to our own learning (Falchikov, 2007; Hamer et al. 2008; Sondergaard & Mulder, 2012). Over and above the benefits of encouraging learner autonomy and self-regulation, being part of a learning community allows students to calibrate their progress against peers; test ideas against a variety of perspectives; be exposed to alternative strategies; develop skills in evaluation; and practise giving and receiving feedback (Nicol & Macfarlane-Dick, 2007; Sondergaard & Mulder, 2012). Placing peers at the centre of this experience encourages students to seek verification and clarify knowledge where they otherwise may not if the feedback is linked to the authority of the expert teacher (Draper, 2009; Fraser & Greenhalgh, 2001). The success of this approach “relies on the individual student’s recognition of the class as a group of fellow travellers on a shared journey” (Sondergaard & Mulder, 2012, p. 34). Contributing student pedagogy (CSP) is one such approach that translates the principles of peer-based formative assessment to create online communities of students who are actively engaged in sharing resources, and assessing their own learning and the quality of others’ contributions (Luxton-Reilly, Denny, Plimmer, & Sheehan, 2012). PeerWise is an online program (http://peerwise.cs.auckland.ac.nz/) founded on a particular form of student contributing pedagogy – constructive evaluation – based on student- generated assessment (Luxton-Reilly & Denny, 2010). Typically, PeerWise is set up as a repository for students to create and answer multiple-choice questions (MCQs) throughout the course of a unit of study, either as a compulsory form of assessment with a participation grade attached, or on a voluntary basis. Students are engaged in authoring questions, providing feedback and rationale, answering questions, as well as rating and commenting on others’ questions. Although using MCQs as an assessment tool is often criticised for encouraging surface learning (Ramsden, 2003), engaging students in authoring and evaluating questions presents a higher degree of cognitive challenge and a richer learning experience (Hardy et al., 2014; McQueen, Shields, Finnegan, Higham, & Simmen, 2014). A summary of the levels of student engagement in PeerWise and analysis of the associated benefits is described in Table 1. The range of learning activities required by PeerWise combines elements of self- and peer assessment, both of which make important contributions to learning (Falchikov, 2007). http://peerwise.cs.auckland.ac.nz/ Australasian Journal of Educational Technology, 2017, 33(1). 34 Table 1 A description of activities in PeerWise in relation to the learning process (based on Luxton-Reilly & Denny, 2010; Luxton-Reilly et al., 2012) PeerWise activity Learning process Author a question and write an explanation Individual learning related to knowledge of content; generate response alternatives, predict misconceptions; provide rationale to justify answer, explain. Answer questions Review existing knowledge of content; self-assessment; formative feedback identifies ambiguities and misconceptions, clarify understanding; refine learning goals. Provide open-ended feedback and comments Correct errors; seek clarification; suggest improvements; comment on relevance or interest to learner; provide further information; view and resolve multiple perspectives. Rate quality and difficulty level Formative feedback for question authors; develop and apply standards for evaluating questions. Receive feedback and ratings on own questions Self-evaluation; compare with peers; self-monitoring. There is a steadily growing community of teachers and researchers who have implemented and evaluated the use of PeerWise across a range of academic disciplines and cohorts (Hardy et al., 2014; McQueen et al., 2014). Overall, the evaluations of PeerWise have been positive, showing high levels of student engagement in producing questions that are of good quality and error free, and associated improvements in learning (e.g., Denny, Luxton-Reilly, & Hamer, 2008; Purchase, Hamer, Denny, & Luxton-Reilly, 2010). Rhodes (2013) explored how the different learning activities in PeerWise contributed to learning in a small cohort of undergraduate nursing students. In particular, writing a rationale to justify the correct answer to an authored question was associated with knowledge building, whereas answering other students’ questions was associated with knowledge consolidation. Hardy et al. (2014) conducted one of the larger scale studies to investigate the benefits of using PeerWise for students of different academic ability. Based on a cohort of 850 students across three science disciplines, Hardy et al. report a significant positive correlation between student use of PeerWise and examination performance. The benefits of using PeerWise appeared to be larger for students of lower ability, which was attributed to the availability of peer feedback and comments on questions. McQueen et al. (2014) found greater benefits for both high ability and lower than average ability students. The results also demonstrate these gains transferred to other types of assessment tasks, such as essays and problem-solving. The authors suggested PeerWise improves learning by encouraging reflection in creating questions and engaging in discussion with peers. These findings support the use of PeerWise to promote self-regulated learning, incorporating many of the characteristics of formative and sustainable assessment practice: students are involved in constructing the test criteria and standards; self-assessment encourages reflection; an explanation or a rationale provide adequate feedback; peer comment and feedback encourage dialogue; there is opportunity for repeated attempts over time to improve motivation and move closer to the learning goals (Nicol, 2007). Given the degree of overlap with principles associated with best practice for teaching EBP (Thomas et al., 2011), PeerWise has considerable appeal in designing teaching and learning environments for developing capability and life-long learning. Teaching context The purpose of this paper is to evaluate the impact of introducing PeerWise as part of a CSP for training professional psychologists. The research was undertaken in the context of a coursework master’s postgraduate psychology degree. The delivery of the course incorporates elements of a blended learning design that “combine the effectiveness and socialization opportunities of the classroom with the technologically enhanced active learning possibilities of the online environment” (Dziuban, Hartman, & Moskal, 2004, p. 3). As in most blended learning courses, the introduction of online activities provided the impetus for changing time spent in class from passive to active learning, promoting increased and higher quality interactions (Vaughan, 2007). Weekly online quizzes were introduced in one unit to test students’ understanding of prescribed reading in preparation for class discussion. Quizzes were conducted using the Australasian Journal of Educational Technology, 2017, 33(1). 35 university learning management system, Moodle, and scores contributed to the overall unit grade. The purpose of the quizzes was to ensure students came to class informed, thereby transforming class time from content-heavy didactic lectures to interaction-rich use of debate, discussion, modelling and practice of clinical skills. Using online quizzes in a similar way has been successful in improving student engagement and learning outcomes across a range of disciplines and blended learning contexts (Angus & Watson, 2009; Kibble, 2011; McKenzie et al., 2013; Peat & Franklin, 2002; Stull, Varnum, Ducette, Schiller, & Bernacki, 2011). On the second iteration of the blended learning model, PeerWise was introduced as an alternative to the online Moodle quizzes in a short pilot of a few weeks towards the end of one first semester unit. The use of PeerWise was initiated in response to student feedback about the high level of perceived stress and workload associated with the summative (Moodle) quizzes. An informal evaluation of students’ initial experience of PeerWise was conducted during class discussions. Early feedback suggests that students found using PeerWise more enjoyable, less stressful, and more self-directed, and that it encouraged deeper reading than when they had to complete the summative quiz. However, it is clear that the graded quiz was valued because it represented reward for effort Students also expressed a need for clear structure around expectations for authoring and answering questions in PeerWise. In response to this feedback, a revised assessment model (see Table 2) was adopted for a second semester unit, bringing together peer-based formative, summative, and collaborative learning activities. Table 2 Cycle of engagement in online assessment tasks for a postgraduate psychology unit, repeated four times during the semester, as a function of type of assessment and percentage contribution to unit grade Timing Assessment requirements Contribution to unit grade Week 1 Answer all contributed questions in PeerWise on that week’s topic prior to lecture. Formative (10%) Answer all contributed questions in PeerWise on that week’s topic prior to lecture. Provide comments and ratings on peer contributions. Week 2 Author 2 questions in PeerWise based on prescribed readings. Answer all contributed questions in PeerWise on that week’s topic prior to lecture. After Week 2 lecture One group of students allocated task of creating questions for the summative online quiz, selecting 10 questions from PeerWise for the previous 2 weeks plus an extra 5 (new) questions. Collaborative (5%) End of Week 2 Complete online quiz in Moodle (25 questions in 30 minutes). Summative (15%) Research questions A mixed method design was used to investigate professional psychology students’ experiences and perceptions of the way online peer-based formative (PeerWise) and summative (Moodle) assessment activities contributed to their learning. To assess the contribution of separate components of the task on learning, an online survey was designed to probe experiences at different levels of cognitive skill based on the revised Bloom’s taxonomy (e.g., remembering, understanding, applying, analysing, evaluating, creating; Krathwohl, 2002). Further questions targeted metacognitive awareness of values and attitudes associated with lifelong learning and capability for EBP. Open-ended questions were included to explore perceptions of the relative benefits of the peer-based formative and summative elements of online quizzes. The research questions focused on exploring the level of engagement in authoring, answering, and commenting on questions using PeerWise; differences in the level of cognitive challenge associated with authoring, answering, and commenting on questions in PeerWise; and differences in perceptions of contributions to learning when completing peer-based formative and summative online quizzes. Australasian Journal of Educational Technology, 2017, 33(1). 36 Method Participants Students enrolled in the first year of a professional psychology master’s course were invited to participate. The class consisted of 27 students (including the first author), 24 females and 3 males, with an estimated age range between 23 and 48 years. Entry to the course required an approved 4-year undergraduate sequence in psychology, minimum grade point average = 3.0 (out of 5) or honours IIA equivalence. All students had at least 2 years paid or voluntary work experience in psychology. Three students had completed other master’s or postgraduate degrees in education. The teaching and learning environment in these units has continuously encouraged students to provide feedback on their learning experiences and actively participate in decisions about assessment. The opportunity to participate in this project was an extension of this process. Necessarily, both the student researcher and supervisor were involved in the delivery of the units in which the participants were enrolled. The student researcher was a fellow student, contributing to PeerWise activities, and overseeing the groups creating the Moodle quizzes. The research supervisor was the unit coordinator. Because of these special relationships, care was taken to address potential ethical conflicts. All responses to the online survey were anonymous. The survey asked for permission to use the anonymously submitted group notes from the class discussion, summary statistics and questions submitted to PeerWise, and de-identified demographic information. This approach was the only way of ensuring that the unit coordinator did not know who may have withdrawn their permission. Because the survey responses were anonymous we undertook not to use any category of data if at least one student did not consent. Students were informed that failure to respond to the survey would be interpreted as permission to use all other forms of data already collected. Ethics approval was obtained from the Monash University Human Research Ethics Committee. All participants gave permission for all forms of data to be used. Design In accordance with Frels and Onwuegbuzie (2013), a mixed method approach was used for (a) development – qualitative feedback obtained from students in class discussions informed the creation of items for the online survey; (b) triangulation – quantitative analysis of Likert rating responses to survey questions to verify the convergence of feedback from individuals and group-based class discussion; and (c) expansion – open-ended questions on the survey are designed to elicit further information about benefits and limitations of the assessment activities. Materials and procedure All participants used PeerWise and Moodle to complete the unit requirements as outlined in Table 2. There were four blocks of quizzes; each block consisted of two PeerWise topics and one Moodle quiz. Students were required to author 2 questions per PeerWise topic and answer all questions per topic to achieve 10% of the unit grade. Each student was allocated to a group to create one Moodle quiz (5%), and received a score for the number of correct answers on the other three Moodle quizzes (5% each; total 15%). In-class group feedback At the end of semester, students were asked to discuss their experiences of using PeerWise and Moodle quizzes in groups during the penultimate class. This session was conducted by the unit coordinator, and the student researcher was not present. The students were allowed approximately 30 minutes to discuss in small groups their evaluation of the unit in response to a set of general prompts (e.g., consider contribution to learning of participation in the debates; practicum; classes; PeerWise & Moodle; and completion of the assignments). A total of eight groups submitted a summary of their responses in note form to the unit coordinator. These notes were analysed by the first author to identify the issues that were prominent in students’ discussion of using PeerWise and Moodle, including the following: • Quizzes provide motivation to complete readings and revision to consolidate learning. • PeerWise questions were repetitious and did not have sufficient coverage of reading materials. • PeerWise had too many questions to answer per topic. Australasian Journal of Educational Technology, 2017, 33(1). 37 • Quiz questions encourage surface learning and rote memorization. • PeerWise had too much variation in the use of comments and ratings of questions. • Quiz questions for the summative assessment task (Moodle) were preferred because they were set by the teacher (or expert) rather than by peers. Online survey development Based on the feedback obtained from the in-class group discussions, and previous research, a survey was developed to assess the student experience of using PeerWise and Moodle quizzes. The survey was divided into six sections, summarised in Table 3. Section 1 satisfied the ethical requirements to obtain permission to use the data. Section 2 addressed specific concerns raised about the workload associated with the task requirements. Section 3 was an attempt to systematically explore student perceptions about how different learning activities in PeerWise and Moodle quizzes contribute to their learning (e.g., authoring questions; answering questions; comments and feedback). Items were written to align with the six levels of cognitive complexity associated with Bloom’s modified taxonomy: remembering, understanding, applying, analysing, evaluating, creating (Krathwohl, 2002). Section 4 addressed metacognitive aspects and students’ perceptions of themselves as a learner and developing practitioner. Open-ended comments were designed to elicit further information about the benefits and limitations of the core components (PeerWise versus Moodle), and a specific recommendation for future cohorts. An online survey was created using Qualtrics. Participants were contacted by email a few weeks after classes had ended to invite them to complete the online survey. Two reminder emails were sent and the survey remained open for several weeks as many students were absent on holidays and returning for the following academic year at different times. Level of engagement Data on the use of PeerWise was downloaded by the unit coordinator, de-identified and sent to the student researcher for analysis. Australasian Journal of Educational Technology, 2017, 33(1). 38 Table 3 Overview of online survey on students’ perceptions of using PeerWise and Moodle quizzes Section Item Type of response 1. Informed consent Yes/No 2. Requirements - Contribute 2 questions was reasonable? If no, ideal number? - Number of questions to answer per PeerWise topic was reasonable? If no, ideal number? - Frequency of PeerWise topics reasonable? If no, ideal frequency? - Number of questions and time for Moodle quiz reasonable? If no, ideal number and time? Ratings + open response 3. Cognitive domains: (i) Creating questions in PeerWise … (ii) Answering questions in PeerWise … (iii) Comments/feedback on PeerWise questions … (iv) Collaborating in a group to create a Moodle quiz … Answering the Moodle quizzes … … helped me to: - recall information from the readings. [Remembering] - expose and challenge misconceptions about the readings. [Understanding] - develop skills needed to implement interventions described in readings. [Applying] - discover links and relationships between ideas related to the readings. [Analysing] - think critically about what I am reading. [Evaluating] - develop my own ideas about what I am reading. [Creating] Ratings 4. Metacognitive/Attitudes: PeerWise/Moodle Overall, participation in ___________ helped me to: - Take ownership of my learning. - Obtain feedback on my level of understanding. - Monitor my learning progress. - Develop as an adaptive practitioner. - Develop confidence in my learning. - Develop my ‘voice’ as a professional psychologist. - Actively participate in classroom discussion. - Become a part of a learning community. - Value other people’s perspectives. Ratings 5. Benefits and limitations Contribution to learning, if any, of scoring system of question ratings and badges in PeerWise? Benefits, if any, of using PeerWise over Moodle quizzes? Benefits, if any, of using Moodle quizzes over PeerWise? Open responses 6. Recommendation for future cohorts o Use only PeerWise o Use only Moodle quizzes o Use a combination of PeerWise and Moodle quizzes o Use neither PeerWise nor Moodle quizzes Select one 7. Open feedback Any further comments about use of quizzes to support student engagement and learning Open response Note: Ratings used a 5-point Likert scale (1 = strongly disagree; 2 = disagree; 3 = normal; 4 = agree; 5 = strongly agree). Australasian Journal of Educational Technology, 2017, 33(1). 39 Results Twenty students completed the online survey, a response rate of 77%. Data from one participant was omitted as there were more than 50% missing responses. Descriptive statistics are presented to summarise students’ perceptions of their learning experiences across the range of activities addressed in the online survey. Due to significant departures from normality on most variables and low sample size (N = 19), non- parametric analyses were conducted. Holm’s procedure was applied to adjust for inflation of Type 1 error rate with multiple comparisons (Aickin & Gensle, 1996). Engagement All but one student completed the requirement to author two questions per PeerWise topic (16 questions in total). The remaining student authored 14 questions. On average, students submitted 414.07 answers (SD = 7.79), as expected if students were answering all questions submitted by their peers for each topic (26 x 2 x 8 = 416). The number of comments written per student ranged from 8 to 151, with an average of 44.81 comments (SD = 38.00). Most students (N = 16; 84.2%) thought the requirement to author two questions per PeerWise topic was reasonable. All three students who did not agree preferred one question per topic. There was less satisfaction with the number of questions to be answered, with 52.6% of students (N = 10) indicating having to answer all questions for each PeerWise topic was not reasonable. Among these students there was clear consensus that 25 to 30 questions would be more appropriate – about half the current number. All but one student (94.7%) indicated the frequency of PeerWise topics was reasonable. Fewer students (N = 15; 78.9%) agreed the frequency of Moodle quizzes was reasonable, with suggestions of both more or less frequent as preferences. Similarly, most students (N = 15; 78.9%) thought the length of time available and number of questions for each Moodle quiz was reasonable. Others suggested no time limit and fewer questions would be preferable. Contributions to learning Students’ perceptions of how PeerWise and Moodle quizzes contributed to their learning were examined for different activities across level of cognitive complexity (e.g., remembering; understanding; applying; analysing; evaluating; creating). PeerWise activities A comparison of perceptions of the contributions of different PeerWise activities to learning is illustrated in Figure 1. A series of Friedman analyses of variance indicates that these differences were significant only for contributions to helping students remember information (χ2 (2) = 11.56, p = .003) and apply knowledge from the readings (χ2 (2) = 14.33, p = .001). In both cases, students were more likely to agree that authoring a question contributes to learning compared to commenting on a question (author – remember vs. comments – remember, T = 7.00, p = .06, r =.30; author – apply vs. comments – apply, T = 15.00, p = .033, r = .36). There were no significant differences in how helpful authoring, answering, and commenting on questions in PeerWise were perceived to be for understanding, analysing, evaluating and creating knowledge. Australasian Journal of Educational Technology, 2017, 33(1). 40 Figure 1. Mean and standard error of students’ ratings of contribution to learning of authoring, answering, and commenting on questions in PeerWise (PW) (1 = strongly disagree to 5 = strongly agree) Answering questions: PeerWise versus Moodle Answering questions in PeerWise was perceived to be more helpful than answering questions on the Moodle quiz across all cognitive levels (see Figure 2). Wilcoxon tests show that this difference was significant in the case of understanding (T = 0, p = .002, r = .46), applying (T = 4.00, p = .031, r = .37), evaluating (T = .00, p = .016, r = .39), and creating information (T = .00, p = .016, r = .40). However, applying Holm’s correction for family wise error, answering in PeerWise was perceived to make a greater contribution to understanding than the Moodle quiz. Figure 2. Mean and standard error of students’ ratings of the contribution to learning of answering questions in PeerWise (PW) versus Moodle quizzes (1 = strongly disagree to 5 = strongly agree) 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 Remember Understand Apply Analyse Evaluate Create Author PW Answer PW Comment PW 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 Remember Understand Apply Analyse Evaluate Create Answer PeerWise Answer Moodle Australasian Journal of Educational Technology, 2017, 33(1). 41 Collaborative quiz groups Students perceived the requirement to work in groups to create a Moodle quiz for other students to be the least helpful for their learning. At least half the students disagreed or strongly disagreed with the statement that this group activity helped their learning, across all levels of cognitive complexity (M = 2.44 (SD = 1.04) to 2.68 (SD = 1.16)). Possible reasons for the lack of endorsement of the collaborative task in this context came to light in the analysis of the qualitative data. Metacognitive awareness: PeerWise versus Moodle Students’ perceptions of how completing PeerWise activities compared to the Moodle quiz helped to develop their metacognitive awareness did not differ across most of the areas explored in the survey questions (summarised in Figure 3). A trend suggesting an advantage for PeerWise in helping students to participate in class discussions (T = 7.00, p = .074, r = .30); and feel like part of a learning community (T = 7.00, p = .031, r = .35) was observed. However, these effects were not significant when Holm’s correction was applied. Figure 3. Mean ratings (and standard error) of students’ perceptions of how participating in PeerWise and Moodle tasks contributed to their overall learning (1 = strongly disagree to 5 = strongly agree) Benefits and limitations: PeerWise versus Moodle When asked to make a recommendation on how PeerWise and Moodle should be used with future cohorts, 73.7% of students (N = 14) nominated using a combination of PeerWise and Moodle. Of the remaining students, three suggested using Moodle quizzes only, one student nominated PeerWise only, and the remaining student responded “neither”. Open-ended comments on the benefits of PeerWise over Moodle (and vice versa) submitted via the survey were analysed by the first author to identify common themes. Comments were broken down into units, where each unit represented a single idea. Content analysis led to the definition of categories used to code responses and record the number of participants who made a comment related to each category. Comments were coded as positive (agreement with the category statement) or negative (contrary to the category statement). The emerging themes related to metacognitive/regulation of learning (self, peer, expert); social aspects of learning (discussion, engagement); and motivational aspects (reading, grade, fun). These categories were then used by the second author to verify the coding of responses. There was a high level of agreement, with less than 5% of coded comments with discrepancies needing to be resolved. A summary of responses appears in Table 4. 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 PeerWise Moodle Australasian Journal of Educational Technology, 2017, 33(1). 42 Overall, students found using PeerWise to be an enjoyable experience that was less stressful than Moodle quizzes. The perceived benefits included self-regulation of learning and increased opportunities to benchmark progress, obtain feedback and discuss understandings with peers. The following examples reflect this majority view: [PeerWise] Gives an opportunity to practice before the graded quiz. Allows you to check in on your readings, that you are on the same track as your peers. View and understand others opinions and critically evaluate the content of the readings. Less pressure to anwer [sic] correctly. Highlights where you need to read further or more deeply. Whilst not applicable to many, I enjoyed peerwise because I would be thinking of potential questions while reading and thinking about interesting ways to write it up. It made me feel involved and like a valuable contributor. Especially when most were selected for the moodle quizzes. I also prefer not having a time limit as I can actually think about questions and not stress about running out of time. Table 4 Number of student comments relating to cognitive, social and motivational benefits of PeerWise versus Moodle based on a thematic content analysis of responses to open-ended survey questions Benefits of: PeerWise over Moodle Moodle over PeerWise Categories Pro (Con) Pro (Con) M et ac og ni tiv e Self Monitor learning and identify gaps in knowledge. 8 - 2 - Peer Check progress against peers; obtain feedback on questions. 4 (1) - - Expert Level of achievement assessed relative to expert/teachers’ perspective. - (2) 5 - So ci al Discussion Stimulate discussion, open to other ideas. 7 - - - Engagement Interactive, valuable contributor. 3 (1) - (1) M ot iv at io n Less pressure Ok to get it wrong; no time limit. 9 - 5 (1) Reading Motivates reading more deeply. 5 (1) 6 (1) Grade Score is important motivator. - - 5 - Fun Enjoyable activity. 2 - - - Although PeerWise was considered to be a useful practice exercise, many students expressed the view that having a grade based on performance on the Moodle quiz was an important motivator and indicator of their progress. Further, there were clear indications that students did not see peer-based feedback on learning to be a legitimate assessment of their progress. Several students (N = 5) clearly expressed a preference for Moodle quizzes that contained questions selected by the expert (unit coordinator). For example: Peerwise seems useful as a tool for self-monitoring learning and progress, but it doesn’t give any indication of actual progress because the content is not set by the lecturer. It [Moodle quiz] legitimizes the reading and peerwise work – it reminds you to take the task seriously because you will need to do the test at the end of the week. Felt a greater sense of accomplishment after completing a quiz. Felt I worked harder. Australasian Journal of Educational Technology, 2017, 33(1). 43 Overall, students appreciated the formative, collaborative nature of PeerWise. However, the grade associated with a Moodle quiz is still regarded as important by students as an indicator of progress – especially if those questions are associated with the status of an expert. The following student comment sums up this viewpoint well: Use Peerwise as a fun and engaging tool that people can use to revise as they read. Use Moodle quizzes, written by the lecturer, as a more reliable and valid measure of students’ progress. PeerWise question ratings and badges The most frequent comment about the scoring system of ratings and badges in PeerWise was about motivation. Eight students indicated it provided extrinsic motivation to participate, although two students presented the contrary view. Some students (N = 6) noted the scoring system was difficult to understand, confusing, and the ratings unreliable, whereas only one student found the system worked well. Four students commented that the ratings and badges system did not contribute to their learning or was distracting. Two students found the system ”fun” but two others noted the novelty soon wore off. The following comment seems to best capture the majority view: I’m not sure the scoring system contributed much to my learning overall. Seeing that questions that were more highly rated tended to be more applied and creative, encouraged me to try and write more and applied and creative questions. However, I don’t thik [sic] everyone rated according to the same criteria with some people being more generous/harsh than others. The badges were useful in so far that they encouraged me to do different things, such as rate, comment etc. Further comments Most frequently students identified a need to improve processes to ensure authored questions in PeerWise provided adequate and even coverage of the readings. Explicit instruction on how to write quality multiple choice questions, agreement on using the ratings of questions and introducing an assessment rubric were also suggested improvements. Discussion The aim of the research was to explore how changing the pedagogical context of using online MCQ quizzes influenced students’ perceptions of their learning experience. Instead of online summative MCQ quizzes, PeerWise was introduced as a CSP, engaging students in authoring, answering and evaluating MCQs as the formative component of the unit. As expected, there was a high level of engagement with PeerWise as all students completed the requirements for authoring and answering questions. This level of participation is not surprising given all students were provisionally or generally registered professional psychologists, and the learning culture openly promoted collegial mentoring and co-responsibility for learning and assessment. Students reported they enjoyed using PeerWise and found the experience less stressful than completing the summative quizzes. However, feedback about workload led to a reduction in required number of questions to be created (and hence answered) for the next student cohort. It was noted with this new arrangement that students put more time into the single questions resulting in even greater depth and complexity of questions. Greater variation in the number of comments on questions per student is evidence of differences in level of engagement for the voluntary (not assessed) components of the task. Students perceived the activities of authoring and answering questions in PeerWise to be equally helpful to their learning across all levels of cognitive complexity from remembering to creating ideas. Authoring questions in PeerWise was perceived to be more helpful for remembering and applying knowledge than commenting on others’ questions. As previously suggested, authoring questions in PeerWise promotes knowledge building through deep learning and reflection (Luxton-Reilly et al., 2010; Rhodes, 2013), and the requirement to include a rationale and explanation for the correct response (McQueen et al., 2014). Alternatively, answering questions is considered to be helpful for revision and knowledge consolidation (Luxton-Reilly et al., 2010; McQueen et al., 2014; Rhodes, 2013), and may therefore have been expected to be more likely associated with remembering than other cognitive processes. Interestingly, answering questions in PeerWise was perceived to be more helpful for learning than answering questions in Moodle across the more complex cognitive skills, understanding, applying, evaluating and creating ideas – and not Australasian Journal of Educational Technology, 2017, 33(1). 44 for the least complex skill of remembering. This finding suggests answering MCQs in PeerWise and Moodle serve equally well as a ”drill-practice” test of recall (McQueen et al., 2014). The perceived advantage for answering questions in PeerWise to facilitate higher level cognitive skills suggests the formative approach promotes deeper engagement with learning. However, further research is required to verify these findings as the difference was only significant for understanding when corrections were applied. The qualitative analysis supports the suggestion that it is the social and formative nature of answering questions in PeerWise that students see as a benefit for their learning. The most frequent themes recognise the importance of “it being ok to get it wrong”, identifying gaps in knowledge, checking progress and obtaining feedback from peers, and being open to other’s ideas. These comments mirror many of the advantages of peer-based assessment identified in the literature (Falchikov, 2007; Nicol & Macfarlane- Dick, 2007; Sondergaard & Mulder, 2012). The value of collaboration through making comments and evaluating peers’ contributions was perceived to be as equally helpful as authoring and answering questions in PeerWise. However, creating questions was perceived to be more helpful for remembering and applying knowledge than making comments on others’ questions. Despite the perceived benefits of PeerWise as a CSP, participants questioned the legitimacy of peer evaluation because the content was not set by the expert (teacher). Similarly, the summative Moodle quiz was seen to offer reward for effort in attainment of grades – but only if recognition of progress was determined by the expert, rather than peers. These findings underline the perceived value of the teacher as an important source of feedback (Nicol & Macfarlane-Dick, 2007). This view perhaps helps to explain why the group collaborative task, where peers have responsibility for curating questions for the summative quiz, was not seen to make a valuable contribution to learning in the current context. Participating in PeerWise and Moodle quizzes was perceived to be equally beneficial in terms of helping students take ownership of their learning, obtain feedback and monitor progress. However, there were slight advantages observed for PeerWise to contribute to their participation in class and feeling part of a learning community, compared to Moodle. Although only a slight, non-significant trend in the survey responses, the feelings of benefits of PeerWise over Moodle in terms of increased engagement and enjoyment were evident in the qualitative feedback. In this sense, introducing PeerWise was perhaps more effective in supporting the broader blended learning model in promoting interaction. These conclusions are constrained by the limitations of the research; in particular, the small sample restricted by size of the cohort eligible to participate reduced the power of statistical analysis, with the trends observed requiring replication. Although subjective ratings of how students perceived their learning experience were supported by qualitative analysis, the subjectivity of the data also needs to be taken into consideration. Due to ethical considerations student perceptions were not linked to learning outcomes in the current study. Future research could consider exploring the relationship between performance-based measures available in PeerWise and other learning indicators. Although students in this cohort found the badges and scoring system provided encouragement, there was some uncertainty about their usefulness. The current research extends the evaluation of PeerWise to the postgraduate psychology context, adding further understanding to how students perceive authoring, answering and evaluating how MCQs contribute to their learning in a shared online environment. Differences observed in students’ perceptions of using PeerWise versus Moodle online quizzes as formative and summative assessment helps to inform decisions about the role of formative and summative tasks in the overall assessment model. A clear majority of students recommended retaining a combination of PeerWise and Moodle quizzes for future cohorts. The results are interesting because, although they underscore the benefits of self and peer-based formative assessment noted in previous research (e.g., Hardy et al., 2014; Luxton-Reilly & Denny, 2010; Nicol & Macfarlane-Dick, 2007), when it came to credentialling achievement students preferred the expert to curate content for assessment. Similar reports of students’ lack of confidence in the quality, value and fairness of peer assessment has been reported in several studies (Ballantyne, Hughes, & Mylonas, 2002). Trust in their own, and in their peer’s, ability to judge their work is identified as an important interpersonal variables influencing students’ perceptions of peer assessment (van Gennip, Segers, & Tillema, 2009). It would be interesting to explore whether these dynamics change in informal learning contexts, such as the increasing student-driven use of social media to support peer study networks (Dabbagh & Kitsantas, 2012). Encouraging students to create networks of peers as critical friends, along with reducing reliance on teacher Australasian Journal of Educational Technology, 2017, 33(1). 45 feedback, may help to advance student capability as lifelong learners and evidence-based practitioners (Boud, 2000; Thomas et al., 2011). Recommendations for future cohorts include providing training on how to write quality MCQs, increased involvement in the expert commentary on questions as part of the PeerWise community, and a combination of teacher input and peer-ratings to be used in the selection of questions for grading purposes. Previous research suggests that the quality of peer assessment improves with both training and experience of the participants (Van Zundert, Sluijsmans, & van Merriënboer, 2010). In this regard, it would be helpful to introduce an assessment rubric and explicit instructions on the use of ratings to support peer-based review of the quality of questions. Explicitly reinforcing the benefits of peer feedback by creating opportunities to illustrate positive examples of giving and receiving feedback – in classroom and practice-based settings – may increase students’ confidence in the validity of peer evaluation. Interestingly, McQueen et al. (2014) report a strong, positive correlation between teacher and peer ratings of the quality of questions submitted to PeerWise (r = 0.73), suggesting students may have cause to be more assured by their peers’ evaluation of standards. In subsequent iterations of the unit the assessment model has been altered to take into account the level of difficulty of questions set, number of questions answered and comments made. Adding questions from the previous year’s PeerWise question bank to the Moodle quizzes has provided further opportunity to maximise learning feedback for students. We have also noticed that reducing the requirement for students to contribute only one question per week has improved the creativity, depth and sophistication of the PeerWise questions. In conclusion, there are clear synergies in the way learning and learners are viewed through the lens of EBP, social constructivism and lifelong learning that can inform decisions about appropriate pedagogy. Engaging students as active, collaborative learners based around self and peer-based formative assessment plays an important part in preparing graduates to adapt their practices to accommodate new knowledge in new contexts (Boud, 2000; Nicol & MacFarlane-Dick, 2007). Introducing PeerWise into the blended learning model for training professional psychologists has been successful in the way it has encouraged deeper learning and promoted engagement with peers. Further refinements to the model should focus on increasing student confidence in their own and peer’s capabilities as a part of the next generation community of practice. References Aickin, M., & Gensle, H. (1996). Adjusting from multiple testing when reporting research results: The Bonferroni vs Holm methods. Americal Journal of Public Health, 86(5), 726–728. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1380484/ Angus, S. D., & Watson, J. (2009). Does regular online testing enhance student learning in the numerical sciences? Robust evidence from a large data set. British Journal of Educational Technology, 40(2), 255–272. doi:10.1111/j.1467-8535.2008.00916.x Babione, J. M. (2010). Evidence-based practice in psychology: An ethical framework for graduate education, clinical training, and maintaining professional competence. Ethics & Behaviour, 20(6), 443–43. doi:10.1080/10508422.2010.521446 Ballantyne, R., Hughes, K., & Mylonas, A. (2002). Developing procedures for implementing peer assessment in large classes using an action research process. Assessment & Evaluation in Higher Education, 27(5), 427–441. doi:10.1080/0260293022000009302 Bauer, R. M. (2007). Evidence-based practice in psychology: Implications for research and research training. Journal of Clinical Psychology, 63(7), 685–694. doi:10.1002/jclp.20374 Belar, C. (2011). Lifelong learning and psychology. Monitor on Psychology, 41(7), 52. Retrieved from http://www.apa.org/monitor/2010/07-08/soe.aspx Black, P., & William, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappa, 80(2), 139–144. Retrieved from https:www.semanticscholar.org Boud, D. (2000). Sustainable assessment: Rethinking assessment for the learning society. Studies in Continuing Education, 22(2), 151–167. doi:10.1080/713695728 Boud, D., Cohen, R., & Sampson, J. (1999). Peer learning and assessment. Assessment & Evaluation in Higher Education, 24(4), 413–426. doi:10.1080/0260293990240405 Boud, D., & Falchikov, N. (2006). Aligning assessment with long-term learning. Assessment & Evaluation in Higher Education, 31(4), 399–413. doi:10.1080/2602930600679050 Boud, D., & Molloy, E., (2013). Rethinking models of feedback for learning: The challenge of design. Assessment & Evaluation in Higher Education, 38(6), 698–712. doi:10.1080/02602938.2012.691462 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1380484/ http://dx.doi.org/10.1111/j.1467-8535.2008.00916.x http://dx.doi.org/10.1080/10508422.2010.521446 http://dx.doi.org/10.1080/0260293022000009302 http://dx.doi.org/10.1002/jclp.20374 http://www.apa.org/monitor/2010/07-08/soe.aspx https://www.semanticscholar.org http://dx.doi.org/10.1080/713695728 http://dx.doi.org/10.1080/0260293990240405 http://dx.doi.org/10.1080/02602938.2012.691462 Australasian Journal of Educational Technology, 2017, 33(1). 46 Collins, F. L., Leffingwell, T., & Belar, C. D. (2007). Teaching evidence-based practice: Implications for psychology. Journal of Clinical Psychology, 63(7), 651–670. doi:10.1002/jclp.20378 Dabbagh, N., & Kitsantas, A. (2012). Personal learning environments, social media, and self-regulated learning: A natural formula for connecting formal and informal learning. Internet and Higher Education, 15(1), 3–8. doi:10.1016/j.iheduc.2011.06.002 Denny, P., Luxton-Reilly, A., & Hamer, J. (2008). The Peer-Wise system of student-contributed assessment questions. In S. Hamilton & M. Hamilton (Eds.), Proceedings of the Tenth Australasian Computing Education Conference (ACE2008) (Vol. 78) (pp. 69–74). Sydney: Australian Computer Society. Retrieved from http://dl.acm.org/citation.cfm?id=1379255 Dzubian, C. D., Hartman, J. L., & Moskal, P. D. (2004). Blended learning. Educause Center for Applied Research: Research Bulletin, 7. Retrieved from http://net.educause.edu/ir/library/pdf/erb0407.pdf Falchikov, N. (2007). The place of peers in learning and assessment. In D. Boud & N. Falchikov (Eds.), Rethinking assessment in higher education: Learning for the longer term (pp. 128–143). London: Routledge. Frels, R. K., & Onwuegbuzie, A. J. (2013). Administering quantitative instruments with qualitative interviews: A mixed research approach. Journal of Counseling & Development, 91(2), 184–194. doi:10.1002/j.1556-6676.2013.00085.x Fraser, S. W., & Greenhalgh, T. (2001). Coping with complexity: Educating for capability. British Medical Journal, 323, 799–803. doi:10.1136/bmj.323.7316.799 Hamer, J., Cutts, Q., Jackova, J., Luxton-Reilly, A., McCartney, R., Purchase, H, … & Sheard, J. (2008). Contributing student pedagogy. SIGSCE Bulletin, 40(4), 194–212. doi:10.1145/1473195.1473242 Hardy, J., Bates, S. P., Casey, M. M., Galloway, K. W., Galloway, R. K., Kay, A. E., Kirsop, P., & McQueen, H. A. (2014). Student-generated content: Enhancing learning through sharing multiple- choice questions. International Journal of Science Education, 36(13), 2180–2194. doi:10.1080/09500693.2014 Kanuka, H., & Anderson, T. (1999). Using constructivism in technology-mediated learning: Constructing order out of the chaos in the literature. Radical Pedagogy, 2(1). Retrieved from http://www.radicalpedagogy.org/radicalpedagogy/Using_Constructivism_in_Technology- Mediated_Learning__Constructing_Order_out_of_the_Chaos_in_the_Literature.html Kibble, J. D. (2011). Voluntary participation in online formative quizzes is a sensitive predictor of student success. Advances in Physiology Education, 35(1), 95–96. doi:10.1152/advan.00053.2010 Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory into Practice, 41(4), 212–218. doi:10.1207/s15430421tip4104_2 Lilienfeld, S. O., Ritschel, L. A., Lynn, S. J., Cautin, R. L., & Latzman, R. D. (2013). Why many clinical psychologists are resistant to evidence-based practice: Root causes and constructive remedies. Clinical Psychology Review, 33(7), 883–900. doi:10.1016/j.cpr.2012.09.008 Luebbe, A. M., Radcliffe, A. M., Callands, T. A., Green, D., & Thorn, B. E. (2007). Evidence-based practice in psychology: Perceptions of graduate students in scientist-practitioner programs. Journal of Clinical Psychology, 63(7), 643–655. doi:10.1002/jclp.20379 Luxton-Reilly, A., & Denny, P. (2010). Constructive evaluation: A pedagogy of student-contributed assessment. Computer Science Education, 20(2), 145–167. doi:10.1080/08993408.2010.486275 Luxton-Reilly, A., Denny, P., Plimmer, B., & Sheehan, R. (2012). Activities, affordances and attitude: How student-generated questions assist learning. Proceedings of ITiCSE’12, 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (pp. 4–9). New York, NY: ACM. doi:10.1145/2325296.2325302 McKenzie, W. A., Perini, E., Rohlf, V., Toukhsati, S., Conduit, R., & Sanson, G. (2013). A blended learning lecture delivery model for large and diverse undergraduate cohorts. Computers & Education, 5, 116–126. doi:10.1016/j.compedu.2013.01.00 McQueen, H. A., Shields, C., Finnegan, D. J., Higham, J., & Simmen, M. W. (2014). PeerWise provides significant academic benefits to biological science students across diverse learning tasks, but with minimal instructor intervention. Biochemisty and Molecular Biology Education, 42(5),371–381. doi:10.1002/bmb.20806 Nicol, D. J. (2007). E-assessment by design: using multiple-choice tests to good effect. Journal of Further and Higher Education, 31(1), 53–64. doi:10.1080/03098770601167922 Nicol, D. J. & Macfarlane-Dick, D. (2007). Formative assesssment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. doi:10.1080/03075070600572090 http://dx.doi.org/10.1002/jclp.20378 http://dx.doi.org/10.1016/j.iheduc.2011.06.002 http://dl.acm.org/citation.cfm?id=1379255 http://net.educause.edu/ir/library/pdf/erb0407.pdf http://dx.doi.org/10.1002/j.1556-6676.2013.00085.x http://dx.doi.org/10.1136/bmj.323.7316.799 http://dx.doi.org/10.1145/1473195.1473242 http://dx.doi.org/10.1080/09500693.2014 http://www.radicalpedagogy.org/radicalpedagogy/Using_Constructivism_in_Technology-Mediated_Learning__Constructing_Order_out_of_the_Chaos_in_the_Literature.html http://www.radicalpedagogy.org/radicalpedagogy/Using_Constructivism_in_Technology-Mediated_Learning__Constructing_Order_out_of_the_Chaos_in_the_Literature.html http://dx.doi.org/10.1152/advan.00053.2010 http://dx.doi.org/10.1207/s15430421tip4104_2 http://dx.doi.org/10.1016/j.cpr.2012.09.008 http://dx.doi.org/10.1002/jclp.20379 http://dx.doi.org/10.1080/08993408.2010.486275 http://dx.doi.org/10.1145/2325296.2325302 http://dx.doi.org/10.1016/j.compedu.2013.01.00 http://dx.doi.org/10.1080/03098770601167922 http://dx.doi.org/10.1080/03075070600572090 Australasian Journal of Educational Technology, 2017, 33(1). 47 Palincsar, A. S. (1998). Social constructivist perspectives on teaching and learning. Annual Review of Psychology, 49(1), 345–375. doi:10.1146/annurev.psych.49.1.345 Peat, M., & Franklin, S. (2002). Supporting student learning: the use of computer–based formative assessment modules. British Journal of Educational Technology, 33(5), 515–523. doi:10.1111/1467- 8535.00288 Purchase, H., Hamer, J., Denny, P., & Luxton-Reilly, A. (2010). The quality of a PeerWise MCQ repository. In T. Clear & J. Hamer (Eds.), Proceedings of the Twelth Australasian Computing Education Conference (ACE ’10) (pp. 137–146). Sydney: Australian Computer Society. Retrieved from http://dl.acm.org/citation.cfm?id=1862238 Ramsden, P. (2003). Learning to teach in higher education (2nd ed.). London: Routledge-Falmer. Rhodes, J. (2013). Using PeerWise to knowledge build and consolidate knowledge in nursing education. Southern Institute of Technology Journal of Applied Research (SITJAR). Retrieved from https://www.researchgate.net/publication/265396191_Using_PeerWise_to_Knowledge_Build_and_C onsolidate_Knowledge_in_Nursing_Education Rolloff, M. (2010). A constructivist model for teaching evidence-based practice. Nursing Education Perspectives, 31(5), 290–293. Retrieved from http://journals.lww.com/neponline/toc/2010/09000 Sondergaard, H., & Mulder, R. A. (2012). Collaborative learning through formative peer review: pedagogy, programs and potential. Computer Science Education, 22(4), 343–367. doi:10.1080/08993408.2012.728041 Spring, B. (2007). Evidence-based practice in clinical psychology: What it is, why it matters, what you need to know. Journal of Clinical Psychology, 63(7), 611–631. doi:10.1002/jclp.20373 Stull, J. C., Varnum, S. J., Ducette, J., Schiller, J., & Bernacki, M. (2011). The effects of formative assessment pre-lecture online chapter quizzes and student-initiated inquiries to the instructor on academic achievement. Educational Research and Evaluation, 17(4), 253–262. doi:10.1080/13803611.2011.6211756 Taras, M. (2002). Using assessment for learning and learning from assessment. Assessment & Evaluation in Higher Education, 27(6), 501–510. doi:10.1080/026022000020273 Taras, M. (2005). Assessment – summative and formative – some theoretical reflections. British Journal of Educational Studies, 53(4), 466–478. doi:10.1111/j.1467-8527.2005.00307.x Thomas, A., Saroyan, A., & Dauphinee, W. D. (2011). Evidence-based practice: A review of theoretical assumptions and effectiveness of teaching and assessment interventions in health professions. Advances in Health Science Education, 16(2), 253–276. doi:10.1007/s10459-010.9251-6 van Gennip, N. A. E., Segers, M. S. R., & Tillema, H. H. (2009). Peer assessment for learning from a social perspective: The influence of interpersonal variables and structural features. Educational Research Review, 4(1), 41–54. doi:10.1016/j.edurev.2008.11.002 Van Zundert, M., Sluijsmans, D., & van Merriënboer, J. (2010). Effective peer assessment processes: Research findings and future directions. Learning and Instruction, 20(4), 270–279. doi:10.1016/j.learnininstruc.2009.08.004 Vaughan, N. (2007). Perspectives on blended learning in higher education. International Journal on E- learning, 6(1), 81–94. Retrieved from https://www.learntechlib.org/p/6310 Vygotsky, L. (1997). Interaction between learning and development. .In M. Gauvain & M. Cole (Eds.), Readings on the development of children (pp. 29–36). New York, NY: W. H. Freeman. (Reprinted from Mind and society, pp. 79–91, 1978, Cambridge, MA: Harvard University Press) Yorke, M. (2005). Formative assessment in higher education: Its significance for employability, and steps toward enhanement. Tertiary Education and Management, 11(3), 219–238. doi:10.1007/s11233-005- 5110-z Corresponding author: Wendy McKenzie, wendy.mckenzie@monash.edu Australasian Journal of Educational Technology © 2017. Please cite as: McKenzie, W., & Roodenburg, J. (2017). Using PeerWise to develop a contributing student pedagogy for postgraduate psychology. Australasian Journal of Educational Technology, 33(1), 32-47. https://doi.org/10.14742/ajet.3169 http://dx.doi.org/10.1146/annurev.psych.49.1.345 http://dx.doi.org/10.1111/1467-8535.00288 http://dx.doi.org/10.1111/1467-8535.00288 http://dl.acm.org/citation.cfm?id=1862238 https://www.researchgate.net/publication/265396191_Using_PeerWise_to_Knowledge_Build_and_Consolidate_Knowledge_in_Nursing_Education https://www.researchgate.net/publication/265396191_Using_PeerWise_to_Knowledge_Build_and_Consolidate_Knowledge_in_Nursing_Education http://journals.lww.com/neponline/toc/2010/09000 http://dx.doi.org/10.1080/08993408.2012.728041 http://dx.doi.org/10.1002/jclp.20373 http://dx.doi.org/10.1080/13803611.2011.6211756 http://dx.doi.org/10.1080/026022000020273 http://dx.doi.org/10.1111/j.1467-8527.2005.00307.x http://dx.doi.org/10.1007/s10459-010.9251-6 http://dx.doi.org/10.1016/j.edurev.2008.11.002 http://dx.doi.org/10.1016/j.learnininstruc.2009.08.004 https://www.learntechlib.org/p/6310 http://dx.doi.org/0.1007/s11233-005-5110-z http://dx.doi.org/0.1007/s11233-005-5110-z mailto:wendy.mckenzie@monash.edu https://doi.org/10.14742/ajet.3169 Teaching context Research questions Method Participants Design Materials and procedure In-class group feedback Online survey development Level of engagement Results Engagement Contributions to learning PeerWise activities Answering questions: PeerWise versus Moodle Collaborative quiz groups Metacognitive awareness: PeerWise versus Moodle Benefits and limitations: PeerWise versus Moodle Further comments Discussion References