The Effects of Face-to-face and Computer-mediated Peer Review on EFL Writers’ Peer Responses and Revisions Australasian Journal of Educational Technology, 2015, 31(1). 1 The effects of face-to-face and computer-mediated peer review on EFL writers’ comments and revisions Mei-ching Ho University of Taipei, Taiwan This study investigates the use of face-to-face and computer-mediated peer review in an English as a Foreign Language (EFL) writing course to examine how different interaction modes affect comment categories, students’ revisions, and their perceptions of peer feedback. The participants were an intact class of 13 students at a Taiwanese university. The computer-mediated peer review involved OnlineMeeting, software specifically designed for peer review activities and featuring a split screen protocol, document sharing, and chat room functions. The results of chi-square tests show that overall students offered more revision-oriented comments than non-revision-oriented ones among different writing tasks in either mode. Also, peer review mode affected some types of peer comments to a certain extent. There were significantly more global alteration comments and fewer local alteration comments in face-to-face than computer-mediated mode. While the participants liked comments via Word’s annotation features over handwritten comments, they felt face-to-face discussions to be more effective than online chat via OnlineMeeting due to the affordance of face-to-face talk (e.g., immediacy and paralinguistic features), that cannot be easily replaced by electronic chat. Pedagogical implications regarding the balanced use of computer-mediated and non-computer-mediated writing activities are discussed, along with suggestions for future research. Introduction As a process approach to writing instruction, peer review has been widely adopted in many English as a Second/Foreign Language (ESL/EFL) writing classrooms. It involves learners working in groups commenting on one another’s writing, and it serves as a source of feedback that supplements teachers’ comments. The purpose of peer review is to afford learners the experience in “expressing, interpreting, and negotiating meaning” through collaboration (Lee & VanPatten, 2003, p. 215). The benefits of peer review in general have been substantiated by many empirical investigations, which have reported that peer review facilitates revision, raises learners’ audience awareness, and exposes learners to various writing styles (Grabe & Kaplan, 1996; Min, 2006, 2008). With the development of computer-assisted language learning software, computer-mediated peer review (CMPR) has gained increasing attention among composition teachers in recent years. CMPR is considered to have an advantage over traditional face-to-face peer review (FFPR) because it can be conducted anywhere at any time with networked computers or tablets (DiGiovanni & Nagaswami, 2001; Guardado & Shi, 2007; Tuzi, 2004; Ware & Warschauer, 2006). While many studies have been conducted on the effect of peer review on learners’ writing quality, attitudes, and affective benefits, most of these studies investigated peer review in single communication mode (Breuch, 2004; Guardado & Shi, 2007; Hyland & Hyland, 2006; Liang, 2010; Liou & Peng, 2009; Miao, Badger, & Zhen, 2006; Min, 2006, 2008; Rollinson, 2005; Tuzi 2004; Wang, 2004). Only several have compared FFPR and CMPR (see Braine, 2001; Chang, 2012; Honeycutt, 2001; Liu & Sadler, 2003). Comparative studies on peer review have shown that the use of different communication modes can lead to differences in the number of comments and subsequent revisions (Hewett, 2000; Jones, Garralda, Li, & Lock, 2006; Liu & Sadler, 2003; Song & Usaha, 2009; Xu, 2007) and affect students’ interaction and attitudes toward peer review in general (DiGiovanni & Nagaswami, 2001; Ho & Savignon, 2007). However, the software (e.g., Norton Textra Connect, MOO’s, MSN, Email) used in previous studies was either outdated or was not specifically designed for peer review (Hewett, 2000). More studies involving up to date peer review software are needed to explore the influences of blended modes on peer review outcomes. In response to this need, this study aims to examine how a two step peer review procedure in two modes may shape EFL learners’ comment patterns, revisions, and perceptions. The CMPR in the Australasian Journal of Educational Technology, 2015, 31(1). 2 current study involves OnlineMeeting, an interface simulating a split-screen protocol and specifically designed for synchronous online peer review. Literature review Research on computer-mediated peer review As information technology becomes widely adopted in language classrooms, peer response has taken on a new dimension. With similar advantages to traditional face-to-face peer review, CMPR can facilitate collaborative learning with fewer temporal and spatial constraints. Many studies have investigated how computer-mediated communication (CMC) may affect peer interaction, comment patterns, and revisions (Guardado & Shi, 2007; Liang, 2010; Liou & Peng 2009; Moradi & Karimpour, 2012; Tuzi, 2004). For instance, Guardado and Shi (2007), examining how electronic feedback via discussion board postings affected 22 Japanese ESL students’ commenting behaviours and subsequent revisions, found that many peer comments were ignored and led to no revision. They speculated that several factors lay behind the poor adoption rate of peer feedback in students’ revisions, including learners’ uncertainty about the quality of the comments and a lack of negotiated interaction due to anonymous online postings. Therefore, they suggest writing teachers train students to interact with reviewers, especially in online posting contexts. Recognizing the importance of peer review training in online environments, Liou and Peng (2009) investigated the training effect of peer review via asynchronous weblog postings on the comment types and revisions of thirteen Taiwanese EFL college students. They found that after training, the students made more revision-oriented comments and more effective revisions. However, the training did not make students more willing to adopt peer comments and they had mixed feelings regarding peer review in a blog environment. Despite their students’ uncertainty about the effectiveness of peer feedback, Liou and Peng (2009) argue that, with proper training, asynchronous peer review via a weblog is still beneficial because it allows more time for formulating well thought out comments. Another line of CMPR research focuses on using synchronous CMC tools for peer review. Investigating online peer review interaction of twelve EFL college students using MSN Messenger, freeware for instant messaging, Liang (2010) found that turns on social talk, task management, and content discussion outnumbered turns on meaning negotiation and error correction. In addition, students’ online interaction and revision greatly depended on group makeup and dynamics. However, it should be noted that Liang’s findings may be influenced by the number of students in one chat session and the software used. Four students in one chat group could easily lead to confusing turn-taking in an online environment. Furthermore, due to the limitation of MSN system, the online chat window may block the view of peers’ drafts in a Word document and students would have to keep moving or minimizing it. This limitation may impede the efficiency of CMPR. Therefore, more studies that involve software specifically designed for peer review are needed. As Herring (2002) suggests, the media used for CMC can affect the style, coherence, and etiquette language used. Research on blended peer review modes Given the differences between face-to-face interaction and CMC, a growing body of research has compared peer responses via different communication modes. One of the research foci of these studies was to compare the effectiveness of various modes by examining peer review interaction, comment type, and revision. Liu and Sadler (2003), for instance, analyzed face-to-face and computer-mediated peer responses of eight ESL college students in the US. They reported that, while students using Word’s track changes function gave more comments on local-area issues (e.g., grammar and sentence-level problems) than commenting with pen and paper, the traditional handwritten responses were more balanced in both global- and local-area issues. Their students’ peer review talk via MOO, a multi-user online chat room, in contrast, generated “more superficial than substantive comments” (Liu & Sadler, 2003, p. 220). Furthermore, more comments made via the traditional mode were incorporated than those made via CMPR. They thus suggested L2 writing teachers combine Word commenting and face-to-face discussion Australasian Journal of Educational Technology, 2015, 31(1). 3 as a two step procedure for effective peer review. More recently, Chang (2012) had twenty-four Taiwanese EFL students conduct peer review using a combination of three modes: asynchronous posting, face-to-face talk, and MSN chat. She found that peer comments via face-to-face talk and asynchronous posting targeted local writing problems. The synchronous comments, on the other hand, emphasized global problems. Although Chang (2012) argued that personal factors (e.g., students’ writing proficiency) might affect comments types, her results may be skewed by a lack of proper training of each peer review mode at different stages of drafting. Another line of peer review research pertains to students’ perceptions of different modes. Many studies report that students have mixed feelings about the use of traditional and electronic peer review modes. DiGiovanni and Nagaswami (2001), for instance, found that only 53% of their 32 ESL students liked synchronous CMPR. Although CMPR provided a less threatening environment for real-time conversation, many of their students felt that FFPR was more efficient with regard to meaning negotiating and clarification. Liu and Sadler’s (2003) study also revealed that the students in technology-enhanced peer review groups liked MOO interaction more than commenting with Word’s track changes feature. However, the analysis of comments and student interaction indicate that the asynchronous Word commenting seemed to be more effective than MOO talk, in which off-task topics and conversation maintenance turns abounded. Although efforts have been made to investigate the effect of blended modes on peer comments and revisions, the results have been inconclusive. As Prior (2005) and Liang (2010) suggest, the use of technology in different contexts dictates the affordance of each mode and can result in distinctive outcomes. Furthermore, in many previous studies, peer review was mostly a one-step process in which students usually did not have enough time to formulate detailed comments in synchronous mode (see Braine, 2001; Jones et al., 2006; Liang, 2010). Nor did they have opportunities for further clarification in asynchronous mode (see Guardado & Shi, 2007; Liou & Peng, 2009). In this study, a two step peer review procedure of written comments followed by face-to-face talk vs. electronic comments followed by synchronous chats was adopted to explore the following three research questions: 1. How do different peer review modes (face-to-face vs. computer-mediated) affect students’ peer comments and what kinds of comments are generated in each mode? 2. To what extent do students revise their papers based on comments made in different modes? 3. What are students’ perspectives on the different peer review modes and the OnlineMeeting interface used in this study? Methodology Setting and participants This research was undertaken in an EFL writing course at a university in northern Taiwan. The course met 2 hours per week for a school year and was required for all English majors in their sophomore year. The main course objective was to foster students’ expository writing skills through a process writing approach. The current study took place in the first 18 week semester. The participants were an intact class of 13 sophomores, comprised of 1 man and 12 women (average age 20 years old). All were native speakers of Chinese and had learned English for an average of 8 years before being admitted to the university. Most students’ English proficiency ranged from intermediate to high-intermediate level on General English Proficiency Test (GEPT), administered by the Language Training and Testing Center in Taiwan, and was approximately between 72-82, on the Internet-based Test of English as a Foreign Language (TOEFL iBT). Only 6 students had done FFPR, and the others did not have peer review experience prior to the study. All the participants had to write four multi-draft expository essays on different topics throughout the semester. The four topics pertained to “How to deal with…(a certain problem)”, “How to use…(an electronic device)”, “Causes of any recent trend,” and “Effects of an event/problem.” For each assignment, Australasian Journal of Educational Technology, 2015, 31(1). 4 the researcher adopted a writing cycle modified from Min’s (2006) study (see Figure 1). Each cycle lasted about 4 weeks and consisted of writing three drafts with revisions in between based on peer comments and teacher feedback. The CMPR in this study was conducted using OnlineMeeting, an interface integrating document sharing and chat room functions (see Figure 2). It features a split-screen protocol, which simultaneously juxtaposes two windows, with one showing the instant chat dialogues and the other showing students’ drafts with peer comments. Figure 1. The writing cycle for each major assignment Australasian Journal of Educational Technology, 2015, 31(1). 5 Figure 2. The OnlineMeeting interface Peer review procedures To familiarise students with the two step peer review procedure, the teacher embarked on a training session, based on Rollinson’s (2005) and Min’s (2006) suggestions, in the first 2 weeks. First, the teacher explained the purposes and benefits of peer review and then showed sample comments that offered specific directions for revision. The teacher also discussed strategies on meaning negotiation, including clarifying writers’ rhetorical purposes and intentions, identifying and explaining problems, and giving concrete suggestions (Min, 2006). Then, with a peer review guidance sheet, the students were asked to review a sample paper on computer, using Word’s track changes function and later to discuss peer comments via instant chat in OnlineMeeting. After the training, students were paired up randomly to review each other’s papers. For the first and third assignments, students provided handwritten comments and then discussed peer comments face-to-face. For the second and forth assignments, students typed comments using Word’s track changes function, and then held a synchronous discussion via OnlineMeeting. The participants were allowed to write comments and discuss with their peers either in English or Chinese during FFPR and CMPR for ease of communication and meaning negotiation. Instruments The data was collected from multiple sources. First, documents related to the four writing assignments were elicited from each participant. These documents included all drafts, written and electronic comments, transcripts of face-to-face discussions, and online chat logs. Second, a questionnaire survey was used to obtain students’ background information and elicit their perceptions about the two different peer review modes and the OnlineMeeting interface. Toward the end of the semester, follow up interviews were also conducted in Chinese to better understand the strengths and weaknesses of the FFPR and CMPR procedures used in this study. Australasian Journal of Educational Technology, 2015, 31(1). 6 Data analysis The analysis of all peer comments from the four writing assignments was based on a modified version of Liu and Sadler’s (2003) analytical scheme in which the coding was based on the “sentential meaningful units” in a given comment (p. 203). To examine whether a comment targeted certain text areas, all peer comments were classified into global- and local-area comments, with global-area comments relating to content, organization, and rhetorical purposes of a writing task, and local-area comments relating to sentence level or copy editing problems such as word choice, grammar, and mechanics. The comments were further categorized into five types by discourse functions: evaluation of good or bad features of writing, clarification (asking for explanation or justification), suggesting directions for revision, alteration (offering specific changes to replace certain words, phrases, or sentences), responding to peer guidance sheet questions, and others. The category others was added mainly to include comments that fulfill other discourse functions such as agreeing/disagreeing with the writers’ opinions, providing reasons for making certain comments, and so on. Furthermore, all the comments were also coded by nature (revision-oriented versus non-revision oriented). All the comments were coded by two independent coders, following a three stage coding procedure. First, the comments from the first step review procedure were separated from those from the second step peer review (i.e., face-to-face talk transcripts and synchronous chat logs). Additionally, all the comments were categorized according to a modified version of Liu and Sadler’s (2003) rubric (See Appendix A). Because one of the purposes of the second step peer review was for students to clarify comments and negotiate meanings, some comments from the face-to-face discussion transcripts and chat logs overlapped with those in written/typed comments. Therefore, the comments from discussions and online chats that overlapped with the handwritten or typed comments were excluded for analysis. Furthermore, to examine how students incorporated peer comments into revision, all the changes in revised drafts were identified and compared with the comments students received. All the changes that could be traced to peer comments were counted as peer-triggered to calculate the comment adoption rate. Two coders independently coded 10% of the comments from the data set. The inter-rater reliability was 80% with a Cohen’s kappa value of .69 in the first round. The two coders then discussed the disagreement cases with a third coder until they reached an agreement. After the norming session, the two coders coded another 10% of the data, and the resulting kappa of .84 with 92% of agreement indicates a satisfactory inter-rater reliability. The two coders then proceeded to code the rest of the data. Findings Distribution of comments in FFPR and CMPR To get a better understanding of whether face-to-face and computer-mediated modes generate different comment patterns by area, nature, and type, the researcher examined the number and percentage of all the comments under the 24 categories in the analysis rubric. Table 1 shows the number and percentage of comments by area and nature among the four writing assignments by two different interaction modes. The results of chi-square tests reveal that statistically significant differences exist among the four paper assignments via either FFPR or CMPR, in terms of global revision-oriented (χ2 = 37.07, df = 3, p <.001), global non-revision-oriented (χ2 = 32.02, df = 3, p <.001), and local revision-oriented comments (χ2 = 54, df = 3, p <.001). In other words, when working on papers of different topics, students gave different proportions of comments by area and nature, except for local non-revision-oriented comments (χ2 = 6.12, df = 3, p >.05). This indicates that interaction mode may not be the only contributing factor behind the different distributions of comments. Other factors, such as the nature and the order of writing tasks might have affected the proportion of global-area and local-area comments. Further studies of this kind are needed to shed more light on this issue. Australasian Journal of Educational Technology, 2015, 31(1). 7 Table 1 Number and percentage of comments by area and nature among 4 papers in different modes FFPR1 CMPR1 FFPR2 CMPR2 p Global/Revision 118 (42%) 193 (52%) 293 (59%) 186 (42%) .000** Global/Non-revision 61 (22%) 66 (18%) 136 (27%) 154 (34%) .000** Local/Revision 86 (31%) 89 (24%) 52 (10%) 80 (18%) .000** Local/Non-revision 16 (6%) 21 (6%) 15 (3%) 28 (6%) .103 Total 281 369 496 448 FFPR1 = Paper 1, CMPR1 = Paper 2, FFPR 2 = Paper 3, CMPR2 = Paper4 **p < 0.01; The percentages do not add up to exact 100% due to rounding. In order to find out whether there was an interaction effect of communication mode and the nature of writing assignments on comment patterns, chi-square tests of homogeneity were conducted. The results show statistically significant differences in the percentage distribution of comments by area and nature in papers 1 and 3 via FFPR (χ2 = 56.29, df = 3, p <.001) and papers 2 and 4 via CMPR (χ2 = 29.45, df = 3, p <.001) modes. However, while no significant differences were found in the percentage distribution of comments in papers 1 and 2 (χ2 = 7.23, df = 3, p >. 05), a significant difference exists in papers 3 and 4 (χ2 = 32.53, df = 3, p <.001). Based on the analysis, when working on paper 3 via FFPR, students gave higher proportion of global, revision-oriented comments (59% vs. 42%) and lower proportion of local, revision-oriented comments (10% vs. 18%) than when working on paper 4 via CMPR. In other words, the nature of writing assignments could be one important co-varying factor that could lead to the differences in students’ comments, despite the fact that no significant difference was found in the proportions of different comments by area and nature in papers 1 and 2. It is worth noting that the first two and the last two writing assignments were similar in nature as papers 1 and 2 were problem-solution essays, and papers 3 and 4, focus-on-causes/effects essays. Further studies that involve an experimental research design are needed to identify exact contributing factors behind the different distributions of peer comments. Figure 3. Percentage of peer comments by area and nature in different PR modes Despite the differences, the revision-oriented comments accounted for at least 60% to 76% of the total comments among the four assignments. Furthermore, among the revision-oriented comments, as illustrated in Table 1 and Figure 3, the proportions of global revision-oriented comments were significantly higher than those of local revision-oriented comments in each writing assignment (χ2 = 62.23, df = 1, p <. 001). This indicates that the participants targeted more global issues than local issues in their peers’ writing and offered significantly more revision-oriented than non-revision oriented comments overall in either mode. 0% 10% 20% 30% 40% 50% 60% FFPR1 CMPR1 FFPR2 CMPR2 GR GN LR LN Australasian Journal of Educational Technology, 2015, 31(1). 8 Table 2 and Figure 4 demonstrate the cross-referenced results of all revision-oriented comments by area and type among the four papers. Even though the four writing assignments generated different distributions of revision-oriented comments, among the global, revision-oriented comments, suggestion was the most frequently used comment type, accounting for 42% and 40% in papers 1 and 3 via FFPR, and 37% and 48% in papers 2 and 4 via CMPR, followed by evaluation comments. As Table 2 shows, the proportions of suggestion and evaluation comments regarding the global area of writing showed no significant differences among the four papers. Furthermore, it is worth noting that the students offered significantly more global alteration comments in FFPR than in CMPR (17% and 18% respectively for papers 1 and 3 via FFPR vs. 7% and 9% for papers 2 and 4 via CMPR). This finding is encouraging, particularly because previous studies (e.g., Chang, 2012; Liu & Sadler, 2003) found no occurrence of global, revision-oriented alteration comments in either mode. Table 2 Number and percentage of global, revision-oriented comment types among four papers in different modes Global/Revision Comment types FFPR1 CMPR1 FFPR2 CMPR2 p Evaluation 27 (23%) 54 (28%) 65 (22%) 37 (20%) .28 Clarification 5 (4%) 24 (12%) 22 (8%) 12 (6%) .04* Suggestion 50 (42%) 71 (37%) 116 (40%) 90 (48%) .12 Alteration 20 (17%) 14 (7%) 54 (18%) 17 (9%) .001** Response 11 (9%) 3 (2%) 7 (2%) 13 (7%) .001** Other 5 (4%) 27 (14%) 29 (10%) 17 (9%) .04* Total 118 193 293 186 * p < 0.05, **p < 0.01; FFPR1 = Paper 1, CMPR1 = Paper 2, FFPR 2 = Paper 3, CMPR2 = Paper 4 The percentages do not add up to exactly 100% due to rounding. Figure 4. Percentage of global revision-oriented comment types in different PR modes 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% FFPR1 CMPR1 FFPR2 CMPR2 Australasian Journal of Educational Technology, 2015, 31(1). 9 Table 3 Number and percentage of local, revision-oriented comment types among four papers in different modes Local/Revision FFPR1 CMPR1 FFPR2 CMPR2 p Evaluation 15 (17%) 14 (16%) 3 (6%) 2 (3%) .005** Clarification 2 (2%) 7 (8%) 2 (4%) 4 (5%) .39 Suggestion 13 (15%) 10 (11%) 9 (17%) 6 (8%) .31 Alteration 49 (57%) 54 (61%) 32 (62%) 67 (84%) .001** Response 0 (0%) 1 (1%) 3 (6%) 0 (0%) n/a Other 7 (8%) 3 (3%) 3 (6%) 1 (1%) n/a Total 86 89 52 80 * p < 0.05, **p < 0.01; FFPR1 = Paper 1, CMPR1 = Paper 2, FFPR 2 = Paper 3, CMPR 2 = Paper 4 n/a: not available due to having observed values less than 5 in chi-square tests. The percentages do not add up to exactly 100% due to rounding. Figure 5. Percentage of local revision-oriented comment types in different PR modes As Table 3 and Figure 5 demonstrate, among the local revision-oriented comments, alteration comments, in which students directly fixed word- or structure/sentence-level problems by replacing the mistakes with a correct usage, occurred most frequently. Additionally, the proportions of local, revision-oriented alteration comments in papers via CMPR were significantly higher than those in papers via FFPR (χ2 = 15.89, df = 3, p <.001). One possible explanation is that during CMPR, the local errors regarding spelling, punctuation, and grammar could be easily identified by the spelling and grammar check function in MS Word and thus made students to make direct local changes in their peers’ writing. This finding corroborates Liu and Sadler’s (2003) results. The peer comment adoption rate in different peer review modes As shown in Table 4, although no statistically significant differences were found in the proportion of adopted comments in FFPR1/Paper1 and CMPR1/Paper2 (74% vs. 69%), the students integrated statistically significantly higher percentage of revision-oriented comments overall in FFPR2/paper 3 than in CMPR2/paper 4 (72% vs. 58%). Since the students worked on two different types of essays for the first two and the last two assignments, the nature of writing assignments might have affected the adoption rate, However, future studies are needed to confirm this. Also of note is the fact that, in both modes, students made significantly more revisions in the global area of their writing (e.g., content, idea development, and organisation) than in the local area at the word- or sentence-level. The overall adoption rate of global revision-oriented comments reached at least 70%, except for the fourth paper (59%). This comment 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% FFPR1 CMPR1 FFPR2 CMPR2 Australasian Journal of Educational Technology, 2015, 31(1). 10 adoption rate is higher than those reported in previous studies. For instance, Liou and Peng (2009) reported a 48% of adoption rate of revision-oriented comments in asynchronous CMPR. In Liu and Sadler’s (2003) research, among the revision-oriented comments, only 41% of e-comments and 27% written comments led to revision. Table 4 Percentage of peer comments leading to revision by mode and by area Modes Area of comments No. of total Comments No. and % of RO comments No. and % of RO comments leading to revision FFPR 1 Global 179 118 66% 90 76% Local 102 86 84% 60 70% Subtotal 281 204 73% 150 74% CMPR 1 Global 259 193 75% 142 74% Local 110 89 81% 53 60% Subtotal 369 282 76% 195 69% FFPR 2 Global 429 293 68% 222 76% Local 67 52 78% 26 50% Subtotal 496 345 70% 248 72% CMPR 2 Global 340 186 55% 109 59% Local 108 80 74% 45 56% Subtotal 448 266 59% 154 58% RO: Revision-oriented; FFPR1 = Paper 1, CMPR1 = Paper 2, FFPR 2 = Paper 3, CMPR2 = Paper4 Students’ perspectives on FFPR and CMPR The analysis of survey and interview data show that, although students felt peer comments from both modes helped them a great deal in the revision process, their preference for a certain mode seemed to be related to the advantages and disadvantages of the tools used for communication in FFPR and CMPR. With respect to the first-step peer review procedure, 92% of the students preferred typed comments to handwritten comments for several reasons. First, typing comments directly in their peers’ drafts using Word’s editing features, such as track changes and highlighting, was more efficient than handwriting. As one student noted, “typing is just faster and less energy-exhausting than writing comments on paper. I can type suggestions next to the problematic sentences and all additions and deletions I made would be automatically documented via track changes. That’s convenient.” When writing comments on a separate sheet of paper, by contrast, students would have to keep mentioning which part of the paper they were commenting on, and that could make the review process tedious. Although making marginal comments was possible, most students did not make them because of the limited space on the printouts of their peers’ writing. Additionally, typed comments enhanced the revision process. On computers, students could directly read the suggestions and changes their partners made on their electronic drafts, and the problematic areas were also easily identifiable with highlights. During revision, students could simply accept the changes their peers made with a single click or copy and paste their peers’ specific examples from alteration comments. Handwritten comments, on the other hand, usually did not have this advantage. As one student mentioned, “even though my partner marked problems in printouts with symbols or highlighters, locating the corresponding comments written on a separate sheet of paper was a hassle.” Some students also reported that the handwriting was illegible. However, despite the disadvantages, some students felt that writing comments on a separate sheet of paper gave them another chance to practice writing skills in comparison to typing using Word’s track changes function. As one female student remarked, “I had to use signposting words and keep telling my partner which part of the paper I was referring to.… And I also need to write clearly with proper transitions. It’s not like you can highlight things on electronic papers and type comments next to the problematic sentences or paragraphs.” Another also commented on the advantage of handwriting comments, “It’s good to train [sic] your writing skills. Because you need to think more about the thing you write. With computers, Word would automatically identify the grammar or spelling mistakes Australasian Journal of Educational Technology, 2015, 31(1). 11 in my comments to my partner.” As for the second step peer review procedure, most students were satisfied with the features of OnlineMeeting, such as document sharing and synchronous chat. Ninety-two percent of the students considered the document sharing and archiving function in OnlineMeeting easy to use. Seventy-seven percent of the students felt the chat room function was useful because the system juxtaposed students’ electronic drafts and the chat dialogue side by side. As one student noted, “it was neat that I could see my partner’s electronic paper showing in the same window when I chatted with her online.” Another reason that students were satisfied with OnlineMeeting was that the system automatically archived the chat history, which allowed them to retrieve it when revising subsequent drafts. Although OnlineMeeting received good evaluations in general, 85% of the students preferred face-to-face discussion to online chat. They felt that face-to-face talk was more efficient than online chat because they could easily express thoughts and negotiate meanings. As one student remarked, “we can go more in-depth, offering reasons and explanations when disagreements occurred. Questions can be immediately clarified with ease.” Face-to-face talk offered a greater sense of immediacy and directness. Although online chat was synchronous in nature, everything was still text-based and mediated through typing on screen. This might explain why some chat discussions tended to be superficial and many topics were not followed up. Another student mentioned that “chatting online is more like writing than speaking. Typing on the computer took me more time to formulate sentences than speaking. Sometimes, it’s just difficult to get my ideas across during instant chat. Miscommunication also occurred easily. My partner and I had to constantly make sure we were reading and talking about the same paragraph.” Furthermore, making indexical references such as paragraph or line numbers to direct peers’ attention to certain trouble sources under discussion made online chat less efficient than face-to-face talk during which students could easily finger point to the problematic areas in printouts. Furthermore, the effectiveness of online chat can be easily compromised by several factors such as students’ lack of typing skills, lack of paralinguistic features (e.g., intonation, facial expressions, gestures, etc.), and the length of wait time for others’ responses. During online discussion, it was also difficult for students to regulate their peer’s off-task behaviors. As one student complained, “sometimes when it took a long time for my partner to respond, I would begin to wonder. Was she still focusing on peer review or was she doing something else online?” Another student said, “I don’t like online chat. For the last paper, it was so inefficient. Whatever I typed, I had to wait for a long time before getting a response. It’s time consuming. Maybe it had to do with the person or her way of doing review online. But I just felt that face-to-face talk was more efficient.” Although group makeup may affect the interaction during online chat session, another possible reason for students’ preference for face-to-face discussion over online chat may have to do with the location where discussions were conducted. Unlike in-class face-to-face talk, online discussions were done outside of class without the teacher. Therefore, it was likely that some students would not pay full attention to online discussions. All of these drawbacks could easily affect meaning negotiation and the flow of interaction during online chat sessions. However, despite the drawbacks, the less vocal students (15%) still preferred online chat to face-to-face talk. As one student said, “I’m more of an introvert person. I sometimes felt awkward and even nervous reviewing others’ work face-to-face. When this happened, my mind just went blank. During online chats, I usually didn’t have this problem.” The other student who liked online discussion was the only non-English major in the writing class. As she mentioned, “I felt like I’m an outsider. I only met the other students once a week.… I felt less intimidated when discussing comments online with someone whom I am not familiar with.” This finding corroborates the previous studies that CMPR offers a less threatening environment for less outspoken students (see DiGiovanna & Nagaswami, 2001; Tuzi, 2004). Also of note is that, during the follow-up interviews, 85% of the students suggested combining Word commenting and face-to-face discussion to maximize the benefits of the two step peer review activity. Furthermore, all students felt peer comments helped them the most with revising the global problems such as identifying irrelevant ideas, offering more supporting ideas, and organisation. Many also said they learned about the strengths of their peers’ writing in both modes. Australasian Journal of Educational Technology, 2015, 31(1). 12 Discussion One of the major findings reveals that the students gave a significantly higher proportion of revision-oriented than non-revision-oriented comments overall (69% vs. 31%) in either mode. Among the revision-oriented comments in each of the four writing assignments, 58% to 76% of such comments identified global-area writing problems, compared to 15% to 42% targeting local-area issues. This indicates that both FFPR and CMPR in the study were effective in helping students identify the global-area writing problems, and both modes generated more constructive or revision-oriented comments than complimentary comments. Also, the participants did not refrain from providing revision-oriented comments to help their peers with revision. This finding corroborates a study by Miao et al. (2006) and is more encouraging than several previous studies (e.g., Carson & Nelson, 1996; Guardado & Shi, 2007), which showed peer review to be ineffective among Asian students studying in native English speaking countries, because they tended to give only complimentary comments. This difference in findings might have to do with group solidarity, as Miao et al. (2006) suggest. In Carson and Nelson’s (1996) study, the Asian students in the US were grouped with ESL students from different cultural backgrounds for peer review. However, the students in this study were culturally and linguistically homogeneous and thus may have been more willing to provide revision-oriented comments, which could be possibly face-threatening. Additionally, as many as 69% of the revision-oriented comments were incorporated into revisions in the first three writing assignments, except for the fourth assignment in which still 58% of revision-oriented comments led to revision. The peer comment adoption rate of the four papers is higher than those reported in previous studies. The communication tools used for CMPR in previous research and the composition of peer review groups might account for the differences. For instance, Liu and Sadler’s (2003) participants, who used MOO’s for online chat, needed to exchange electronic drafts via email first, and were not able to see their peers’ drafts and the chat dialogues in the same window on screen. The asynchronous weblog postings in Liou and Peng’s (2009) study lacked a sense of immediacy and timely responses. Furthermore, unlike the dyad grouping in this study, the peer review groups in Liu and Sadler’s (2003) study consisted of four students, which could explain why confusing turn taking and numerous conversation maintenance turns abounded in their study. All these factors could have affected the effectiveness of CMPR and thus led to poor peer-triggered revision rates. The tools used for communication at different stages of peer review also seemed to affect students’ preference for interaction modes. Most felt that typing comments was more efficient than handwriting and that discussing comments face-to-face was more effective than online chat. The students tended to spend more time discussing comments online than via face-to-face talk. The average length of online chat was about 28 minutes, which is almost two times longer than face-to-face discussion. This disparate discussion time indicates that comments via synchronous CMC might take longer to formulate, and a lengthy wait time for responses might be another factor. Although the occurrence and length of off-task talk were similar in both modes, with an average of nearly 3 minutes occurring in about half of all discussions, some groups were more likely to talk off-task. It was observed that while as many as seven occurrences of off-task talk were found in one group’s online discussion, some groups did not talk about anything off-task throughout the entire discussion session. This suggests that in addition to the medium used for communication during peer review, some other factors (e.g., students’ online behaviors and social bonds) might have affected the quality of peer review discussion. However, further studies along this line are needed in order to better understand the factors that might contribute to the effectiveness of online discussion. With respect to students’ subsequent revision, among the revision-oriented comments, FFPR elicited more peer-triggered revision than CMPR (74% and 72% in papers 1 and 3 vs. 69% and 58% in papers 2 and 4). This result implies that students’ preference for certain communication modes at different steps of peer review activities may not necessarily determine how they respond to and incorporate peer comments in their revised drafts in general. Most importantly, all students felt that peer comments from either mode were useful and that they facilitated the revision process because most of their classmates gave specific directions for further revision. Australasian Journal of Educational Technology, 2015, 31(1). 13 Conclusion This study has examined how different peer review modes affect EFL students’ commenting patterns and the degree of peer-triggered revisions. The findings raise issues regarding the balanced use of FFPR and CMPR in writing classes. The participants of this study favoured one mode over the other at different steps of reviewing process, as each mode had its advantages. However, simply replacing traditional FFPR with CMPR may hinder the reviewing process. To increase the effectiveness and efficiency of peer review, teachers should incorporate CMPR in ways that draw on the strengths of the traditional mode and take into consideration various factors, such as students’ familiarity with new software/program, assessments, and impact on students’ reviewing and revision process. For CMPR to yield positive outcomes, elaborate training sessions are needed to familiarize students with new peer review software/program and this may take a large portion of class time. Moreover, since computer-mediated activities may change group dynamics, require ICT skills, and entail different levels of participation on the part of students, composition teachers would also need to adjust assessment procedures to ensure that highly computer anxious students are not disadvantaged. As several researchers have noted, “altering the balance of student activity within a course significantly impacts on many, if not all other aspects of the learning transaction” (Twining, 1999, p. 11; Ware & Warschauer, 2006) Writing teachers need to recognize this impact from the outset and accommodate individual students’ preferred mode of interaction and learning style if the effectiveness of peer review is to be maximized. Furthermore, for a better review outcome, teachers should adopt ways of monitoring students’ computer-mediated interaction to reduce possible off-task discussions and to avoid students engaging in irrelevant online activities while doing CMPR. As with other classroom-based research, this study was limited due to the small number of participants. Although the findings are contextualized in a specific setting, a sample of one writing class does not allow generalisation to other EFL contexts. Another limitation has to do with a lack of specific pairing up procedures. The peer review dyads in this study were formed randomly. Because the analysis of discussion transcript and online chat logs shows that some groups had more off-task talk than the others, future investigations may want to adopt a specific pairing up procedure to examine how different group dynamics might affect the quality of peer comments and discussion in different modes. Also, given the nature and constraints of different modes and software, future research could explore various types of ICT programs/software (e.g. voice or video chat) during different drafting processes, in or outside of the class to shed more light on how technology shapes students’ commenting and revision behaviors. Finally, since none of the previous research that compared FFPR and CMPR controlled contextual factors (e.g., order effects of the writing tasks, types of assignments, students’ preferred learning styles and individual differences), studies along this line with an experimental design would shed more light on the effectiveness of blended peer review activities. Acknowledgements The author would like to thank anonymous reviewers and the editors for their helpful suggestions. This research was supported by the Ministry of Science and Technology in Taiwan (Project no.98-2410-H-133-010). References Braine, G. (2001). A study of English as a foreign language (EFL) writers on a local-area network (LAN) and in traditional classes. Computers and Composition, 18, 275-292. doi:10.1016/S8755-4615(01)00056-1 Breuch, L. K. (2004). Virtual peer review: Teaching and learning about writing in online environments. Albay, NY: State University of New York Press. Carson, J. G., & Nelson, G. L. (1996). Chinese students’ perceptions of ESL peer response group interaction. Journal of Second Language Writing, 5(1), 1-19. doi:10.1016/S1060-3743(96)90012-0 Chang, C. F. (2012). Peer review via three modes in an EFL writing course. Computers and Composition, 29, 63-78. doi:10.1016/j.compcom.2012.01.001 Australasian Journal of Educational Technology, 2015, 31(1). 14 DiGiovanni, E., & Nagaswami, G. (2001). Online peer review: An alternative to face-to-face? ELT Journal, 55(3), 263-271. doi:10.1093/elt/55.3.263 Grabe, W., & Kaplan, R. B. (1996). Theory and practice of writing. New York, NY: Longman. Guardado, M., & Shi, L. (2007). ESL students’ experiences of online peer feedback. Computers and Composition, 24, 443-461. doi:10.1016/j.compcom.2007.03.002 Herring, S. C. (2002). Computer-mediated communication on the internet. In B. Cronin (Ed.), Annual review of information and science technology (pp. 109-168). Medford, NJ: Information Today, Inc. doi:10.1002/aris.1440360104 Hewett, B. (2000). Characteristics of interactive oral and computer-mediated peer group talk and its influence on revision. Computers and Composition, 17, 265-288. doi:10.1016/S8755-4615(00)00035-9 Ho, M. C., & Savignon, S. J. (2007). Face-to-face and computer-mediated peer review in EFL writing. CALICO Journal, 24(2), 269-290. Honeycutt, L. (2001). Comparing email and synchronous conference in online peer response. Written Communication, 18(1), 26-60. doi:10.1177/0741088301018001002 Hyland, K., & Hyland, F. (2006). Contexts and issues in feedback on L2 writing. In K. Hyland & F. Hyland (Eds.), Feedback in second language writing: Contexts and issues (pp. 1-19). New York, NY: Cambridge University Press. doi:10.1017/CBO9781139524742 Jones, R. H., Garralda, A., Li, D. C. S., & Lock, G. (2006). International dynamics in on-line and face-to-face peer-tutoring sessions for second language writers. Journal of Second Language Writing, 15(1), 1-23. doi:10.1016/j.jslw.2005.12.001 Lee, J. F., & VanPatten, B. (2003). Making communicative language teaching happen. New York, NY: McGraw-Hill. Liang, M. Y. (2010). Using synchronous online peer response groups in EFL writing: Revision-related discourse. Language Learning & Technology, 14(1), 45-64. Liou, H. C., & Peng, Z. Y. (2009). Training effects on computer-mediated peer review. System, 37(3), 514-525. doi:10.1016/j.system.2009.01.005 Liu, J., & Sadler, R. W. (2003). The effect and affect of peer review in electronic versus traditional modes on L2 writing. Journal of English for Academic Purpose, 2(3), 193-227. Miao, Y., Badger, R., & Zhen, Y. (2006). A comparative study of peer and teacher feedback in a Chinese EFL writing class. Journal of Second Language Writing, 15(3), 179-200. Min, H. T. (2006). The effects of trained peer review on EFL students’ revision types and writing quality. Journal of Second Language Writing, 15(2), 118-141. doi:10.1016/j.jslw.2006.09.004 Min, H. T. (2008). Reviewer stances and writing perceptions in EFL peer review. English for Specific Purposes, 27(3), 285-305. doi:10.1016/j.esp.2008.02.002 Moradi, M. R., & Karimpour, Z. (2012). The effect of online peer feedback on the academic writing ability of Iranian EFL learners. International Education Studies, 5(2), 113-117. doi:10.5539/ies.v5n2p113 Prior, P. (2005). Moving multimodality beyond the binaries: A response to Gunther Kress’s “Gains and Losses.” Computers and Composition, 22, 23-30. doi:10.1016/j.compcom.2004.12.007 Rollinson, P. (2005). Using peer feedback in the ESL writing class. ELT Journal, 5(1), 23-30. doi:10.1093/elt/cci003 Song, W., & Usaha, S. (2009). How EFL university students use electronic peer response into revisions. Suranaree Journal of Science and Technology, 16(3), 263-275. Tuzi, F. (2004). The impact of e-feedback on the revisions of L2 writers in an academic writing course. Computers and Composition, 21, 217-235. doi:10.1016/j.compcom.2004.02.003 Twining, P. (1999). Learning matters: Adjusting the media mix of academic advantage. ALT-J: Research in Learning Technology, 7(1), 4-11. doi:10.1080/0968776990070102 Wang, M. J. (2004). A study on online peer feedback. English Teaching and Learning, 29(2), 63-77. Ware, P., & Warschauer, M. (2006). Electronic feedback and second language writing. In K. Hyland, & F. Hyland (Eds.), Feedback in second language writing: Contexts and issues (pp. 105-122). New York: Cambridge University. doi:10.1017/CBO9781139524742.008 Xu, Y., (2007). Re-examining the effects and affects of electronic peer reviews in a first-year composition class. The Reading Matrix, 7(2), 1-21. Australasian Journal of Educational Technology, 2015, 31(1). 15 Appendix A The analytical scheme for classifying peer comments Examples taken from students’ comments in the study. Area Global Local Nature Revision-oriented Non-revision- Oriented Revision-oriented Non-revision- oriented Type Evaluation I think your first main idea is a bit out of logic [sic]. The ideas connect well because of the transition “despite….” The sentence marked with * sounds strange. Language use is really great because you use different vocabulary to explain the same thing. Clarification What’s the terrible feeling in paragraph 3? Can you describe more about it. (No example for this category) What do you mean by “original asset”? Do you mean “personal property?) (No example for this category) Suggestion Maybe you can tell a story about yourself. For example, you may write your own story about your learning and how the pressure influences you. (No example for this category) Maybe you can add a transition, “in addition” to connect the ideas. (No example of this category) Alteration Maybe you can say that “In addition, because my sister studied abroad, I experience…. (No example for this category) Change “they didn’t teach me right concept” to “they didn’t teach me the right concept.” (No example for this category) Response There is no topic sentence here. There is a thesis statement in the introduction. There is no transition to connect ideas in the previous paragraph. Yes, the writer used some pronouns to avoid repetition. Other I don’t think you need to talk about studying abroad here, unless you mentioned how to enhance one’s ability in the thesis statement. (No example for this category) I think this sentence needs to be changed but I don’t know how to do it. I’m not sure if this grammar is right or not. (No example for this category) Corresponding author: Mei-ching Ho, mch0532@gmail.com Australasian Journal of Educational Technology © 2015. Please cite as: Ho, M. (2015). The effects of face-to-face and computer-mediated peer review on EFL writers’ comments and revisions. Australasian Journal of Educational Technology, 31(1), 1-15.