INVESTIGATING CHINESE EFL STUDENTS’ PERCEPTIONS OF WRITING PROMPTS OF DIFFERENT AMOUNTS OF INFORMATION SU YOU Investigating Chinese EFL Students’ Perceptions of Writing Prompts of Different Amounts of Information INVESTIGATING CHINESE EFL STUDENTS’ PERCEPTIONS OF WRITING PROMPTS OF DIFFERENT AMOUNTS OF INFORMATION Su You School of Humanities, Beijing University of Posts and Telecommunications E-mail: suyou@bupt.edu.cn APA Citation: Su, Y. (2015). Investigating Chinese EFL students’ perceptions of writing prompts of different amounts of information. English Review: Journal of English Education, 3(2), 230-238 Received: 11-04-2015 Accepted: 26-04-2015 Published: 01-06-2015 Abstract: This study aims to explore how Chinese EFL students perceive the advantages and disadvantages of prompts providing different amount of information, namely prompt with more information and prompt with less information. Both qualitative and quantitative data were collected through questionnaire survey and semi-structured interview. Research results indicate that: 1) Respondents hold a mixed attitude towards the prompt effect on their task accomplishment. 2) Students believe that prompt type can affect their expression in writing; 3) Students generally agree that prompt with more information facilitate their writing in terms of content and organization; 4) Students’ preference for the prompt type differs across different English proficiency level. Keywords: prompt effect, English writing, testing writing, perceptions INTRODUCTION Since the role of writing in second language education increases, there is also a greater demand to test writing in a valid and reliable way. In writing assessment, writing prompts are designed with a view to providing an adequate sample of written discourse to make appropriate and reliable assessments of the linguistics skill of the candidate (O’Loughlin & Wiggleworth, 2007). However, there is also the risk of “construct-irrelevant variance” (Messick, 1996) or what Jenning, Fox, Graves, and Shohamy (1992:456) have called a “prompt effect”. In other words, in writing performance assessment, prompts are systematically varied for different students, raising the possibility of a prompt effect and thus affecting the validity, reliability and fairness of these tests. However, the question of the extent to which the specific task prompt affects writing in a foreign language is “a vexed one” (O’Loughlin & Wigglesworth, 2007:380). Weigle (2011:69) also points out that it is still not clear in terms of which specific differences in writing prompts affect examinee’s performance and in what ways. Testing specialists (Bachman and Palmer, 1996; Alderson et al, 2000; Weigle, 2011) point out the usefulness of task specifications in terms of test objectives and test takers’ interpretation. Meanwhile, some scholars (Li, 2001; Wu, 2008; Gu & Gao, 2007) worried that too much amount of information in Chinese provided in the prompt might result in test takers’ direct translation of the prompt and it might deduce the validity of the writing test. This raises the question that whether the prompt with or without an outline will affect test takers’ writing performance and how prompts with different amount of information affect test takers’ ENGLISH REVIEW: Journal of English Education ISSN 2301-7554 Vol. 3, Issue 2, June 2015 http://journal.uniku.ac.id/index.php/ERJEE performance. Besides, one of the objectives in developing assessment prompts is to ensure comparability across different administrations. In order to achieve this, it is crucial that we understand better about the effects of prompts, how test takers approach them, and what affect the difficulty of a prompt. Whether the changes to the task specification affect test takers’ performance, has received some amount of attention. According to Hinkel (2002), prompt wording affect test takers’ writing performance since they insert language from the prompt into their essays. In Brossell and Ash’s (1984) study, they examined the possible effect of the wording of essay topics by looking at whether phrasing the task as a question or as a statement make a difference in test takers’ writing performance. They also analyzed whether it matters when a topic is addressed to the writer in personal way (“you”) or is addressed in impersonal or neutral terms. This study provided no evidence to support the claim that small changes in the wording of essay test topics affect test takers’ outcomes. Studies have also dealt with the amount of information given in prompts and the extent to which task difficulty and test takers’ performance will be affected by it (Kroll and Reid,1994; Brossell, 1983; O’Loughlin and Wigglesworth, 2007). O’Loughlin and Wigglesworth (2007) investigated the extent to which the difficulty of IELTS Academic Writing task 1 is affected by the amount of information provided to the candidate and the extent to which the difficulty of the task is affected by the presentation of the information to the candidate. Analysis revealed that there were no substantial differences in difficulty between the tasks and it appears that tasks providing less information actually elicit more complex language. Brossell (1983) discovered that the information load apparently can affect a test taker’s tendency to begin writing purposefully by helping or hindering the focusing and organizing of a test taker’s thoughts. Another investigation conducted by Oh and Walker (2007) evaluated whether new Scholastic Assessment Test’s essay prompt type (either a simple one-line prompt or a prompt including a short passage) affects test-takers’ essay production. Research findings indicated that the one-line prompt and the prompt with a passage providing more information have a similar impact on the test-takers’ essay performance. Other studies have revealed that test takers do take the generality and specificity of prompts into their consideration when it comes to decide which one to choose (Polio & Glew, 1996; Powers & Fowles, 1998). Chiste and O’shea (1988) found that ESL writers preferred the shorter prompt but ESL writers studied did not perform significantly better on short questions because such questions may limit writers’ range of topic choices and offer less insight into an essay’s development and structure. Thus, it is claimed that prompt should be phrased as succinctly as possible and prompts should be comparable in length (Chiste and O’shea, 1988). Besides, Brossell (1983) claims that a medium level of specification and information load is the best choice for prompt design as this facilitates examinees focus without overloading them with information or narrowing their thoughts. Based on the review of theoretical concerns on writing and writing assessment, one can see that prompts SU YOU Investigating Chinese EFL Students’ Perceptions of Writing Prompts of Different Amounts of Information can be systematically varied for different test takers, raising the possibility of a prompt effect and thus affecting the validity, reliability and fairness of writing performance tests. However, research on the designing of writing prompts, specifically, the prompt effect, has not been well addressed. As prompt with necessary content and structure support is the format frequently used in the writing assessment in China(Gu, Yang & Feng, 2009), there is the necessity to investigate Chinese EFL students’ perceptions of different writing prompts. Considering the issues and problems mentioned above, this study aims to explore how test takers may perceive the relative advantages and disadvantage of writing prompts with different amount of information. To achieve the research purpose, the research questions of this study are: 1) What are students’ perceptions of the advantages and disadvantages of the prompts that provides more information (specific-points prompts)? 2) What are students’ perceptions of the advantages and disadvantages of the prompts that provides less information (basic-points prompts)? METHOD Participants A total of 102 Chinese EFL students took part in this study. Among the 102 participant, 61 of them were male while 41 were female. They are all non-native speakers of English and speak Chinese as their mother tongue. The average age of these participants is 17. All the participants started to learn English when they entered junior high school and they have just started the third year’s study of the senior high school at the time when they participated in the present study. As third-year senior high school students, they were required to write coherent and well-structured short essays to express their opinions and attitudes (NECS: 2001). To prepare them for College Entrance Examination, prompt with necessary content and structure support is the format frequently used by the teachers. Data Collection Method To serve the purpose of the present study, both qualitative and quantitative data were collected through questionnaire survey and semi-structured interviews. The questionnaire was designed in the form of Likert Scale, which is adapted from Xu (2006) whose research was on identifying and controlling writing task difficulty factors in English tests. The questionnaire intends to extract students’ perceptions in terms of task fulfillment, vocabulary use, structure organization, cohesion, expression and grammar use. Participants were asked to number from 5 to 1 to indicate the degree they agreed with each statement (5=strongly agree, 4=mostly agree, 3=not sure, 2=mostly disagree 1=strongly disagree). For the accuracy of responses, the questionnaire was designed in Chinese. The reliability of the questionnaire was reliable with a Cronbach Alpha of .8127. Based on the results of the questionnaires, it was intended to use interviews to gain deeper insight into participants’ conceptions of some specific issues related to the advantages and disadvantages of the amount of information provided in the prompt. Interviews with the students were conducted after the questionnaire survey. Six students across three proficiency levels, basic, intermediate, and advanced, were chosen as the participants for the interview. The average time length of the interviews was about 15 minutes. All interviews were recorded and later transcribed by the researcher. ENGLISH REVIEW: Journal of English Education ISSN 2301-7554 Vol. 3, Issue 2, June 2015 http://journal.uniku.ac.id/index.php/ERJEE RESULTS AND DISCUSSION Questionnaire responses on the Liker scale were presented after data processing with SPSS (16.0). One thing worth noting here is that we combine “agree” and “strongly agree” on the scale in the following discussion. The specific areas under investigation were: language use, writing process, and writing quality. As for the writing quality, it includes content, structure and organization, cohesion, expression and word choice, and grammar. Perceptions of the specific-points prompt Language use Data from the questionnaire survey showed that only 35% of the students felt that their writing on the specific-points prompt can reflect their English writing ability objectively (No. 1). For a closer examination of the statistics on language use, one can see that less than 20% of respondents felt that prompts with specific points could make their expression more native-like (No. 6) and around 40% of them believed that this kind of prompt hindered the variety of vocabulary in their writing (No.7). However, around half of the respondents to the questionnaire felt that the specific-points prompt enabled them to write more (No.3). Table 1 presents the specifics of statistics including the frequency, mean score, and standard error. Table 1. Students’ perceptions of specific-points prompt: language use I think the prompt with more information: Frequency (%) N=102 SD D N A SA M S.D. 1 reflects my English writing ability objectively. 6.0 13.0 46.0 24.0 11.0 3.21 1.01 3 enables me to write more. 5.2 16.7 31.3 36.5 10.4 3.30 1.04 6 makes my expression more native-like. 4.0 28.7 47.5 15.8 4.0 2.87 0.87 7 Hinders the variety in vocabulary. 7.9 24.8 26.7 25.7 14.9 3.15 1.19 Note: SA = strongly agree; A = agree; N = not sure; D = disagree; SD = strongly disagree Writing process More than half of the respondents to the questionnaire felt that they were more familiar with the prompt with specific points and had inherent routines to complete the task (No.5). For a closer examination of the statistics on the writing process, one can see that 75% of the respondents agreed that specific-points prompt made them tend to directly translate the Chinese prompts into English (No.8). This echoes some scholars’ (Li, 2001; Wu, 2008; Gu & Gao, 2007) assumption that too much amount of information in Chinese provided in the prompt might result in students’ direct translation of the prompt and it might deduce the validity of the writing test. Besides, more than half of the respondent felt that the prompt with more information restricted their thoughts and hindered the performance of their wiring ability (No. 4). Additionally, we are surprised to find that there were still 44% of the respondents felt that the specific-points prompt actually restricted their writing in terms of the content (No.9). The interview indicated that students thought sometimes they found it hard to cover all those information provided in the prompts while some others held that they could nor write other things that they would really want to write if the prompt gave too much information. This could be the reason why many students still felt this kind of prompt challenging (No.2). Table 2 presents the specifics of statistics. SU YOU Investigating Chinese EFL Students’ Perceptions of Writing Prompts of Different Amounts of Information Table 2 Students’ perceptions of specific-points prompt: writing process I think the prompt with more information: Frequency (%) N=102 SD D N A SA M S.D. 2 is not challenging, decreasing my interest in writing. 18.8 38.6 30.7 8.9 3.0 2.39 0.99 4 restricts my thoughts, thus hinders the performance of my writing ability. 8.0 21.0 12.0 40.0 19.0 3.41 1.24 5 makes me more familiar with this mode of writing and I have inherent routines to complete the task. 2.0 17.8 23.8 48.5 7.9 3.43 0.94 8 makes me tend to directly translate the Chinese prompts. 5.0 12.0 8.0 51.0 24.0 3.77 1.10 9 restricts my writing in terms of the content. 5.0 26.0 25.0 32.0 12.0 3.20 1.11 Writing quality As for writing quality, the investigation focuses on students’ perceptions of the advantages and disadvantages of specific-points prompt from the perspectives of organization, coherence, and grammar. Table 3 presents the specifics of statistics. Table 3. Students’ perceptions of specific-points prompt: writing quality I think the prompt with more information: Frequency (%) N=102 SD D N A SA M S.D. 10 helps me develop the overall structure. 2.0 6.9 17.8 62.4 10.9 3.73 0.82 11 enables me to pay more attention to the transition between sentences. 1.0 18.8 33.7 37.6 7.9 3.33 0.91 12 makes the writing structure more complete. 4.0 17.0 24.0 47.0 8.0 3.38 0.99 13 makes my writing more coherent and logical. 3.0 17.0 43.0 31.0 6.0 3.20 0.90 14 promotes the accuracy of grammar. 5.0 24.0 46.0 22.0 3.0 2.94 0.89 15 reminds me of constant checking of the accuracy of grammar while writing. 2.0 32.3 42.4 21.2 2.0 2.89 0.83 16 promotes the variety of grammatical structures. 5.0 26.7 49.5 16.8 2.0 2.84 0.83 17 promotes my accuracy of tense. 2.0 21.8 42.6 29.7 4.0 3.12 0.86 The questionnaire survey indicated that another advantage of the specific-points prompt is that it facilitates students in writing in terms of organization and structure. Around 80% of the respondents believed that prompt with more information helped them develop the overall structure (No.10) and more than half of the respondents agreed that it made their writing structure more complete (No.12). This result echoes with the findings of the empirical study conducted by Xu (2006) which indicated that providing stimulus may help students with idea generation. In addition, 45% of the students felt that specific-points prompt enabled them to pay more attention to the transition between sentences and 37% of them believed it made their writing structure more complete. However, one can see that respondents hold very neutral attitude towards the advantages and disadvantages of prompt with more information from the perspective of grammar use (No.14, No.15, No.16, and No.17). Perceptions of the basic-points prompt The specific areas under investigation were: language use, writing process, and writing quality. As for the writing quality, it includes content, structure and organization, cohesion, expression and word choice, and grammar. Language use Data from the questionnaire survey indicated that more than 63% of the respondents believed that their writing on the basic-points prompt reflected their English writing ability more objectively ENGLISH REVIEW: Journal of English Education ISSN 2301-7554 Vol. 3, Issue 2, June 2015 http://journal.uniku.ac.id/index.php/ERJEE (No. 1). This is a sharp contrast with respondents’ perceptions of specific-points prompts (35%). In other words, in the test takers’ eyes, prompt with less information is a better way to measure their English writing ability. Besides, 34% of the students thought that prompt with less information enabled them to write more (No.3). Table 4. Students’ perceptions of basic-points prompt: language use I think the prompt with more information: Frequency (%) N=102 SD D N A SA M S.D. 1 reflects my English writing ability more objectively 5.0 5.0 26.7 37.6 25.7 3.74 1.06 3 enables me to write more 5.0 25.0 36.0 28.0 6.0 3.05 0.99 6 makes my expression more native-like 5.0 21.0 53.0 14.0 7.0 2.97 0.92 7 promotes the variety in vocabulary 5.0 6.0 31.0 50.0 8.0 3.50 0.92 For a closer examination of the statistics on language use, we can see that more than half of the respondents felt that the basic-points prompt promoted the variety of word choice in their writing (No.7). This might because it is possible for students to avoid direct translation of the Chinese prompt. Follow-up interviews revealed that students usually found it hard to choose the exact word to translate the prompt if there were too much information, but it would be easier for them to try other ways of expression or other similar words to directly express their opinions when dealing with the prompt with less information. As for the expression, it seemed that students held a neutral view on whether basic-points prompt made their expression more native-like (No.6). Table 4 above presents the specifics. Writing process Another advantage of the basic-points prompt students considered was that it broadened their thought and thus promoted the performance of their writing (No.4, 69.3%). Besides, around half of the respondents thought that basic-points prompt is more challenging and increases their interest in writing (No. 2). This might be the reason why many students believed that their performance on the prompt with less information reflected their writing ability more objectively. However, there were 59% of responses showed that the missing of outline in the prompt made it difficult to complete the writing task (No.5). This is different from students’ perception of the prompt with more information which was believed to be easier for students in terms of task fulfillment. Table 5 below presents the specifics. Table 5 Students’ perceptions of basic-points prompt: writing process I think the prompt with more information: Frequency (%) N=102 SD D N A SA M S.D. 2 more challenging, increasing my interest in writing 5.9 6.9 36.6 33.7 16.8 3.49 1.05 4 broadens my thought, thus promotes my writing 4.0 4.0 22.8 48.5 20.8 3.78 0.96 5 makes it difficult to complete the task since the outline is missing 8.0 21.0 12.0 53.0 6.0 2.98 1.10 8 makes it possible for me to avoid direct translation of the Chinese prompts 4.0 12.9 14.9 49.5 18.8 3.66 1.05 9 makes my writing more free and open in terms of content 4.0 2.0 16.0 58.0 20.0 3.88 0.89 SU YOU Investigating Chinese EFL Students’ Perceptions of Writing Prompts of Different Amounts of Information By examining the statistics closely, we can see another obvious advantage of basic-points prompt was that it made students’ writing more free and open in terms of content (No.9, 78%). Besides, most students felt that it is possible for them to avoid direct translation of the Chinese prompt (No.8). This result is consistent with participants’ perception of specific-points prompt where 44% of the respondents felt that prompt with more information restricted their writing. The interview also revealed that students, especially those of high English proficiency, felt that there was more flexibility in terms of what to write instead of being limited by the information provided in the prompt. Table 18 below presents the specifics. Writing quality As for the quality of students’ writings, respondents to the questionnaire felt that it was difficult for them to organize the structure and make the writing coherent and logical when responding to the basic-points prompt. Only 28% of the students felt that basic-points prompt helped them develop the overall structure and 16% of them believed that this kind of prompt made their writing structure more complete. Besides, around 30% of the students thought that basic-points prompt enabled them to pay more attention to the transition between sentences. However, we can see that students held a neutral attitude towards the advantages and disadvantages of basic-points prompts from the perspective of grammar use (No.14, No.15, No.16, and No.17). Table 6. Students’ perceptions of basic-points prompt: writing quality I think the prompt with more information: Frequency (%) N=102 SD D N A SA M S.D. 10 helps me develop the overall structure. 3.0 20.0 49.0 20.0 8.0 3.10 0.92 11 enables me to pay more attention to the transition between sentences. 2.0 19.8 46.5 23.8 7.9 3.16 0.90 12 makes the writing structure more complete. 3.0 21.0 60.0 13.0 3.0 2.92 0.76 13 makes my writing more coherent and logical. 3.0 23.8 51.5 16.8 5.0 2.97 0.85 14 promotes the accuracy of grammar . 2.0 28.0 48.0 18.0 4.0 2.94 0.84 15 reminds me of constant checking of the accuracy of grammar while writing.. 3.0 20.8 46.5 22.8 6.9 3.10 0.91 16 promotes the variety of grammatical structures. 3.0 20.8 41.6 27.7 6.9 3.15 0.93 17 promotes my accuracy of tense. 3.0 22.8 53.5 15.8 5.0 2.97 0.84 The prompt type that students preferred Participants were also asked which prompt type they prefer. Results indicated that perception of students differed across different English proficiency level as they self-rated. In the interview, the interviewee of advanced English level expressed that the prompt with less information gave them more freedom to write and they felt that the choice of words was not limited so that the accuracy of their writing was ensured. As for the reason why they preferred the prompt with less information, some students also thought that it gave them opportunity to write something of themselves instead of writing things that shared high similarity with other students. When asked why they preferred the prompt ENGLISH REVIEW: Journal of English Education ISSN 2301-7554 Vol. 3, Issue 2, June 2015 http://journal.uniku.ac.id/index.php/ERJEE with more information, some students responded that they didn’t have to generate ideas and this saved much time in a time-controlled testing situation while others thought that more information provided in the prompt could enable them to write long sentences and have a better control over the structure of their essay. The reason for the difference is that student of lower proficiency have more demand for content and outline support than those of higher proficiency who don’t want to get restricted by the information provided in the prompt. This is also in line with previous studies which revealed there is interaction between prompt preference and proficiency level (Chiste & O’shea, 1988; Jennings, et al., 1999). This indicated that students’ language ability can be an important factor that determines their perceptions of the advantages and disadvantages of the two different prompt types. CONCLUSION This study investigated into participants’ perceptions of the advantages and disadvantages of writing prompt with different amount of information. The areas explored include students’ general opinions on task accomplishment, and factors considered important in writing such as content, structure and organization, cohesion, expression and word choice, grammar, and mechanics. Research results indicated that respondents held a mixed attitude towards the prompt effect on their task accomplishment. Students also felt that prompt type can affect their expression in writing. They agree that translation effect may appear when they are taking the prompt with too much information. Besides, students agreed that prompt with more information facilitates their writing in terms of content and organization. Most students believed that the prompt providing structure and content support made their writing more free and open in terms of content. It also reveals that students’ preference for the prompt type differs across different English proficiency level. Students of advanced level preferred the prompt with less information while many students of the basic level would like to write on the prompt with more information. In terms of future research, quantitative and qualitative analysis of students writing performance on different prompts could be conducted to further examine the prompt effect. ACKNOWLEDGEMENTS This paper is supported by the Fundamental Research Funds for the Central Universities (2014RC0703) and 2014 Research Projects for National College Foreign Language Teaching (S2014175). REFERENCES Alderson, J. C., Clapham, C., & Wall, D. (2000). Language test construction and evaluation. Beijing: Foreign Language Teaching and Research Press. Bachman, L., & Palmer, A. (1996). Language testing in practice. Oxford: Oxford University Press. Brossell, G. (1983). Rhetorical specification in essay examination topics. College English, 45(2), 165-173 Brossell, G., & Ash, B. H. (1984). An experiment with the wording of essay topics. College Composition and Communication, 35(4), 423-425 Chiste, K. B., & O'Shea, J. (1988). Patterns of question selection and writing performance of ESL students. Teachers of English to Speakers of Other Languages, 22(4). Gu, X., & Gao, X. (2007). An investigation into the writing tasks of NMET 2007. China Examinations, (12), 28-36. SU YOU Investigating Chinese EFL Students’ Perceptions of Writing Prompts of Different Amounts of Information Gu, X., Yang, R., & Feng, N. (2010). A study on the quality of the writing tasks of NMET. Educational Measurement and Evaluation , (12), 47-50. Hinkel, E. (2002). Second language writer’s text: Linguistic and rhetorical features. Mahwah, NJ: Lawrence Erlbaum. Jennings, M., Fox, J., & Graves, B. (1999). The test-takers' choice: an investigation of the effect of topic on language-test performance. Language Testing (16), 456-462 Kroll, B., & Reid, J. (1994). Guidelines for designing writing prompts: clarifications, caveats and cautions. Journal of Second Language Writing. 3(3), 231-255. Li, X. (2001). The science and art of language testing. Changsha: Hunan Education Press. Ministry of Education. (2001). National English curriculum standards. Beijing: Beijing Normal University Press. Messick, S. (1996). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23. Oh, H., & Walker, M. E. (2007). The effects of essay placement and prompt type on performance on the new SAT. New York: The College Board. O’Loughlin, K., & Wigglesworth, G. (2007). Investigating task design in academic writing prompts. In Taylor, L. (Ed.), IELTS collected papers: research in speaking and writing assessment. 379-418. Cambridge: Cambridge University Press. Polio, C., & Glew, M. (1996). ESL writing assessment prompts: How students choose. Journal of Second Language Writing, 5(1), 35-49 Powers, D. E., & Fowles, M. E. (1998). Test takers' judgment about GRE writing test prompts (RR 98-36). NJ: Princeton: ETS. Qi, L., (2004). A study on the washback of NMET. Foreign Language Teaching and Research, 36(5), 357-363. Weigle, S. C. (2011). Assessing writing. Beijing: Foreign Language Teaching and Research Press. Wu，Z. (2008). Theory and practice of English language testing. Beijing: Foreign Language Teaching and Research Press. Xu, Y. (2006). Identifying and Controlling writing task difficulty factors in English Tests. Unpublished PhD dissertation, Shanghai International Studies University.