LANGUAGE CIRCLE: Journal of Language and Literature 14(1) October 2019 p-ISSN 1858-0165 Available online at http://journal.unnes.ac.id e-ISSN 2460-853X 39 The Question of Choice in an Achievement Test: A Study on the Sudanese Case Mahmoud Abdelrazig Mahmoud Barakat Faculty of Education, University of Khartoum & Omdurman Islamic University, Sudan. E-mail: barakatmahmoud99@yahoo.com Abstract Achievement tests administered at national level play a significant role in the lives of test-takers as well as the whole society. This paper aims to investigate the effect of giving students a choice between two optional questions on their overall performance in a high-stake achievement test for university admission. It is hypothesized that questions targeting writing-based productive skills and language system necessitate display of abilities which are different from fact-based questions designed around story content. The two items are assumed to reflect different constructs that require different criteria of assessment. Consequently, the student’s overall score is affected by the item they choose to answer, which might not be reflective of their real language abilities. An open-ended interview was carried out with ten teachers working with grade 3 students in model secondary schools to investigate the nature of the two types of test items and their impact on the student’s performance. The data has proved that giving choice in an achievement test generates different performances that are assessed differently. It is recommended that in order to address the question of fairness, it is important to clearly define and balance the construct of the items that affect the student’s choice and performance. Keywords: achievement test, assessment, choice, fairness, performance. INTRODUCTION English language is one of the basic seven subjects that determine the student's admission to faculty. A student is required to pass the English language examination in the Sudan School Certificate Examination SSCE and should obtain a high grade if they are aiming to go to top colleges. However, in the case we are studying now the students are to choose between two sections. The first is a language section that contains questions targeting writing a letter, doing a summary, sentence transformations and a gap-filling question on prepositions. The other section includes supplementary reading materials which require the students to respond to wh- questions, MCQs, a matching table and true- or-false questions about two simplified stories. Henceforth, the former will be referred to as section (1), the latter section (2). Each of the two sections is given 25 marks. This paper will try to investigate the nature of the items in each of the two sections and whether they result in different types of performances. It is assumed that the students' overall grades will be influenced by the section they choose to answer. Achievement tests conducted by the end of a decisive learning stage may be designed around syllabus content or syllabus objectives. Although each of the two approaches has its advantages and drawbacks (Hughes 1992, pp. 11-12), they may be related in some way. According to Harmer (2010: 166), achievement tests "include a variety of test types and measure students' abilities in all four skills, as well as their knowledge of grammar and vocabulary." By definition, achievement tests assume a well defined procedure that allows measurement of performance. When writing specifications for test design, items key features and language characteristics need to be clearly described LANGUAGE CIRCLE: Journal of Language and Literature, 14(1) October 2019 40 (Fulcher 2007, pp. 69-70). This means that the nature of the tasks or test items, whether derived from the same learning materials or based on the objectives, should reflect exact abilities which the students are required to display. If this criterion is not maintained, it can be claimed that, problems of fairness and test validity are in danger. Weir (2005, p. 12) states that "the focus in validation is …primarily on the examination score or grade as a reliable measure of a trait or construct." Therefore, indirect test items, for example on prepositions and grammar fail to maintain content validity if they are equated with measuring students' ability to write a composition. They obviously neglect other essential features of the writing skill and language knowledge (Brown 2003, pp. 23-24; Bachman et al 1996, p. 68). On the other hand, neglecting discrete point test items in achievement tests, or making them optional or alternative to writing-based productive skills, will bring about negative washback (Coombe et al 2013, p. 46). Defining ability should, therefore, consider the language domain and resulting decision to be made in high-stake tests (Bachman et al 2008, p. 465). Fairness is about giving the same opportunity to the test-takers so that every individual can demonstrate the underlying ability (Lynch 2001, p. 232). To ensure fairness, "students should know exactly how they are to be evaluated" (Siddiqui 2009, p. 18); and how the evaluation of open-ended items differs from close-ended items and affects the student's overall grade (Hubbard 2000 et al., p. 258). METHODOLOGY The study uses the interview to approach the problem of choice in achievement tests. Ten teachers working in model secondary schools were selected according to the criterion that they should have experienced teaching the optional Supplementary Readers Programme SRP to grade 3 students. The interview questions were designed to help deduce relevant ideas about the nature of the two sections of the English examination of the SSCE, the effect of the students' choices on their overall performances and the impact of having the two optional sections on the students' preferences. The interview questions and the sample size were essentially meant to gather data of qualitative nature. FINDINGS AND DISCUSSION It was noticeable from the teachers' accounts that their students displayed great interest in section (2) of the examination paper of the English language in the SSCE. It is important to draw the attention to the fact that these students represented only the model schools included in the study sample. Although the SRP is an optional component in the syllabus and the examination paper, the teachers of the sample stressed its rewarding advantage in terms of marks when compared with Section (1). Nine out of ten (90%) believed that doing Section (2) is a big boost to the students' overall grades in the English language. One teacher reported that all her students in the school, except four or five, chose to do Section (2) in the school tests and she expected that all of them would choose to answer Section (2) in the SSCE. Another reported about three quarters of the students chose Section (2) in the monthly tests and he also expected that the number would increase in the SSCE, because those who answered Section (2) in the State Mock Examination (a large-scale test administered in Khartoum State just about two months before the SSCE, and when almost above 90% of the syllabus is covered) got very high marks. The rest of the sample gave similar reports that supported the idea that doing Section (2) has a positive effect on the students' overall grades in the English language. It was also obvious that almost all of the teachers agreed about the simplicity and straightforwardness of the questions of Section (2) in the SSCE. One teacher said, "The LANGUAGE CIRCLE: Journal of Language and Literature 14(1) October 2019 p-ISSN 1858-0165 Available online at http://journal.unnes.ac.id e-ISSN 2460-853X 41 questions are clear and directly related to the stories." Another said, "The questions are simple and objective in contrast to the difficult and subjective questions of Section (1)." Another described the questions of Section (1) as "tough because they include writing as well as summary." One teacher reported that Section (2) "could be answered in 10 minutes, which could help the students to spare much time for the compulsory language questions of the paper." Another described the transformational items as "difficult" and said, "The student might lose the whole mark of the question for the slightest mistake," since the question is scored in an indivisible way in the SSCE, i.e., the student either loses or gains the alloted mark. To sum up this point, it can be stated there is little challenge in the questions of Section (2) if compared to their equivalents in Section (1). This feature obviously attracts the students to choose the former. Nevertheless, the students’ achieve- ments and the high marks obtained by the students who answer Section (2) cannot be attributed to the materials of the SRP or the role that they have really played in improving the students’ language. These can be explained by the nature of the examination and the type of questions and skills tested in the questions of Section (2) in the SSCE. Excellent language students will avoid doing the writing tasks since they know they might lose marks. It is then much safer for them to choose a straightforward objective question about the people, events and content of a simple story and guarantee the full or, at least, a high mark. As for the items in Section (2) in the SSCE, the teachers reported that the students preferred the MCQs, True-or-false and matching questions as they usually encountered no problems in answering them. However, the students would face some difficulty with the open-ended wh-questions as they might make vocabulary, spelling and grammar mistakes. One teacher gave examples of some of these mistakes. According to the experience of one of these teachers in scoring the questions of Section (2) in the SSCE, the marking scheme does not penalize the student for making such mistakes when the point that is asked about is given correctly. In other words, the teachers have to focus on assessing students' comprehension and ignore any language mistakes in their answers. Of course, this practice can raise conflicting controversies about the purpose and validity of the wh- questions in Section (2). In contrast, mistakes of such kind in Section (1) are usually penalized in the summary and letter-writing questions. The criterion of fairness can, accordingly, be brought to questioning. The existence of the two optional questions the way they appear in the SSCE will for definite affect the overall reliability and validity of the English language examination. This could explain why most of the teachers of the sample expressed their concern about the optionality of the SRP as a learning material in the syllabus and as a tested component in the SSCE. The teachers who teach the components of the SRP know well that their students will outperform their peers who have been disadvantaged by not being able to study the books of the SRP. Moreover, in order for the potential benefits of the SRP to be realized, the teachers felt it should be made a compulsory component of the English language syllabus. In their testing practices, the teachers revealed varying attitudes toward testing the components of Section (1), ranging from total withdrawal from the monthly tests to dominant presence. One teacher reported that they did not usually include the questions of Section (2) in their school examinations, until the students complained about this absence, and made a plea for the administration to include Section (2) in the final examination. One of the teachers of the sample reported that although most of his students usually did well in Section (2) in the school tests, many of them did not choose it in the SSCE. This was regarded as an odd situation that could be refuted by the reports of the majority. LANGUAGE CIRCLE: Journal of Language and Literature, 14(1) October 2019 42 Another teacher reported that although Section (2) is included in the school tests, its marks are not usually added to the student's overall grade in the subject. It can, therefore, be stated that the testing situation of having these two optional sections in the school tests as well as in the SSCE has created trouble and unbalanced assessment practices. It is relevant to this point to draw the attention to the fact that the whole subject of the English language might be affected. If the students determined that they would do Section (2), there would be no incentive left to them to spend time with, or do practice on, the language items that have no direct reward to them. Some students would drop important sections of the syllabus, a negative backwash that impacts the teaching-learning process. CONCLUSION The presence of two competing sections, each of a different nature, in the same achievement test is an unhealthy feature. Each of the two sections discussed in this paper generates a different type of performance and requires a certain type of assessment criteria. The content validity of the test will be affected since each section tests different skills. Examining the students' writing skills as in the letter-writing and summary questions or their grammatical knowledge of sentence building (transform- ational items) and knowledge of the prepositions cannot be equated with examining their knowledge of some story content. The two tasks are entirely dissimilar and necessitate different types of assessment: one being objective, and another subjective. It is also worth noting that not all students in the different parts of the country can enjoy the high marks boon, because the supplementary readers tested in Section (2) are part of an optional component. If it were necessary to provide the test-takers with options, then it would be of great importance to consider the factors that affect the balance of items and ensure the test satisfies the essential qualities of validity, reliability and fairness. REFERENCES Bachman, L. F., & Palmer, A. S. (1996). Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University Press. Bachman, L. F., & Purpura, J. E. (2008). Language assessments: Gate-Keepers or door-openers? In Spolsky, B. & Hult, F. M. (2008). The Handbook of Educational Linguistics. USA & UK: Blackwell Publishing Ltd. Brown, H. D. (2003). Language Assessment: Principles and Classroom Practices. UK: Pearson Longman. Coombe, C., Folse, K., & Hubley, N. (2013). A Practical Guide to Assessing English Language Learners. Michigan: The University of Michigan Press. Fulcher, G., & Davidson, F. (2007). Language Testing and Assessment: An Advanced Resource Book. USA & Canada (published simultaneously): Routledge. Harmer, J. (2010). How to Teach English (6th impression). UK: Pearson Education Limited. Hubbard, P., Jones, H., Thornton, B., & Wheeler, R. (2000). A Training Course for TEFL (16th impression). Oxford: Oxford University Press. Hughes, A. (1992). Testing for Language Teachers. Cambridge: Cambridge University Press. LANGUAGE CIRCLE: Journal of Language and Literature 14(1) October 2019 p-ISSN 1858-0165 Available online at http://journal.unnes.ac.id e-ISSN 2460-853X 43 Lynch, B. K. 2001. The ethical potential of alternative language assessment. In Milanovic, M. (ed.) (2001). Studies in Language Testing: Experimenting with Uncertainty. Cambridge: Cambridge University Press. Siddiqui, M. H. (2009). Educational Evaluation. New Delhi: APH Publishing Corporation. Weir, C. J. (2005). Language Testing and Validation: An Evidence-based Approach. New York: Palgrave MacMillan.