The particular dialect or language that a person chooses to use on any occasion is called a code Validity, Reliability and Practicality Of The First Certification in English (FCE) and The Business Language Testing Service (Bulats) Muhammad Ahkam Arifin ahkam.arifin@parahikma.ac.id; ahkam.arifin@gmail.com Institut Parahikma Indonesia Received: 10 Oktober 2018; Accepted : 28 November 2018 URL : http://ejournal.iainpalopo.ac.id/index.php/ideas Abstract This paper begins with the test specifications of the two tests – the First Certification in English (FCE) and the Business Language Testing Service (BULATS). It will then go on to the evaluation of the test usefulness: reliability, (construct) validity, backwash, and practicality (Bachman & Palmer, 1996; see Kunnan, 2004 for a slightly different perspective). This paper explores the test specifications at the outset in that a test would be evaluated (as estimated) based on the test purpose and construct that it is trying to measure (Luoma, 2004). To begin the evaluation, the test (score) reliability would be evaluated first, for a test would not be considered valid if it is not reliable (Brown, 1996; but see Moss, 1994 when a test could be valid without reliability). Throughout this paper, the term “test(ing)” will be used more or less synonymously with “assess(ment)” and “measure(ment)”, in that Bachman and Palmer point out that in the field of language testing these terms have been very broadly defined “as the process of collecting information” to make decisions (2010, p. 20). (See Bachman, 1990; Cohen & Swedlik, 2010; Douglas, 2010 for the distinctions, e.g., a test is a tool for assessment.) Keywords: evaluation, validity, reliability, practicality, first certification in English, business language testing service 80 Copyright © 2018 The Author IDEAS is licensed under CC-BY-SA 4.0 License Issued by English study program of IAIN Palopo IDEAS Journal of Language Teaching and Learning, Linguistics and Literature ISSN 2338-4778 (Print) ISSN 2548-4192 (Online) Volume 6, Number 2, December 2018 pp. 49 – 61 http://u.lipi.go.id/1457703302 mailto:ahkam.arifin@parahikma.ac.id mailto:ahkam.arifin@gmail.com Muhammad Ahkam Arifin Validity, Reliability And Practicality Of The First Certification In English (Fce) And The Business Language Testing Service (Bulats) Introduction Both the FCE and BULATS are produced by the University of Cambridge Local Examinations Syndicate (UCLES), both of which offer either a paper- based or computer-based exam, covering the four skills1. This paper will focus upon the FCE paper-based speaking test and BULATS computer-based online speaking test. The FCE is intended for learners who want to live, work or study in an English-speaking environment (UCLES, 2015). The FCE is targeted at level B2 (Upper-Intermediate) on the CEFR2, while the BULATS at all levels (UCLES, 2011). The BULATS is intended for “people at work or students studying business courses” (UCLES, 2011, p. 2). It takes 14 minutes for the FCE and 15 minutes for the BULATS to complete. The FCE is composed of four sections: (1) an interview between the interlocutor3 and each of the two candidates, (2) a presentation from each candidate, (3) a peer-peer interaction between the candidates, and (4) a discussion between the examiner and candidates. The BULATS, however, involves five parts; (1) interview, (2) reading aloud, (3) presentation about a work-related topic, (4) presentation with graphics, and (5) a communication activity by responding to questions on a specific situation. The construct definitions of the FCE include grammatical and lexical resource, discourse management, pronunciation and interactive communication (please see Appendix 1A, 1B & IV for detailed definitions); the construct definitions include what construct / abilities / skills, that a test is intended to measure (Hughes, 2003; see Chappele, 1998 for three types of construct definitions). The BULATS focuses upon the student task achievement, coherence / discourse management, language resource, pronunciation, and hesitation / extent (see Appendix II). For the “reading- aloud” part, the constructs are focused upon the student overall intelligibility (pronunciation), individual sounds, stress, including rhythm, and intonation (see Appendix III). The FCE employs analytic rating scales, for they set a number of criteria each of which has descriptors at the different levels of the scale, whereas the BULATS employs holistic rating scales, that is, to report an overall impression of the student ability in one score (Fulcher, 2003). The FCE interlocutor uses the holistic rating scale (see Appendix 1C), whereas the assessor uses the analytic one (Galaczi, 2008). Regarding the interpretations of the test results, both the FCE and BULATS can be categorized into criterion-referenced testing (CRT) in that the student scores are interpreted in relation to one or more standards, objectives and other criteria, e.g., what they can and cannot do (Hughes, 2003; see Brown, & Hudson, 2002 for the development of CRT in response to NRT4). However, Wiliam argues that “the requirement that a criterion is useful for distinguishing 1 The four skills: reading, writing, listening and speaking. 2 CEFR stands for Common European Framework of Reference. 3 In Cambridge ESOL terms, the “interlocutor” is the examiner who participates in the test and provides a global mark based on a holistic scale; while the nonparticipating examiner, called the “assessor”, awards four analytical marks. 4 , Norm-referenced testing (NRT) IDEAS, Vol. 6, No. 2, December 2018 ISSN 2338-4778 (Print) ISSN 2548-4192 (Online) levels of performance means that we have to use norms, however implicitly” (1993, p. 341). Considering that both assess the student level of language ability without respect to any particular program or curriculum, both could be considered proficiency tests (Brown, 1995). However, both could also be regarded as admission tests, for the FCE is used for “entry to undergraduate programmes” 5 and BULATS is “for admission to study business-related courses” 6. Reliability Evaluation This section will in turn focus upon the reliability of the FCE and BULATS; reliability is simply defined as the “consistency of scoring or measurement” (Bachman & Palmer, 1996, p. 19). That is, the more similar scores the students have for the same test by taking it in different settings or situations, the more reliable the test is said to be (Hughes, 2003). One common way to analyse the reliability of a test is through “correlation”, that is, a statistical indicator for the strength of relationship between two (or more) sets of measures which are considered to be related (Luoma, 2004). This relationship is then known as the correlation coefficient (Davies et al., 1999). Theoretically, values for correlation coefficients range between .00 and 1.00. While values close to zero indicate no relationship, values close to 1 means a perfect positive correlation (Hughes, 2003), notwithstanding neither extreme never occurs in practice. Carr (2011) argues that a reliability of .80 is generally set as a minimum level for high-stakes testing, whereas Lado (1961) claims that the speaking test ranging from .70 to .79 would be reliable enough. The FCE speaking test has a reliability of .847, which could be considered to be quite high (Lado, ibid). Hacket (2002) reports that the overall reliability alphas of the BULATS vary from .95 to .96, while each section ranges between .85 and .92. Using Item Response Theory (IRT) (e.g., the Rasch model) 8, Jones (2000) shows that the standard error measurement (SEM) of the overall BULATS test is .33, whereas the overall FCE test has a SEM of 2.78 and of 1.50 for the speaking test9. The SEM concerns with the reliability of individual scores rather than the reliability of tests, that is, the estimation of how close the individual actual (or true) score; the variability caused by other factors (e.g., motivation or tiredness) is called error (Hughes, 2003; McNamara, 1996). Hence, statistically the BULATS could be said to have higher reliability than the FCE. Throughout this paper the term “reliability” is synonymous with “dependability”10, or what Brown (1990) calls “decision consistency”. 5 http://www.cambridgeenglish.org/exams/first/ 6 http://www.bulats.org/why-bulats 7 http://www.cambridgeenglish.org/research-and-validation/quality-and-accountability/ 8 IRT is a general measurement theory. It assumes that for an item with a given level of difficulty, the probability that a test taker will answer correctly depend on their level of ability. Rasch model is one type of IRT (Carr, 2011). 9 http://www.cambridgeenglish.org/research-and-validation/quality-and-accountability/ 10UCLES (2013) uses both these terms more or less interchangeably. 82 Muhammad Ahkam Arifin Validity, Reliability And Practicality Of The First Certification In English (Fce) And The Business Language Testing Service (Bulats) Dependability has been used for CRT and reliability for NRT (e.g, Brown & Hudson, 2002; see also Ennis, 1999 for the use of “consistency” over “reliability”). Orr (2002) reveals that the raters for the FCE speaking test are found to not heed the same aspects of criterion, which then will result in giving different scorers. The same issue may apply to any other types of test (McNamara, 1996), which means that it could also apply to the BULATS. (Further research is needed.) However, inasmuch as the UCLES provides rater training, the reliability then could be enhanced in the sense that the raters would arguably give a similar score albeit on different occasions, and the similar score would also be given by another rater; the former is called “intra-rater reliability”, or “internal consistency” (Luoma, 2004), while the latter “inter-rater reliability” (Hughes, 2003). Moreover, Bonk and Ockey (2003) report that returning raters will tend to move toward better consistency, as they get more experience. The use of rating scales (or rubric), more varied pattern of interaction in the FCE (as well as two examiners) and BULATS may also reduce the subjectivity of the scores that will affect the reliability. (See Bachman, 1990 for the problematic distinction between the so-called subjective and objective tests.) To conclude, Weigle`s claim that a holistic scale have weaker reliability than an analytic scale may not apply to all contexts (Weigles, 2002), particularly the speaking tests for both the FCE and BULATS. Validity Evaluation Secondly, both tests would be evaluated whether they accurately assess what they are intended to assess, this evaluation is called validity (Davies, 1990; Brown, 2005). Nonetheless, Green (2014) considers this definition to be classic, and goes on claiming that this “definition is now seen to be too limited and somewhat misleading” (2014, p. 75). Standards for Educational and Psychological Testing defines validity as “the degree to which evidence and theory support the interpretation of test scores entailed by proposed tests” (American Educational research Association et al., 1999). Carr (2011) also warns that we should not speak of validating a test per se, but we should speak of validating specific uses of a test. Throughout this paper validity is used synonymously with construct validity, given that the broader field of educational and psychological measurement and testing uses construct validity as “the whole of validity theory” (Shepard, 1993, p. 418; see Thorndike & Hagen, 1986, Cronbach & Meehl, 1995 for the past use of construct validity under validity). To evaluate the degree of the construct validity of the FCE and BULATS tests, one essential type of validity evidence (not type of validity) to explore is content validity, also called ‘‘definition validity’’ and ‘‘logical validity’’ (Newman et al., 2006). A measurement could be said to have content validity if its content represents the full range of constructs (or knowledge, skills, abilities) that it is intended to cover (Alderson et al., 1995). Using observation checklists which contain a set of functions (see Appendix IV), O'Sullivan et al. (2002) report that the contents of the FCE have constituted a representative sample of those functions (or construct). For the BULATS, in the handbook a list of possible functions that will be tested in the speaking test is provided. IDEAS, Vol. 6, No. 2, December 2018 ISSN 2338-4778 (Print) ISSN 2548-4192 (Online) Thus, I assume the BULATS have content evidence validity, considering also that the BULATS is constructed by a group of language testing experts, who will have to provide argument regarding the type of underlying construct for each of the test item (or prompt). (Further research is needed). A similar type of evidence is face validity. As a stakeholder, I believe that both tests have face validity in that they directly ask the learners to speak (Green, 2014), albeit the BULATS could be considered to be semi-direct, for there is no face-to-face interaction with an interlocutor (Fulcher, 2003; but see Carr, 2011 who finds the distinction of direct and indirect tests problematic). Other types of evidence are concurrent-validity and predictive validity, both are under “criterion-related validity” (Weir, 2005). Using predictive validity evidence by examining university students, Al-Musawi and Al-Anshari (1999, p. 389) claim that “the multivariate of the GPA from the scores on the FCE is very accurate”. However, they do not focus solely on the speaking section. Weir (2005) points out that concurrent validation is concerned with the comparison of test scores with another measure of performance, e.g., another well-established test taken at the same time, teacher ranking of students, or student self-assessment. Regarding the form of computer test of the BULATS, Chambers and Ingham (2011) report that test takers and examiners show overall positive feedback on the use of computer for the speaking test, and it could be considered to be type of evidence validity called “response validity” (Alderson et al., 1995). Nevertheless, they also find that some may find it more stressful, for they will not have an interlocutor support particularly when they need the questions to be repeated. Washback and Practicality Evaluation Messick claims that “for a fully unified view of validity” (1989, p. 18), social values and consequences of a test should be taken into considerations, and these consequences have been called “impact”, that is, the effects of a test on “individuals, policies or practices, within classroom, the school, the educational system or society as a whole” (Wall, 1997, p. 291). It then leads to the creation of Code of Ethics for the International Language Testing Association (see Davies, 2003). Seemingly the most commonly discussed aspect of impact is backwash11 (or washback). Backwash has been generally defined as the beneficial or harmful effects the tests have on teaching and learning (e.g., Hughes, 2003; Alderson & Wall, 1993). Comparing the BULATS and FCE, I believe that the latter has more beneficial backwash because it is more authentic than the former, which can be considered to be semi-direct (Carr, 2011). If I were teaching to prepare my students for the FCE, I would create more opportunities for my student to have more discussion among them in my classroom, for the FCE provides peer-peer interaction and discussion. However, the FCE also may bring harmful impact to the extent that it may create subjectivity as the examiners can directly identify the students (lack of privacy and confidentiality), which then may lead to the fairness issue; privacy and confidentiality are basic rights 11 Although washback is rarely found in dictionaries, it has been commonly used in applied linguistics today. 84 Muhammad Ahkam Arifin Validity, Reliability And Practicality Of The First Certification In English (Fce) And The Business Language Testing Service (Bulats) to the test takers (Bachman & Palmer, 1996). Regarding practicality, in classroom contexts the FCE could be considered to be more practical in that teachers may need to provide computers and the internet connection for teaching the BULATS. Nonetheless in administrating the test, the BULATS could be more practical as it will not require the examiners to be in the testing room, rather it uses computer to record the student voice. At any rate, Bachman and Palmer (1996) cautions that a test should not consider practicality to be less important than other qualities, instead all qualities should be weighed the same. References Alderson, J. C., & Wall, D. (1993). Does washback exist? Applied Linguistics, 14(1), 115–129. Alderson, J.C., Clapham, C & Wall, D. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press. American Educational Research Association, American Psychological Association and National Council on Measurement in Education. (1999). Standards for Educational and psychological testing. Washington, DC: American Educational Research Association Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press. Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press. Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice. Oxford: Oxford. University Press. Bonk, W. J. & G. J. Ockey. (2003). A many-facet Rasch analysis of the second language group oral discussion task. Language Testing 20(1), 89–110. Brown, J. D. (1990). Short-cut estimators of criterion-referenced test consistency. Language Testing, 7(1), 77-97. Brown, J. D. (1995). The elements of language curriculum: A systematic approach to program design. Boston: Heinle & Heinle Brown, J. D. (1996). Testing in language programs. Upper Saddle River, NJ: Prentice Hall Regents Brown, J. D. (2005). Testing in language programs: A comprehensive guide to English language assessment. New York, NY: McGraw Hill College. Brown, J. D., & Hudson, T. (2002). Criterion-referenced language testing. Cambridge: Cambridge University Press. IDEAS, Vol. 6, No. 2, December 2018 ISSN 2338-4778 (Print) ISSN 2548-4192 (Online) Carr, N. T. (2011). Designing and analyzing language tests. Oxford: Oxford University Press. Chambers, L and Ingham, K. (2011). The BULATS online speaking test. Research Notes 43(1), 21–25. Chapelle, C. (1998). Construct definition and validity inquiry in SLA research. In Bachman, L. and Cohen, A., (eds), Interfaces between second language acquisition and language testing research. Cambridge: Cambridge University Press, 32–70. Cohen, R. J., & Swerdlik, M. E. (2010). Psychological testing and assessment: An introduction to tests and measurement (7th ed.). New York: McGraw-Hill. Cronbach, L. J., & Meehl, P. E. (2010). Construct validity in psychological tests. Psychological Bulletin, 52(1), 281-302 Davies, A. (1990). Principles of language testing. Oxford: Blackwell. Davies, A. (2003). Three heresies of language testing research. Language Testing 20(4), 355-368. Davies, A., Brown, A., Elder, C. and Hill, K. (1999). Dictionary of Language Testing. Cambridge: Cambridge University Press. Douglas, D. (2010). Understanding language testing. New York: Routledge Ennis, R. H. (1999). Test reliability: a practical exemplification of ordinary language philosophy. In R. Curren (ed.), Philosophy of education. Urbana, IL: The Philosophy of Education Society, 242–48. Fulcher, G. (2003). Testing second language speaking. London, UK: Pearson- Longman Galaczi, E. D. (2008). Peer–peer interaction in a speaking test: The case of the First Certificate in English examination. Language Assessment Quarterly, 5(2), 89-119. Green, A. (2014). Exploring language assessment and testing: language in action. New York: Routledge. Hackett, E. (2002). Revising the BULATS standard test. Research Notes 8(1), 7-10 Hughes, A. (2003). Testing for language teachers. Cambridge: Cambridge University Press. Jones, N (2000). BULATS: a case study comparing computer-based and paper-and-pencil tests. Research Notes 3, 10–13. 86 Muhammad Ahkam Arifin Validity, Reliability And Practicality Of The First Certification In English (Fce) And The Business Language Testing Service (Bulats) Kunnan, A. J. (2004). Test fairness. In M. Milanovic and C. Weir (Eds.), European language testing in a global context. Cambridge: Cambridge University Press. Lado, R. (1961). Language testing. London: Longman. Luoma, S. (2004). Assessing speaking. Cambridge: Cambridge University Press. McNamara, T. (1996). Measuring Second Language Performance. London: Longman. McNamara, T. F. & Roever, C. (2006). Language testing: The social dimension. Oxford: Blackwell. Messick, S. (1989). Validity. In R. Linn (ed.), Educational Measurement. New York: Macmillan, pp. 13–103 Moss, P. A. (1994). Can there be validity without reliability?. Educational researcher, 23(2), 5-12. Newman, I., Newman, C., Brown, R., & McNeely, S. (2006). Conceptual statistics for beginners (3rd ed.). Lanham, MD: University Press of America. Orr, M. (2002). The FCE speaking test: Using rater reports to help interpret test scores. System, 30(2), 143-154. O'Sullivan, B., Weir, C. J. and Saville, N. (2002). Using observation checklists to validate speaking-test tasks. Language Testing 19(1): 33–56. Shepard, L. A. (1993). Evaluating test validity. Review of Research in Education, 19(1), 405–. 450. Thorndike, R. L., & Hagen, E. P. (1977). Measurement and evaluation in psychology and education. New York: Wiley. UCLES (University of Cambridge Local Examinations Syndicate). (2015). Cambridge English First: Handbook for Teachers. Cambridge: University of Cambridge Local Examinations Syndicate. UCLES (University of Cambridge Local Examinations Syndicate). (2011). BULATS Business Language Testing Service: Information for Candidates. Cambridge: University of Cambridge Local Examinations Syndicate. UCLES (University of Cambridge Local Examinations Syndicate). 2013. Principles of good practice quality management and validation in language assessment: Validity, reliability, impact, practicality. Cambridge: University of Cambridge Local Examinations Syndicate. IDEAS, Vol. 6, No. 2, December 2018 ISSN 2338-4778 (Print) ISSN 2548-4192 (Online) Wall, D. (1997). Impact and washback in language testing, in C. Clapham and D. Corson (eds.), Encyclopedia of Language and Education, 291–302 Weigle, S. C. (2002). Assessing writing. Cambridge: Cambridge University Press. Weir, C. J. (2005). Language testing and validation: An evidence based approach. Houndgrave, Hampshire: Palgrave MacMillan. Wiliam, D. (1993). Validity, dependability and reliability in national curriculum assessment. The Curriculum Journal, 4(3), 335-350. 88 Muhammad Ahkam Arifin Validity, Reliability And Practicality Of The First Certification In English (Fce) And The Business Language Testing Service (Bulats) IDEAS, Vol. 6, No. 2, December 2018 ISSN 2338-4778 (Print) ISSN 2548-4192 (Online) 90 Muhammad Ahkam Arifin Validity, Reliability And Practicality Of The First Certification In English (Fce) And The Business Language Testing Service (Bulats) IDEAS, Vol. 6, No. 2, December 2018 ISSN 2338-4778 (Print) ISSN 2548-4192 (Online) 92 Muhammad Ahkam Arifin Validity, Reliability And Practicality Of The First Certification In English (Fce) And The Business Language Testing Service (Bulats) IDEAS, Vol. 6, No. 2, December 2018 ISSN 2338-4778 (Print) ISSN 2548-4192 (Online) 94 Muhammad Ahkam Arifin Validity, Reliability And Practicality Of The First Certification In English (Fce) And The Business Language Testing Service (Bulats) Introduction Reliability Evaluation Validity Evaluation Washback and Practicality Evaluation