THE ENGLISH IMMERSION PROGRAM: Jane Lockwood The English Immersion Program: Measuring The Communication Outcomes THE ENGLISH IMMERSION PROGRAM: MEASURING THE COMMUNICATION OUTCOMES Jane Lockwood Department of English, City University of Hong KongEmail: lockwood@cityu.edu.hkAPA Citation: Lockwood, J. (2015). The English immersion program: Measuring the communicationoutcomes. Indonesian EFL Journal, 1(1), 98-107Received: 10-09-2014 Accepted: 13-10-2014 Published: 01-01-2015 Abstract: This paper explores how language assessment is typically used to measure language gain asa result of the Immersion experience abroad. It also explores ways in which this might be improved.This study explores a recent experience where Australian immersion providers, participants andfunders all report significant intercultural awareness raising and improved confidence inunderstanding and speaking in English as a result of the sojourn. However, it transpires that theimmersion providers used traditional proficiency focused language assessment tools on entry and exitto measure communication outcomes across the skills of listening, speaking, reading and writing. Itappears therefore, that there is a gap between what the perceived outcomes and value are of theImmersion experience, and how they are currently measured. This article reports on a small scalestudy exploring the perceptions of two immersion providers in Australia, one immersion coordinatorin Hong Kong and four returnees on the language assessments they used and experienced, particularlyprobing on how well they felt these assessments measured their communication gains as a result ofthe immersion experience. Keywords: Language assessment, immersion, indigenous criteria. INTRODUCTIONMillions of students head off tointernational destinations to spend timeliving with ‘homestay’ families and studyingin tertiary institution that offer differentlanguage and cultural experiences. OECDfigures predict that this number will reach 8million by 2025 (Davis 2003). In Asia,Universities have, over the last two decades,been sending increasing numbers of studentsto English speaking destinations such as UK,USA, Canada, Australia and New Zealand toimprove their English language skills in theauthentic contexts. (Bodycott & Crew 2001).In 2008-2009, the Hong Kong Institute ofEducation spent just under HKD2 millionsubsidizing short term immersion programsin such English speaking destinations as UK,Canada and Australia, but little data exist onhow effective such programs are in terms ofEnglish language communication gain.This article reports on two languageimmersion providers’ tools and processesused to measure language proficiency gain.These measures are briefly described and then the outcomes of key stakeholder groupsincluding the providers, the returnees and theHong Kong based immersion coordinator areanalyzed in terms of the efficacy of suchmeasures. The Immersion experience of HongKong Institute of Education undergraduatestudents takes place over a full semesterwhere they live with homestay families andattend university programs in either Canada,Australia or the UK. Each student does a pre-and post - course assessment as well as thecompletion of an ethnographic study as partof their program.If intercultural awareness is a key areain which sojourners are expected to makeprogress while abroad, then appropriatemethods of assessment should be used tomeasure the gains they have made. Whilemuch attention has focused on thepreparation, format, and the content of thestudy abroad programs, relatively little hasbeen published about the modes ofassessment (Jackson 2005:165). This articlewill report on student and staff perceptionsof the communication gains and finally it 98 Indonesian EFL Journal, Volume 1 (1) January 2015ISSN 2252-7427report on student and provider perceptionsof the efficacy of the assessments used andsuggestions for improvement. What do English language assessment practitioners and researchers say about measuring communications skills?English language assessment hasreceived a great deal of attention fromapplied linguists over the last 35 years. In thepast, teachers have mostly concernedthemselves with simply testing the discretelanguage learning outcomes of programdelivery in language learning classrooms.Commercial English language tests aboundbut tend to focus on the measurement ofgeneric or academic English languageproficiency gain across the skills of listening,speaking, reading and writing. These trendsin turn have resulted in a preoccupation withthe summative assessment comprising of aseries of discrete language items that add upto an overall score. This emphasis on thequantitative score as the sole indicator ofprogram success reflects a psychometricapproach to language program evaluationand has been seriously challenged bylanguage assessment experts (Hamp Lyons1991; Shohamy 1992; Brindley 1995;Bachman and Palmer 1996, McNamara 1996,Douglas 1999).More recently much of this research hascentred around how language teachers andassessment providers can improve their toolsand processes to reflect the whole context aswell as the specific purpose for assessment.There has been a paradigmatic shift frompositivist models of language assessment tomore constructionist models where thelanguage assessment tools and processes aredesigned to be more responsive to theparticular contexts and purposes for languageassessment.Much discussion has also taken placeabout the introduction of ‘formative’ languageassessment tasks e.g. assessment for learningtasks that take place in the class room,portfolio assessment and other on-goingassessment tools (Hamp Lyons 1991) that arequalitative but can also yield scores. As well,much discussion has also taken place aboutthe veracity of reporting on language proficiency indicators as opposed to languageperformance indicators as being more or lessappropriate for different contexts andpurposes of language training (McNamara1996); of qualitative rather than quantitativeindicators (Shohamy 1998); of tailor madeassessments as opposed to standardizedassessment ( Douglas 1999, Pennington1998).The two unifying themes in the mostrecent language testing and educationalliterature revolve around the related issues ofbeing open to different forms of assessmentand being open to how these forms ofassessment can be context and purposesensitive. The language testing literature isrich in the discussions about the relativemerits of using different applied linguisticframeworks to achieve construct validity andreliability in the design and use of any test.Much of the discussion has revolved aroundpushing criteria for language assessmentbeyond the traditional preoccupations withpronunciation and grammatical accuracy to aconsideration of broader communicativedomains such as discourse and interactivecapabilities (Canale and Swain 1980;Bachman and Palmer 1982; Bachman andSavignon 1986; Bachman 1990, Davies 1988;Hughes 1989; Weir 1993, Douglas 2005).Assessment practices that reflectcommunicative approaches to languagetraining in Language for Specific Purposes(LSP) contexts is also well covered (Lumleyand Brown 1996; McNamara 1997; Douglas2000; Elder 2001), and well–documentedaccounts of test development for differentgroups of occupations and professionalsabound. McNamara (1990,1996);Elder (2001)and McDowell (1995) have both developedstandardised and performance-based testsfor a range of teachers and healthprofessionals in Australia and Douglas (2000)looks at specific language use situations todevelop test content and test methods forhighly specific LSP, such as English for pilotsand air traffic control. Such frameworks,however, have yet to be fully understood in anindustry context and incorporated intobusiness language assessment practices on-site (Lockwood 2002; Lockwood 2008).In recent discussions about language for 99 Jane Lockwood The English Immersion Program: Measuring The Communication Outcomesspecific testing (Jacoby and McNamara 1999,Douglas and Myers 2000, Douglas 2001),there has been a call for what they havetermed “indigenous assessment criteria”.Such criterion is derived from the targetlanguage use (TLU) context. Performanceassessment practices are seen as morerelevant to knowing whether a candidate canhandle a complex professional and/or socialsituations where mere language proficiencyindicators are not likely to provideappropriate performance profiles. TheImmersion context can be viewed as another‘indigenous assessment’ site encompassingthe homestay context and the universitycontext. What is interesting about this contextis the intercultural richness of the immersionexperience on the one hand and therequirement, on the other hand, to reportoutcomes in terms of language proficiencygains to secure and maintain funding.Much has been written on interculturalassessment (Kramasch and Sullivan 1996;Byram and Fleming 1998). All start withattempts to define what it is exactly we aretrying to assess when we assess interculturalperformances; is it knowledge?; is it skills?; isit behaviours?; is it motivation?; is it all ofthese? Some even argue the efficacy of such agoal as the perfect mastery of a secondlanguage where acculturalization is the aim(Seelye, 1984). A large number ofstandardized measures exist to assessintercultural knowledge ( Allen & Herron,2003; Coleman, 1995; Redden, 1975; Corbitt,1998; Pedersen, 2010); these are typicallyachieved through surveys, inventories,proficiency exams and multiple choice teststo quantify the amount of knowledgeabsorbed regarding the target culture.However, Earley and Ang (2003) suggest thatthe assessment of intercultural intelligence ismuch more than mere cognition andknowledge. They propose a threedimensional model that encompassesknowledge, behaviour and motivation. Theyfurther propose that non - psychometricmethods can be applied appropriately toassess the motivational and behaviouralcomponents of cultural intelligence (CQ).Jackson (2005) came to much the sameconclusion in the study she carried out which involved introspective, qualitative assessmentof the sojourn experience as captured by agroup of Hong Kong students when away onimmersion. In this study students wereencouraged to use first person introspectiveaccounts (in the form of diaries) to assess thelearning processes of short term sojournersas a way of measuring intercultural andcommunication gains. These were thenassessed against standardized gradingcriteria (A-excellent ----F-fail).In an earlier study which involved HongKong students on immersion (Evans, Alanoand Wong 2001), three levels of interculturalawareness were hypothesized as ‘emergent,growing and enhanced’ (p.96) andvideotaped discussions Hong Kong studentson English speaking Immersion programswere analyzed for both linguistic and culturalgains. Another study in the late 1990’s(Murdoch and Adamson 2001) investigatedthe extent and range of English use beforeand during a 4 - week immersion experiencein Australia as a way of explaining thesociolinguistic gain from this experiencereported by students. Not surprisingly thesestudies showed considerable increase inEnglish use in a range of social andeducational settings and situations. Thesekinds of studies may hold the key forimagining more enriched assessment toolsand processes for Immersion programs thattake Hong Kong students away overseas. METHODThis study is a qualitative enquiry intothe pre and post course languageassessments of two HKIED languageimmersion providers domiciled in Australia.Two immersion provider coordinatorsprovided documentation regarding theassessments they currently use, the most upto date records of the results, as well as theend of course reports. They were interviewedspecifically about the assessment processesin a semi- structured interview on the phone.Four returnees also agreed to be interviewedabout what they perceived to be the maingains of the immersion experience and howthese were assessed and evaluated. Theyagreed to provide their end of programethnographic studies and in one case, one of 100 Indonesian EFL Journal, Volume 1 (1) January 2015ISSN 2252-7427the returnees volunteered her diary whichshe had kept during her stay overseas. Finally,the Hong Kong immersion coordinator wasinterviewed to elicit his views on what he feltwere the gains of the immersion experienceare and how these match with the tools andprocesses used to measure communicationgain. All interviewees were encouraged toprovide suggestions for improved andalternative measurement and these arediscussed in some detail in this article. FINDINGS AND DISCUSSION What do the pre and post immersion program scores indicate about the improved language levels of the students? The English language assessment toolsused by the two providers are described indetail later in this section. They are similar inthat they assess the four skills of listening,speaking, reading and writing. Interestingly,the results indicate gains across all the skillsand are not just confined to the skills oflistening and speaking, reading and writingalso appeared to report gains. No studentappeared to go down in any of the skills areas,although there were differences in thepercentage gains particularly in the areas oflistening and speaking where one studentappeared to have made gains of over 100%on the initial score for speaking. The averagegains across the four skills for the twoproviders are tabulated below:Table 1: Average language gains of students on immersion Listening Speaking Reading WritingProvider 1 20% 30% 10% 10%Provider 2 4.3% 6.5% 3.9% 3.75%In the data provided by both providergroups it appears that the speaking levelsfollowed by the listening levels showed mostimprovement with a fairly even distributionacross the writing and reading levels. What isinteresting in the data is the significantdifference in the reported % of gain acrossthe skills of listening and speaking betweenthe two provider groups. Provider 1 reportsmuch higher post immersion gains thanProvider 2. Given that both immersionprograms took place in Australia at about thesame time, and given that the students werenot ‘streamed’ into levels (i.e. a more ablegroup going to Provider 2 and a less ablegroup going to Provider 1), one of thevariables that needs to be considered is theassessment tools and processes themselves. What do the providers do and think about the assessment processes they carry out? Provider 1Provider 1 complied with the tenderregulation that requires pre and post courseassessment by administering an (LanguageProficiency Assessment for teachers of English)LPATE type test. This comprises a reading andlistening comprehension that test thecomprehension of authentic type texts throughcloze testing, multiple choiceand open-ended questions.The open-ended questions probereading skills that go beyond informationextraction to eliciting students’ interpretative,inferential and analytic skills of what they haveread and listened to. The writing test is alsodesigned to mirror the LPATE test andassesses both the ability to write extensive textas well as the ability to recognize student errorand use appropriate metalanguage to explainthe errors. Similarly, the speaking test mirrorsthe LPATE test tasks and assessment criteria.The students areasked to read aloud anextensive passage and are marked withcriteria that are related to pronunciationability as well as the ability to read aloud withmeaning. The last criteria is highly contentiousin that students are really being assessed ontheir reading comprehension ability as well astheir pronunciation. The second part of thespeaking test is a short semi - spontaneouspresentation on a given topic. Here thestudents are assessed on their ability toorganize what they are going to say as well astheir ability to use grammatical and lexicalitems accurately and showing range. Finallythe students are askedto participate in anopen discussion with threeother participantsaround a given topic. Here the criteriaforassessment relate to their ability to interactwell with their peers. 101 Jane Lockwood The English Immersion Program: Measuring The Communication OutcomesThe pre and post assessment record issent to the funding provider in compliancewith the tender regulation. However none ofthe returnees knew what their languageassessment results were, nor did they appearconcerned to know, as they understood thatthese were not high stakes test. The LPATEtest is marked on a 5 point scale with .5 levelsrecorded if the candidate exhibits features inthe whole number score.Interestingly Provider 1 felt there wereextreme limitations on what this languageassessment could measure as it relates to theimmersion experience. Its value seemed torelate more to compliance with the tenderregulation. However, this provider also said:We knew that the assessment would not capturewhat we felt were the real communications gainsof immersion but nonetheless we wantedsomething that was valid and reliable; somethingthat would be useful to the student and wouldalso comply with the tender regulation. Afterconsulting (an external assessment group) wedecided it would be most useful for students todo an LPATE type test as we know this is a highstakes test for the immersion students when theyreturn to Hong Kong.During the course of the interview, thecourse provider coordinator expressed herviews about the gap between the currentassessment procedure as described aboveand what she felt the real gains were in termsof communication improvement of the cohortof English major students from the HongKong Institute of Education. She said: We know that students make huge gains beyond what is routinely assessed in the pre and post course test. For example the students develop great confidence in their ability to speak and understand. At the end of immersion they are able to engage much more in informal day to day interaction as a result of living with English speaking families, travelling around Australia and making friends on campus. They allow themselves more thinking time when speaking and they rephrase what they say to make their meaning clear…there has obviously been a shift from worrying about themselves as communicators to worrying about whether they are making their meaning clear to the person they are talking to…a sure sign of a good communicator. This shift from a preoccupation with the mechanics of language and grammar accuracy to real communicative ability including more confident paralinguistic and intercultural behaviours are huge. Provider 1 suggested it would bedifficult to measure, in a standardized format,this kind of shift and suggested one vehiclefor tracking this change would be in theactual presentation of the ethnographic study,which is an end of course requirement, aswell as the quality of observation containedtherein. The provider cited one student in the2008/2009 cohort as follows: One of the mainland students made a remarkabletransformation in his body language andcommunicative confidence during immersion.When it came to presenting his ethnographicstudy, he stood up straight (where previously hehad stooped), he made eye contact and deliveredhis findings in such a confident and articulateway…I couldn’t believe the transformation butthis is not unusual.Provider 1 suggested that perhapsattempting to develop some kind ofstandardized marking scheme to capture thisshift either as part of the ethnographic studyassignment and /or the post test in speakingwould be worth considering. She suggestedthe videoing this final presentation mayprovide a source of evidence for furtherspeaking assessment and gain. It should benoted however that Provider 1 postimmersion speaking and listening scoresreported very significant gain. It is the natureof this assessment construct that is beingchallenged for validity in this chapter Provider 2Provider 2 Immersion coordinator alsorepresented a tertiary provider in Australia.Like Provider 1, Provider 2 felt there werehuge gains made in confidence andintercultural awareness that translated intoimprovements in communications strategiesnot reflected in the Provider 2pre and postcourse language assessments, which showedrelatively small gains. This coordinator alsobemoaned the lack of standardization acrossthe immersion provider group in terms of preand post course assessment. He reported thata core group of the immersion providers hadinformally agreed to administer the IELTStests as a way of aligning, although he did notbelieve that this to be an ideal tool. Currentlythis provider uses the Professional English 102 Indonesian EFL Journal, Volume 1 (1) January 2015ISSN 2252-7427Assessment for Teachers (PEAT) developed inNew South Wales. This assessment has beenspecifically designed to evaluate whether ateacher’s English proficiency is good enoughto interact effectively in a school setting. Thetest consists of four components: Listening,Reading, Writing and Speaking and resultsare given in terms of Bands A, B, C or D butare ultimately expressed in terms of anoverall percentage score. Band A means the candidate can for Listening, comprehend easily and accurately in all personal and professional contexts; for Reading, are able to read all styles and forms of the language pertinent to their professional needs; for Writing, are able to write fluently and accurately on all levels normally pertinent to their personal and professional needs ;and for Speaking, are able to use language fluently and accurately on all levels normally pertinent to their personal, social, academic or professional needs.(htt://www.trb.sa.edu.au/english_tests.htm (accessed 5/2/2010)Provider 2 had two main concerns withthe current tool and processes. First, he feltthat students do not necessarily show theirstrengths soon after arrival. He felt that in thefirst few days many students are sufferingfrom culture shock and a dip in their ownconfidence to communicate. This may be theresult of being ‘thrown in the deep end’ withtheir homestay family and generally being ina strange new environment; but whatever thereasons, it was felt that the test at this stagemay not be yielding results that arecommensurate with what the students canreally do.Provider 2 was also concerned about theareas that the test ‘do not tell you about’.When probed on this point he cited theenormous confidence gain that the studentsexperienced during the immersionexperience, which he said were evidenced bya range of improved communicationstrategies such as initiating conversations,turn taking, changing the topic, maintainingeye contact, responding to cultural references,improved pronunciation (particularlyprosodic features) and so forth. He describedthe linguistic and intercultural challenges inthe process of carrying out the ethnographicstudy as follows: To complete the ethnographic study component of the immersion program, the students first have to interview NNS students on campus…we send them out with cassette players. They then follow this up by interviewing a native speaker and then their home families. Finally they do a 4th interview of their own choice but it must be at a much deeper level.Students finally write up and presenttheir findings before they leave. From thedata collected the students chose to focus ona broad range of ethnographic study topicssuch as comparative studies betweenAustralians and Chinese on racism, public andprivate transport choices, part-time workchoices and one study was carried out toexplore the reasons for the prevalence ofgraffiti in Melbourne. What do students do by way of assessment and do they think it measures the communication gain made on immersion? What suggestions do they have? The student interviewsThree of the students interviewed hadcompleted their immersion program inProvider 1 university. When self reportingtheir perceived language gains they wereunanimous about their improved speakingand listening ability and general increase inself confidence and intercultural awarenesswhen interacting in English. Whilst thehomestay families varied in terms of age,social status and numbers of family members,the students attributed much of theirnewfound communicative confidence gainsand intercultural insights to the regularinteraction with their homestay families andto their close observations of how thesefamilies lived. The students all appreciatedthe time homestay family members took tofind out, at mealtimes, about their dailyactivities. One of the students who wasethnically Chinese, but a native speaker ofEnglish having been educated internationally,recounted the biggest challenge as beingintercultural:I had to gauge how to deal with things going on inmy homestay family like telling them that themeat portions are way too big; that I was reallyscared of one of their dogs and that I wantedthem to pick me up on Saturday night from the 103 Jane Lockwood The English Immersion Program: Measuring The Communication Outcomescity as I was too afraid to come home on the trainby myself…Even though I had the language tocommunicate with them perfectly well therewere these kinds of situations, especially in thebeginning, where I wasn’t sure whether it was theright thing to do…the right thing to say.One of the students said she wassurprised at the easy intimacy displayed byher homestay ‘mum’ which, she said, wasunusual in Chinese culture. She said shelearned about how making jokes withinfamilies about individual members was a kindof endearment and was a hallmark ofacceptability even when the jokes were risky.This same student observed this kind ofjoking among students she befriended on thecampus . She said, You know you are accepted when they make a joke about you!These students also said they feltdifferent kinds of communicative gains whenthey went travelling. They reportedserendipitous friendships struck up withother international travelers from Japan,Holland and Belgium when they travelledinterstate. As English was the ‘lingua franca’they reported a different kind of experiencein talking with fellow tourists.It was a different kind of talking because we bothhad English as a second language in common andbecause we had already spent about 10 weeks inAustralia with our families we were able todisplay our knowledge of the culture…With theJapanese speakers we adjusted our English sothat they could easily understand…We felt veryconfident at this stage of our immersion stay andvery proud of our ability to communicate.A deeper kind of cultural shift appearedto have taken place in the students when theyself reported the value of attending thedifferent programs at the provider university.One of the students articulated this well whenshe said: I had learned about student-centred and enquirybased learning before but I felt what this was likein the classroom in Australia. The lecturer onlytook 25% of our time and we had to make ourown decisions for what to do next we were left onour own a lot but for my classmates this wasquite usual.Whilst some of the students madefriends on campus and these endure through Facebook, some said they spent their socialhours with each other. This finding wasevident in one of earlier studies whichreported:The greatest deficiency (as shown by figures forreported use as well as reported perceptions)lies in opportunities presented by this programto expand sociolinguistic competence related tomeeting, conversing and socializing withAustralian peers-students of a similar age…(Murdoch and Adamson 2001:111)One of the students commented on theodd appearance of young people inMelbourne which she found strange and quiteintimidating:On my way home I had to catch a bus atFrankston station. The young people there woreweird clothes and had lots of piercings andtattoos. They looked quite rude and I didn’t daretalk to them.For all of the students the pre- and post -course assessment was completed under nopressure. One student reported that theywere told it was an institutional requirementonly and not high stakes. None of them knowtheir results from the test but recognized thatthe assessment was based around the fourskills assessed in the LPATE exam. Some saidthe assessment was very easy and all agreedthat they were not able to display the realcommunicative competency gains they feltthey had made on the immersion. They foundthe question of how to mend this gapbetween the gain and measuring the gainbetter, confounding, and commented:It’s really hard to measure what we have learnedin terms of socialization. I mean you know whensomeone is being socially or culturallyinappropriate, but it is hard to measure exactlyhow good you have become.Another student reported her ownobservations of her friend who had arrivedwith relatively low confidence in her speakingability compared to when she left, and put itdown to confidence more than languageproficiency gain. On the last day before we left my friend and I and her homestay mum went to a movie together. It was amazing to see her chatting with her host 104 Indonesian EFL Journal, Volume 1 (1) January 2015ISSN 2252-7427 mum…she could really hold her own…they were joking and messing around. When we arrived in Australia she could only talk, hesitating all the time and looking for words.All the students commented on the valueof the ethnographic project they wereexpected to complete during their immersionstay and cited this as perhaps evidence of oneaspect of their communicative development.One student suggested ‘writing about’ thegain as part of their final assessment, but ofcourse recognized its limitation as‘knowledge’ evidence rather thancommunication skills effectiveness. The Hong Kong coordinatorThere was very broad agreement fromthe Hong Kong coordinator’s point of viewwith those of the Australian providers andthe returnees about the mismatch betweenthe real communication gains and theassessment practices currently being carriedout pre- and post - immersion stay.Interestingly, he made the point that thesecurrent assessment practices may reinforcein the students’ minds that language gain isall about making improvements the kinds ofacademic language tasks in the LPATE andPEAT assessments and may inadvertentlywork against the development of the lessformal communication skills for which theimmersion experience is reported as sovaluable. He agreed however that to assessthe reported gains in confidence, interculturalawareness and the ability to participate incasual conversation to be extremelychallenging and doubted whether they couldbe captured in a quantitative score. He alsoagreed that the lack of systematic tools andprocesses across the providers would meanthat reported results could not be relied on asa measure of success and gain.Rather than changing the currentassessment practices by changing the toolitself, this coordinator felt something muchmore radical may be considered. Hesuggested that perhaps the Immersionexperience could be built into their coursebecome a 3 credit program of study roughlydivided into 3 segments, each attracting onecredit point as follows: (i) Pre departure studies –e.g. preparing e.g.how to keep a journal (one credit)(ii) Journal keeping when they are away(one credit)(iii) Portfolio when they return (one credit)This would both ensure that thestudents would make better preparation anduse of their immersion experience as well asbeing formally assessed as part of the degreestudies. This program could also embed intoit autonomous learning principles andimproved skills in self and peer assessing arange of linguistic and intercultural skills.It would appear that the overseasproviders, and the students themselves, feltconstrained and dissatisfied by the languageassessment tools and processes theycurrently use to measure the communicationoutcomes of the immersion experience. Itwould appear however, that the assessmentsused did in fact report gain across the skillswith specific reference to speaking andlistening. One of the big problems seems to bethe lack of a systematic proficiency test usedby all providers. This would generate morereliable results than the ones reported in thisstudy, assuming calibration and moderationprocesses were also in place. The other issueto emerge from this study is the need for supplementary information about theperceived ‘other’ communication gains thatare not captured in the domains of thecurrent tools being used. The fundinginstitution itself expressed doubt about thecurrent use of the pre- and post-courseassessment tools and processes and proposedbetter ways to measure the outcomes of theimmersion experience. Can the results of thepre and post course assessment really reflectthis shift? Clearly the gains appear to impactthe listening and speaking skills particularly,but go far deeper than improved languageproficiency. All interviewed in this studyreported that the immersion experienceoffers the opportunities of observing andparticipating in authentic English speakingcontexts at home and at university. These tworich contexts throw up ‘indigenous criteria’that may be able to be mapped intoimmersion communication assessment toolsand processes. As well, it could also be arguedthat a scale of intercultural gain, as suggested 105 Jane Lockwood The English Immersion Program: Measuring The Communication Outcomesby Evans et al (2003) could be mapped ontosuch a revised assessment. However, does itneed to be a test? The current literature asoutlined in a previous section of this articledescribes other forms of assessment asmeasures of gain. On-going assessmentprocesses such as diary keeping, focus groupdiscussions that probe critical experiences athome and on campus, ethnographic studiesthat involve data collection and analysis onthe people around them, all that yield richqualitative information that demonstrate “allthose shifts that formal pre and post courseassessment doesn’t”(Provider 2). A radicallyrevised program that is credit bearing, asproposed by the funding institutionalcoordinator could also embed tasks andmeasurements that reflect themultidimensional communication gains madeon immersion.One of the greatest limitations of thisstudy has been the small scale of theinvestigation and the difficulty in accessingprovider assessment tools and processes.Both tools had been developed for teachingcontexts rather than to probe language gainsfrom an immersion experience. It would be ofgreat interest to know if any of the providershave attempted to map into their languageassessment tools and processes any of the‘indigenous criteria’ that Douglas(2001)argues for in any language for specificpurpose (LSP) testing situation. Self –reporting, peer rating, provider rating of theethnographic study presentation and evenhomestay ratings using a validated rating toolmay supplement the pre and post course testas it currently exists and ultimately somekind of standardized measure may well bedeveloped. CONCLUSIONWhilst the tender documents clearlystate a requirement for pre and post courseassessments, it seems the funding authorityin this study rarely demanded to see theseresults as proof of success of the immersionexperience. Reports written by the twoproviders all reported language gains of thestudents as ascertained by the LPATE andPEAT type tests although that the percentagegains were significantly divergent. However, the returnees’ own confidence in their ownimproved language skills and in their newfound skills of living and travellingindependently were testimony ofcommunication improvement at quiteanother level. Perhaps in critiquing such toolsand processes, it is not the instrument initself that is wholly to blame, but more thelimitation of its use alone as the proposedreporting and program evaluation instrument,as well as a lack of a standard tool. This studyhas reported on a range of approaches andtypes of evidence that could be usedsystematically to evaluate success incommunication as a result of the immersionexperience. Perhaps it is therefore not somuch the assessment tools and processesthat should come under scrutiny but morethe program evaluation brief that providerinstitutions are being asked to comply with todemonstrate that the Hong Kong fundinginstitutions are getting value for money. ReferencesBachman, L & Palmer, A. (1982). The constructvalidation of some components of communicativeproficiency. TESOL Quarterly, 16, 449-465.Bachman, L & Palmer, A. (1996). Language testing in practice. Oxford: OUPBachman, L & Savignon, S. (1986). The evaluation ofcommunicative language proficiency: A critique ofthe ACTFL oral interview. Modern Language Journal. 70, 380-390Bachmann, L. (1990). Fundamental considerations in language testing. Oxford: OUPBodycott, P. & Crew, V. (2001a). Short term Englishlanguage immersion: Paradox, culture shock,participant adjustment. In P. Bodycott & V. Crew(Eds.), Language and cultural immersion: Perspectives on short term study and residence abroad (pp. 1-9). Hong Kong: Hong Kong Instituteof Education.Bodycott, P. & Crew, V. (Ed.). (2001b). Language and cultural immersion: Perspectives on short term study and residence abroad. Hong Kong: Hong KongInstitute of Education.Brindley, G. (1995). Language assessment in action.Research Series 8. NCELTRByram, M & Fleming, M. (1998). Language learning in intercultural perspective: Approaches through drama and ethnography. Cambridge. CambridgeUniversity Press.Canale, M. & Swain, M. (1980). Theoretical bases ofcommunicative approaches to second languageteaching and testing. Applied Linguistics 1, 1-47.Davies, A. (1988). Communicative language testing. InHughes, A. (ed) (1988). Testing English for university study. ELT Document No. 127. Modern 106 Indonesian EFL Journal, Volume 1 (1) January 2015ISSN 2252-7427English PublicationsDavis, T.M. (2003). Atlas of student mobility. New York:Institute of International EducationDouglas, D. (2000) Assessing languages for specific purposes. Cambridge: Cambridge University PressDouglas, D. (2001). Language for specific purposesassessment criteria: where do they come from? Language Testing, 18, 171-185.Douglas, D. & Myers, R.K. (2000). Assessing thecommunication skills of veterinary students:whosecriteria? In Kunnan, A. (eds.), Fairness in validation of language assessment. Selected papers from 19thLangauge Testign and research Colloquium. Studiesin Language Testing 9. Cambridge. CambridgeUniversity press.pp60-81.Elder, C.(2001). Assessing the language proficiency ofteachers: Are there any border controls? Language Testing, 18 (1) 149-170.Evans, M; Alano, C & Wong, J (2003) Does a ShortLanguage Course in an Immersion Setting Helpteachers to gain in Communicative Competence? InP. Bodycott & V. Crew (Ed), Language and cultural immersion Perspectives on short term study and residence abroad. Hong Kong: Hong Kong Instituteof Education.Freed, B. (1995). What makes us think that studentswho study abroad become fluent? In B. Freed (Ed.), Second language acquisition in a study abroad context (pp. 123-148). Amsterdam: John Benjamins.Hamp-Lyons, L. (1991). Reconstructing ‘academicwriting proficiency’. In Hamp-Lyons, L (ed) Assessing second language writing in academic contexts. Norwood. N.J. AblexHughes, A. (1989). Testing for language teachers.Cambridge: Cambridge University Press.Jackson, J. (2005). Language, identity and study abroad.Equinox.Jacoby, S & McNamara, T. (1999). Locating competence. English for Specific Purposes, 18(3), 213-241.Ife, A. (2000). Language learning and residence abroad:How self-directed are language students? Fremdsprachen und Hochschule, 60, 42-63.Kramasch, C &Sullivan,P. (1996). Appropriate pedagogy. ELT Journal, 50(3),199-212Lockwood, J, (2002) Language Training Design and Evaluation Processes in Hong Kong Workplaces.Unpublished PhD. thesis. Hongkong: HKULockwood, J. (2008) What does the Business ProcessingOutsourcing Industry (BPO) industry want fromEnglish language assessment? Prospect 23,2 (2) Special Issue pp.67-78.Lumley, T. & Brown, A. (1996) Specific purposelanguage performance tests: task and interaction. Australian Review of Applied Linguistics.McAllister, L., Whiteford, G.., Hill, B., Thomas, N., &Fitzgerald, M. (2006). Reflection in interculturallearning: Examining the international experiencethrough a critical incident approach. Reflective Practice, 7(3), 367-381. Mc Dowell, C. (1995). Assessing the languageproficiency of overseas-qualified teachers- TheEnglish Language assessment (ELSA) In Brindley, G(1995). Language assessment in action Australia.Sydney: NCELTRMacFarlane, A. & Wesche, M. B. (1995). Immersionoutcomes: Beyond language proficiency. Canadian Modern Language Review, 51(2), 250-274.McNamara, T. (1990). Assessing the second language proficiency of health professionals. UnpublishedPhD. Melbourne: University of Melbourne.McNamara, T. (1996) Measuring Second Language Performance. London: LongmanMcNamara, T. (1997) Interaction in second languageperformance assessment: Whose performance? Applied Linguistics, 8, 446-466.Murdoch, P., & Adamson, B. (2003). The effects of ashort term study abroad programme uponsociolinguistic competence in the use of English asa second language. In P. Bodycott & V. Crew (Ed.), Language and cultural immersion: Perspectives on short term study and residence abroad (pp. 101-112). Hong Kong: The Hong Kong Institute ofEducation.Pedersen, P. (2010). Assessing interculturaleffectiveness outcomes in a year-long study abroadprogram. International Journal of Intercultural relations. 70-80.Pennington, M. (1998). Designing language programevaluations in specific purposes. In Li,E. & James,G. Testing and evaluation in second language education. Hongkong: HKUST.Seelye, H.N. (1984) Teaching culture: Strategies for intercultural communication. Lincolnwood. Illinois:National Textbook Company.Sercu, L. (2004). Assessing intercultural competence: Aframework for systematic test development offoreign language education and beyond. Intercultural Education, 15(1), 73-89.Shohamy, E (1992) New modes of assessment: Theconnection between testing and learning. InShoahmy, E &Walton, A.R. (eds) Language assessment for feedback, testing and other strategies.Dubuque IA: Kendall/HuntShohamy, E (1998) Alternative assessment in languagetesting: Applying a multiplism approach. In Li,E&James,G (eds) testing and Evaluation in Second Language Education. Hong Kong Language Centre,HKUST .Tilleman, H.H. (2000). Belief change towards self-directed learning in student teachers: Immersionin practice or reflection on action. Teaching and Teacher Education, 16(5-6), 575-591.Weir,C. (1990) Communicative Language Testing HemelHampstead, UK: prentice HallWilliams, T.R. (2005). Exploring the impact of studyabroad on students' intercultural communicationskills: Adaptability and sensitivity. Journal of Studies in International Education, 9(4), 356-371. 107