EJAL Article template Focus on ELT Journal (FELT) Vol 1, Issue 1, 2019 ISSN: 2687-5381  Contact: yesimkeslidollar@gmail.com A meta-analysis of the effect of bimodal subtitling on vocabulary learning among adult EFL learners aReem Ali Jaber and bYeşim Keşli Dollar a Independent Researcher, Jerusalem, Palestine, reem.amj94@gmail.com b Independent Researcher, Texas, United States of America, yesimkeslidollar@gmail.com To cite this article: Jaber, R. A., & Keşli Dollar, Y. (2019). A meta-analysis of the effect of bimodal subtitling on vocabulary learning among adult EFL learners. Focus on ELT Journal (FELT), 1(1), 43-52. https://doi.org/10.14744/felt.2019.00001 ABSTRACT A meta-analysis is conducted to investigate the impact of English subtitling on EFL learners’ vocabulary improvement. This study was conducted by collecting and analyzing previous research to investigate the effect of bimodal subtitles on vocabulary learning, among EFL learners in different contexts and settings. The main point here is to systemize the existing literature on bimodal subtitles in relation to vocabulary learning as a topic and to compare the results of different studies in this respect. Thus, second language development could be addressed through this area, which can be considered as one effective teaching method for EFL learners. In an effort to elaborate previous literature, a meta-analysis is developed to measure the overall effect size of the study, and to guide English instructors accordingly. Stata 14 software is used for the analysis. The results extracted from the 10 papers found overall positive effect of the bimodal subtitling on vocabulary learning among adult EFL learners. Keywords: bimodal subtitles subtitling meta-analysis vocabulary learning Introduction Having English as a dominant language around the world, studies in EFL (English as a Foreign Language) SLA (Second Language Acquisition) continue to grasp the language researchers’ attention into building a better system, for a better language education. In this regard, Cohen (1998) suggests that language learning requests certain strategies in classroom for guidance of materials in order to be learnt or needed; including language categories, branches, and language rules. In addition to that, beliefs and thoughts affect learners’ behaviors and attitudes in general, and vocabulary learning in specific (Easterbrook, 2013). Vocabulary learning is considered to be on top as being an indefinite component of language, Alqahtani (2015) approves by suggesting that the vocabulary system does not follow fixed rules, unlike other language components. Hence, language learners should take some strategies into account in line with language exposure; which is considered to be crucial for language learners to adapt the communicative and authentic part of language. Language exposure is reviewed as important as language instruction in SLA and EFL fields. Viera (2017) emphasizes the importance of language exposure by picturing the https://orcid.org/0000-0002-5404-1313 https://orcid.org/0000-0001-9741-0759 mailto:reem.amj94@gmail.com mailto:yesimkeslidollar@gmail.com https://doi.org/10.14744/felt.2019.00001 Jaber, R., A. & Keşli Dollar, Y. / Focus on ELT Journal, 2019 1(1) 44 Focus on ELT Journal (FELT) foreign language learning as process of changes the learners constantly experience. To illustrate, Ellis (2002) comments on language exposure, suggesting that learners have to meditate language in order to discover it. The idea thus sheds the light on the necessity that language learners have to continuously observe the language. Accordingly, the noticing hypothesis calls for the necessity of language observance. The noticing hypothesis runs from Schmidt’s experiences as a language learner of Portuguese, when Schmidt realized that linguistic forms begin to make forms and structures by their own (as cited in Iwanaka, 1998). One rich source that has eased and opened up for several methods and improvements in education is technology. Hence, technology is enlarging to become both; a rich source and a tool to boost language education and acquisition (Dovedan, Seljan, & Vučković, 2002). The main debate here revolves around the power of multimedia and audiovisual in particular, as it is believed to strengthen the acquisition of language components; including reading, writing, speaking, and listening. For instance, Gibriel (2017) suggests that language exposure as in incidental vocabulary learning; whether through reading, listening and/ or watching videos, promotes language acquisition and learning. Correspondingly, this study investigates bimodal subtitling or same-language subtitling as an effective method in language acquisition. Subtitling is defined by number of researchers differently. For example, subtitles or captions are defined by Luyken et al. (1991) as “condensed written translations of original dialogue which appear as lines of text, usually positioned towards the foot of the screen. Subtitles appear and disappear to coincide in time with the corresponding portion of the original dialogue” (as cited in Kuo, 2014, p. 62). Whereas Cintas & Remael (2014) identify subtitling as “a translation practice”, which includes written data and dialogue of speakers as the soundtrack (p. 8). As for Schroter (2005), subtitling is a form of translation called “additive translation”; featured with gestures, expressions, and utterances conveyed to captions (as cited in Zarei & Rashvand, 2011, p. 619). Whereas in Turkey, there are recent turnouts on using videos with subtitles in teaching English. Yıldız (2017) believes that videos with or without subtitles are implemented more often, in order to enlighten search on subtitling and vocabulary learning. According to article studies used in this paper, other countries as Iran, Norway, and Brazil showed major interest in experiments and research on different kinds of subtitles in relationship to different components of language. This indeed indicates the importance of subtitling which was found discover-worthy in different countries and settings. According to the research conducted, a good number of studies reported positive effect when it comes to subtitles and language learning. In general, subtitles were found to be advantageous in language learning in general, and vocabulary learning in specific. However, some studies or experiments were not very supportive in using subtitles due to multiple reasons. And some even suggested that subtitling has no use or effect on language learning (Naghizadeh & Darabi, 2015). Jaber, R., A. & Keşli Dollar, Y. / Focus on ELT Journal, 2019 1(1) 45 Focus on ELT Journal (FELT) This study however shall determine the effectiveness of bimodal subtitling of English, with its relationship with vocabulary learning among adult EFL learners. In order to do so, a meta-analysis was conducted and guided by the following research questions. 1. Is using bimodal subtitles effective in teaching vocabulary for adult EFL learners? 2. What are previous studies and results reported in terms of methodology? 3. What is the overall effectiveness of the studies on bimodal subtitling and vocabulary learning among adult EFL learners? Method As this study takes the form of meta-analysis, it is essential to define what meta-analysis is and what it mainly aims at as a research methodology. Meta-analysis study was first conducted by Karl Pearson in 1904. It is defined as a sum of quantitative results, based on several studies of a specific topic, to reach the quantitative effect size using multiple calculations and findings (Çoğaltay & Karadağ, 2015). Some of the advantages of conducting meta-analysis studies include the significance in practical sum of findings. In other words, meta-analysis studies provide evidence to support or reject a specific practice. Additionally, meta-analysis can determine the overall relationship for effect size in other approaches, thus researchers are encouraged to use data for evaluation of their literature. This study aimed to systemize previous literature which was conducted on bimodal subtitles along with vocabulary learning, from different settings and countries. As this meta-analysis explores the results of bimodal subtitles in relation to vocabulary learning, studies from different settings were concerned, within and outside the scope of Turkey. In order to collect data, different journal articles, master’s theses, and doctorate dissertations related to the topic were used for the study. During the search, articles and dissertations with treatments associated with other types of subtitles were eliminated, and bimodal subtitling articles published within the last 20 years were selected. After the initial search, 44 studies met the criteria of the effect of bimodal subtitles on language learning. However, only 10 studies compiled with the researcher’s strategy of the inclusion and exclusion criteria; indicating that two studies were believed to be efficient enough to avoid bias for a subject in meta-analysis studies (Valentine, Pigott, & Rothstein, 2010). Following the implementation of inclusion and exclusion criteria, the meta-analysis resulted in investigating 10 studies; which were conducted between 2010 and 2016, on English adult learners aged above 16. Table 1 provides distribution and analysis of the ten studies that were involved in the methodological analysis of the meta-analysis; which underlines the importance of empirical studies made on the topic. Jaber, R., A. & Keşli Dollar, Y. / Focus on ELT Journal, 2019 1(1) 46 Focus on ELT Journal (FELT) Table 1. List of 10 Studies Included in the Meta-Analysis and Their Sources Title of study Authors Type of Publication Year of publication Database Country of publication The effect of viewing subtitles videos on vocabulary learning Harji, Woods, & Alavi Journal Article 2010 EBSCO/ERI C Malaysia Effectiveness of using subtitled videos in learning English: a study on Iranian learners Alavi Master Thesis 2011 BAU Academic Library/Pro Quest Theses & Dissertations Malaysia The impact of watching subtitles animated cartoons on incidental vocabulary learning of ELT students Karakaş & Sarıçoban Journal Article 2012 ULAKBIM Turkey The effects of captioning texts and caption ordering on l2 listening comprehension and vocabulary learning Roohani, Domakani, & Alikhani, Journal Article 2013 ULAKBIM Iran The difference between the effectiveness of authentic and pedagogical films in learning vocabulary among Iranian EFL students Soltani & Soori, Journal Article 2015 ERIC Iran Authentic video and subtitles on English vocabulary enhancement Rizkiani Journal Article 2015 BAU academic library Indonesia The effect of movies with different types of subtitles on incidental English vocabulary learning of Iranian high school EFL learners Mardani & Najmbadi Journal Article 2016 ULAKBIM Iran Effects from using subtitled audiovisual material in second language acquisition York Master Thesis 2016 ULAKBIM Norway The impact of subtitles in second language acquisition Eye Master Thesis 2016 ULAKBIM Norway Watching subtitled films can help learning foreign languages Birulés- Muntané & Soto-Faraco Journal Article 2016 BAU Academic Search Complete (EKUAL) Spain Studies in this meta-analysis were given codes to establish some kind of mutual measurement criteria for the effect size which the bimodal subtitling can leave learners of EFL. In regard to the studies distribution and data, each study provided sample size, Jaber, R., A. & Keşli Dollar, Y. / Focus on ELT Journal, 2019 1(1) 47 Focus on ELT Journal (FELT) conditions, and measurement tools to evaluate the effect of its own empirical research. Stata 14 software was used for the evaluation, to test the relationship between the effect of bimodal subtitling as an independent variable and vocabulary learning as the dependent variable. As the ten studies vary in the sample size, this Meta-analysis program provided estimation of the effect size; in order to avoid errors in the results measured. Results To examine the effect of bimodal subtitling on vocabulary learning among adult EFL learners, 10 articles were reviewed for the study. In order to answer the research questions addressed, detailed analysis was given through meta-analytic tests performed by professional statisticians. It is significant to point here that the three research questions are jointly related, and are answered through the different tests applied in this analysis. Therefore, the results and discussion chapters are supposed to follow arguments and evidence coherently. Table 2 reflects the general meta-analytic results of the 10 studies with a summary of the significance level of each study. Table 2. Summary of the studies used in the Meta-analysis Type of Effect Effect Type Number of Studies Statistically significant Effect of Bimodal Subtitling on Vocabulary learning Positive Effect 6 Negative Effect 1 Statistically non-significant effect of Bimodal Subtitling on Vocabulary learning Positive Effect 2 Negative Effect 1 Table 3 presents results of the meta-analysis, which were used to examine the effect of bimodal subtitling on vocabulary learning among adult EFL learners of the studies. As observed from the analysis conducted on the articles reviewed, the Heterogeneity test reflected the dimension of variability among the articles reviewed. The heterogeneity was estimated by examining whether the treatment effect employed in the articles experiences variability within the studies. Since the test of heterogeneity was tested to be statistically significant as it is computed; p-value less than 0.05, (< 0.0001), then the random effect model was considered. From table 3, it can be observed that the Q-value and I-Squared statistics were used to determine the statistical heterogeneity among the articles being studied. The Q-value examined the variation between the treatment effect as well as the common effect exceeding the expectation by chance. The Q-statistic was computed as Q = 248.75, p <.0001. This showed statistical significance as it is an indication that the studies used in this paper to examine the effect were heterogeneous in nature. I-Squared was also used to measure the heterogeneity for the study, as it sought to measure the heterogeneity level and how it is presented among the studies. I-Squared ranges between 0 to 100%, and it examines the percentage of variations of the reviewed articles (Israel & Richter, 2011). From the analysis conducted, it was observed that the I-Squared was computed to be equal to 96.4%, which showed that the level of heterogeneity was considerable because I- Squared was greater than 75%. Jaber, R., A. & Keşli Dollar, Y. / Focus on ELT Journal, 2019 1(1) 48 Focus on ELT Journal (FELT) Table 3. Results of meta-analysis of the 10 studies conducted Model Estimates Fixed Random Effect size and 95% confidence interval Number of Studies 10 10 Point Estimate -1.207 -1.207 Lower Limit -1.870 -1.870 Upper Limit -0545 -0.545 Test of null (2-Tail) Z-value 14.52 3.57 P-value <0.0001 >0.0001 Heterogeneity Q 248.75 df(Q) 9 p-value <0.0001 I-Squared 96.4 Tau-Squared Tau-Squared 1.0693 Standard Error 3.57 Variance 11.289 Tau 1.0341 As this study comprised of 10 articles to examine the effect size of bimodal subtitling and vocabulary learning, the main focus was the mean difference in the vocabulary skill of the students; who were included in the treatment after being exposed to the bimodal subtitling. The analysis focused on the standardized mean difference (SMD) by estimating the random effect model. The random effect model considered in this study indicated that there was a significant effect of bimodal subtitling on vocabulary learning among adult EFL students. This was as observed from the standardized mean difference (SMD) computed by the meta-analysis considered in the study (SMD) = -1.207; z = 3.57, p < 0.0001 (See Table 4). Table 4. Summary of SMD estimate across articles with 95% Confidence Interval Study SMD [95% Conf. Interval] % Weight J. Birulés- & Muntané -0.330 -0.690 0.031 10.36 Madhubala et al -1.245 -1.561 -0.929 10.43 Somayeh & Afshin -0.691 -1.213 -0.170 10.02 Ali & Arif -0.263 -0.693 0.166 10.23 Siska Rizkiani -6.249 -7.323 -5.175 8.34 Henrik Eye 0.240 -0.367 0.847 9.81 Ali Roohani et al -1.003 -1.211 -0.795 10.57 Erlend Urkedal 0.739 0.165 1.313 9.89 Mahdi & Abedin -3.554 -4.132 -2.977 9.88 Zhinoos -0.622 -0.918 -0.326 10.46 D+L pooled SMD -1.207 -1.870 -0.545 100.00 D+L pooled SMD -1.207 -1.870 -0.545 100.00 To assess the small-study reporting bias in the meta-analysis, a contour-enhanced funnel plot was used to illustrate the statistical significance of the study effect. See figure 1 as a graph on the significance of this study’s effect estimated. The estimates displayed raised concerns that small-study effects; as given in this analysis, question the correct Jaber, R., A. & Keşli Dollar, Y. / Focus on ELT Journal, 2019 1(1) 49 Focus on ELT Journal (FELT) interpretation of the overall effect. To explore the apparent associations between the effect size and the study size, graphical approaches and statistical tests were employed. The funnel plot presented can be interpreted to be asymmetric, which indicates that smaller studies tend to give solid results emphasizing the effect of bimodal subtitling on vocabulary learning among adult learners of English. The plot revealed that smaller studies were not only found in the areas of statistical significance given by the shaded areas, but also in the areas of non-significance which is given by the non-shaded areas. Therefore, the level of asymmetry might have been caused by several factors and not solely by publication bias. Figure 1. Statistical significance of the study effect All the calculations above assessed that despite having a small meta-analysis study; comprising of 10 studies, the number of studies used were enough to claim the measurement of the addressed effect. In other words, the plot provided evidence of the asymmetry of study to have used sufficient number of studies, to find an overall significant effect of bimodal subtitling on vocabulary learning among adult EFL learners. Discussion As mentioned earlier, the main aim of this meta-analysis study was to evaluate the effectiveness of implying new interventions in English Education. The first goal of this study explored to what extent bimodal subtitles would be advantageous in acquiring new language terminology in the language learning process. To answer this question, each one of the 10 studies was analyzed and interpreted in terms of results and findings. Among 10, 6 studies were to provide statistically positive significant influence of the effect of bimodal subtitling on vocabulary recognition (Alavi, 2011; Birulés-Muntané & Soto-Faraco, 2016; Mardani & Najmbadi, 2016; Rizkiani, 2015; Soltani & Soori, 2015; York, 2016). On the other hand, a study reported bimodal subtitling has statistically significant negative effect Jaber, R., A. & Keşli Dollar, Y. / Focus on ELT Journal, 2019 1(1) 50 Focus on ELT Journal (FELT) on vocabulary learning of English students (Harji et al., 2010). Some studies reported non- significant change after implying bimodal subtitling on students, with one of negative insignificant effect (Roohani et al., 2013), and two with an estimation of positive insignificant effect (Eye, 2016; Karakas & Sariçoban, 2012). There are many research studies made on different types of subtitles in accordance to different aspects of language in general, and vocabulary learning in particular. During the literature search, it was found that subtitling is enlarging by interests of language researchers and instructors. This does not necessarily mean effectiveness in language aspects only; but also includes motivation, anxiety, and important components of the learning process. Regardless, some other research studies; that did not meet the inclusion and exclusion criteria, were not included in this meta-analysis study but did support our hypothesis of the effectiveness of bimodal subtitling on vocabulary learning. Some studies suggested that bimodal subtitling does not have any implications or effects at all. As for the results of this meta-analysis, the 10 studies together reported an overall positive effect of bimodal subtitles on vocabulary learning. The results supported the assumption of the significant relationship between bimodal subtitling and vocabulary learning, which in turn supported our hypothesis. The random effect indicated statistically significance from the standardized mean difference (SMD) = -1.207, p<0.0001. This is after considering the heterogeneity level of the studies, which makes the data and meta-analysis reliable for publishing. Conclusions There is still a need for wider research with more varieties in the samples. It is especially that some studies were eliminated if no effect or low effect size was found in the study. In addition, this study is conducted on papers that were collected by the researcher. Some other papers were unavailable or believed to be unreliable enough, that the researcher had to eliminate from the study. Other studies were eliminated as they did not meet criteria of the age of the participants. Other limitations of the study included the impossibility to find studies which used same test and/or data measurements. Also, researchers criticized meta- analysis due to the potentiality of error, bias, and effort needed which is considered problematic. Additionally, studies with poor methodological quality can be confusing, as well as using multiple findings of the same study for the analysis (Shelby & Vaske, 2008, p. 105-106). Following the limitations during the search of this study, direction for further studies was made. Future work is always a good idea to draw the attention and interest on a specific topic. As the studies of subtitles in language learning open up for new questions to be investigated by language researchers, studies in specialized areas in language education can be further addressed. For example, fields as in CLIL (Content and Language Integrated Learning) and ICT (Information and Communication Technology) in Education can be integrated with bimodal subtitling in language education. Moreover, subtitling can be a topic of interest in relation to language awareness and analysis, and how incidental language learning is linked to the language awareness process. Additionally, exploring different types of subtitles on different aspects of language can be targeted as well. Jaber, R., A. & Keşli Dollar, Y. / Focus on ELT Journal, 2019 1(1) 51 Focus on ELT Journal (FELT) Qualitative data can be collected through the perceptions of students and teachers; hence implications can be drawn in guidance with the recommendations. In the present study, the results met the expectations of the effectiveness of bimodal subtitles on vocabulary learning of English adult learners. However, it is crucial to apply this kind of studies on different levels of participants, in order to enlarge studies and draw conclusions in accordance. This is achievable by conducting meta-analysis studies on different addressed groups, settings, and with different comparisons of subtitles in Turkey and abroad. Disclosure Statement No potential conflict of interest was reported by the authors. References Alavi, Z. K. (2011). Effectiveness of using subtitled videos in learning English: A study on Iranian learners (Doctoral dissertation, Multimedia University (Malaysia). Alqahtani, M. (2015). The importance of vocabulary in language learning and how to be taught. International Journal of Teaching and Education, 3(3), 21-34. Birulés-Muntané, J., & Soto-Faraco, S. (2016). Watching subtitled films can help learning foreign languages. PloS one, 11(6), e0158409. Çoğaltay, N., & Karadağ, E. (2015). Introduction to meta-analysis. In Leadership and organizational outcomes (pp. 19-28). Springer, Cham. Cohen, J. (2008). Listening to learn: boosting vocabulary with interactive activities. Online Submission. Dovedan, Z., Seljan, S., & Vučković, K. (2002, January). Multimedia in foreign language learning. In Proceedings of the 25th International Convention MIPRO (pp. 72-75). Easterbrook, R. M. (2013). The process of vocabulary learning: Vocabulary learning strategies and beliefs about language and language learning (Doctoral dissertation, University of Canberra). Eye, H. (2016). The impact of subtitles in second language acquisition: an experimental study on Norwegian upper secondary students (Master's thesis, NTNU). Gibriel, A. M. S. A. (2017). Effectiveness of learning vocabulary through listening to TV English news stories of the BBC World a case study of 4th year college of languages students majoring in English as a foreign language at Sudan University of Science and Technology for the academic year 2017 (Doctoral dissertation, Sudan University of Science and Technology). Harji, M. B., Woods, P. C., & Alavi, Z. K. (2010). The effect of viewing subtitled videos on vocabulary learning. Journal of College Teaching & Learning (TLC), 7(9), 37-42. Israel, H., & Richter, R. R. (2011). A guide to understanding meta-analysis. Journal of Orthopaedic & Sports Physical Therapy, 41(7), 496-504. Iwanaka, T. (1998). Roles of noticing in English language learning : A literature review. Kagawa University Journal, 8, 53-67. Karakaş, A., & Sarıçoban, A. (2012). The impact of watching subtitled animated cartoons on incidental vocabulary learning of ELT students. Teaching English with Technology, 12(4), 3-15. Kuo, S. Y. (2014). Quality in subtitling: Theory and professional reality. Mardani, M., & Najmabadi, A. (2016). The effect of movies with different types of subtitles on incidental English vocabulary learning of Iranian high school students. Research in English Language Pedagogy, 4(1), 52-63. Naghizadeh, M., & Darabi, T. (2015). The impact of bimodal, Persian and no-subtitle movies on Iranian EFL learners’ L2 vocabulary learning. Journal of Applied Linguistics and Language Research, 2(2), 66-79. Rizkiani, S. (2015). Authentic video and subtitles on English vocabulary enhancement (A Quasi- Experimental Study in One Private College in Bandung). ELTIN Journal, Journal of English Language Teaching in Indonesia, 3(2), 91-96. Roohani, A., & Domakani, M. R. (2013). The effects of captioning texts and caption ordering on L2 listening comprehension and vocabulary learning. Applied Research on English Language, 2(2), 51-64. Shelby, L. B., & Vaske, J. J. (2008). Understanding meta-analysis: A review of the methodological literature. Leisure Sciences, 30(2), 96-110. Jaber, R., A. & Keşli Dollar, Y. / Focus on ELT Journal, 2019 1(1) 52 Focus on ELT Journal (FELT) Soltani, S., & Soori, A. (2015). The difference between the effectiveness of authentic and pedagogical films in learning vocabulary among Iranian EFL students. Advances in Language and Literary Studies, 6(1), 199-202. Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010). How many studies do you need? A primer on statistical power for meta-analysis. Journal of Educational and Behavioral Statistics, 35(2), 215-247. Viera, R. T. (2017). Vocabulary knowledge in the production of written texts: a case study on EFL language learners. Revista Tecnológica-ESPOL, 30(3), 89-105. Yıldız, T. A. (2017). The effect of videos with subtitles on vocabulary learning of EFL learners. International Journal of Humanities and Social Sciences, 7, 125-130. York, E. U. (2016). Effects from using subtitled audiovisual material in second language acquisition: An experimental study in a second language classroom in Norway (Master's thesis, NTNU). Zarei, A. A., & Rashvand, Z. (2011). The effect of interlingual and intralingual, verbatim and nonverbatim subtitles on L2 vocabulary comprehension and production. Journal of Language Teaching and Research, 2(3), 618-625. Copyrights Copyright for this article is retained by the author(s), with first publication rights granted to the Journal. This is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (CC BY-NC-ND) (http://creativecommons.org/licenses/by-nc-nd/4.0/).