Höl, D. (2022). A Bibliometric Analysis of Twenty-eight Years on ‘Assessing Pronunciation in English as a Second/Foreign Language Classes’ (1993-2021). International Online Journal of Education and Teaching (IOJET), 9(4), 2057-2075. Received 19.07.2022 Revised version received : 29.09.2022 Accepted : 30.09.2022 A BIBLIOMETRIC ANALYSIS OF TWENTY-EIGHT YEARS ON ‘ASSESSING PRONUNCIATION IN ENGLISH AS A SECOND/FOREIGN LANGUAGE CLASSES’ (1993-2021) Research article Devrim Höl https://orcid.org/0000-0001-5151-2581 Faculty of Education Pamukkale University, Denizli, Turkey devrimh@pau.edu.tr Biodata: Devrim HÖL is an Assistant Professor in Department of Foreign Language Teaching at Pamukkale University, Faculty of Education. His study areas are testing, evaluation, assessment in second/foreign language teaching, educational psychology. Copyright © 2014 by International Online Journal of Education and Teaching (IOJET). ISSN: 2148-225X. Material published and so copyrighted may not be published elsewhere without written permission of IOJET. mailto:devrimh@pau.edu.tr Höl 2058 A BIBLIOMETRIC ANALYSIS OF TWENTY-EIGHT YEARS ON ‘ASSESSING PRONUNCIATION IN ENGLISH AS A SECOND/FOREIGN LANGUAGE CLASSES’ (1993-2021) Devrim Höl devrimh@pau.edu.tr Abstract This bibliometric study examines the characteristics of the overall research trends, patterns of productivity, and publications on “assessment in second language pronunciation”. Bibliometric data were retrieved from Web of Science (WoS on 1 September 2021 and the results of the study reveal that the first publication appeared in 1993 and, during the period of 28 years, there have been 118 publications between 1993 and 2021 in total. It was found that studies in this field have increased in recent years. The publications include articles and proceeding papers written by 2.31 authors per publication. The most cited document received 139 citations. It was also discovered that the most frequently used word is intelligibility and the trending topic is pronunciation. As for the affiliations, the most productive university is Concordia University in Canada. In the following headings, detailed information is discussed in detail. Keywords: Bibliometric analysis, biblioshiny, second language education, assessing pronunciation 1. Introduction Pronunciation is an important element of spoken language, and it is usually used as one of the most vital components in assessing speaking skills in teaching English as a second language. However, there is enough evidence that, until lately, the pronunciation was disregarded in L2 language study, instruction, and evaluation (Baker, 2013; Derwing & Munro,2009) despite its vital role in learning a foreign/ second language. As it has an important role in communication skills, it needs to be assessed directly owing to its critical role to understand and to be understood by others (Canale & Swain, 1980; Bachman, 1990; Bachman & Palmer, 1996; Luoma, 2004). Pronouncing in the target language does not have to be perfect but learning an “adequate” level of English pronunciation is certainly a target in learning English. However, both teaching and assessing pronunciation may appear troublesome to teachers and it is also challenging for learners because it is dependent on many factors including learners’ effort, psychomotor, cognitive, and affective factors (Pennington & Richards, 1998). In addition, it can be very difficult when compared with other fields of language learning, and the differences between first (L1) and second language (L2) may play a crucial role in mastering pronunciation for learners (Spring & Tabuchi, 2021). All those may be an explanation why assessing pronunciation is neglected in communicative language instruction (Isaacs, 2014). According to studies, many raters lack pronunciation expertise and are unsure how to evaluate this component. According to Levis (2005), the major problem in assessing pronunciation is that nativeness and intelligibility are seen as the two most important criteria to the assessment of pronunciation. According to Pennington and Richards (1998), although spoken language has been widely examined ever since the 1950s, there seems to be little consideration given to reliability and validity focusing specifically on pronunciation, and thus, existing pronunciation measures within standardized tests have many weaknesses (Harding, 2017; Levis, 2006; Trofimovich & Isaacs, 2017). However, the growing mailto:devrimh@pau.edu.tr International Online Journal of Education and Teaching (IOJET) 2022, 9(4), 2057-2075. 2059 interest in communication skills, mostly speaking in the target language, led stakeholders including researchers, teachers, and linguists to focus on assessing pronunciation, which has been regarded as one of the most influential sub-skills. However, there is no doubt, it’s a difficult field to teach and evaluate, as a result of these challenges, it is mostly neglected in classrooms due to lack of time or the instructors not feeling competent to teach pronunciation (Nagle et al., 2020). On the other hand, there are some suggestions for pronunciation teaching and assessment such as computer-assisted pronunciation teaching (CAPT) and self-assessment of pronunciation proficiency. As another suggestion, self-assessment may increase learners' motivation to improve their second language pronunciation (Luis Luchini& Mariel Ferreiro, 2018). To summarize, it is evident that there is a great need to use linguistically meaningful measures of second language proficiency based on perceptual cues used by humans to evaluate pronunciation. (Graham et al., 2017). In addition, it is also interesting to note that there have been no bibliometric studies to examine and search for research trends in assessing pronunciation in the EFL context. Bibliometrics supply the recent trends for researchers, teachers and institutions in any field. It also yields findings by using search keywords and related fields in a specific period of time decided by researchers, so this recent but a vital tool is assumed to provide important findings for all stakeholders. As bibliometric studies offer researchers quantitative aspects for recent research trends in a specific field (Moher et al., 2009), this study will offer insights on the research trends in exploring assessing pronunciation in the EFL environment. It will also pave the way for researchers to follow the research trends in general and specific in the above- mentioned field. Thus, we examined the studies’ content and reviewed bibliometric analysis on the grounds of our findings from Web of Science (WoS), and tried to answer the following research questions: 1. What are the publishing and citation preferences on assessing pronunciation in the EFL context? 2. Which countries, institutions, and researchers are the most prolific ones in assessing pronunciation in the EFL context? 3. Which journals do researchers prefer to publish their studies in assessing pronunciation in the EFL context? 4. What are the most preferred keywords and trend topics in assessing pronunciation in the EFL context? 1.1 A Comparison between Web of Science (WoS) and Scopus Databases In the literature, it was found that Web of Science (WOS) and Scopus are the most widely accessed databases in many scientific domains for conducting a systematic review (Guz&Rushchitsky, 2009). From the historical perspective, over a long time, WOS from Thomson Reuters (ISI) was the sole citation database and publication that covered all areas of research. Nonetheless, Elsevier Science created the Scopus database in 2004, and it has gradually become a viable alternative to WOS (Vieira & Gomes, 2009). As a result of the introduction of the new citation database, Scopus, scientific libraries must consider which citation database would best suit the needs of its users, and for them, the conflicts between the WOS and Scopus databases are considerable (AghaeiChadegani et al., 2013). This rivalry has resulted in improvements in the services they provide, and as the final step, many articles have recently reviewed WOS with Scopus breadth, properties, and citation analysis skills (Bakkalbasi, 2006; Burnham, 2006; LaGuardia, 2005; Deis& Goodman, 2005; Dess, 2006; Li et al., 2010). These WOS and Scopus comparison analyses show that both databases are constantly evolving and improving. Researchers also find that the considerable benefit of using Höl 2060 one of these two sources is dependent on and related to the subject's field, so some researchers suggest conducting a subject-specific research to determine which databases are best for certain areas or time periods (Bar-Ilan et al., 2007; Bakkalbasi et al., 2006; Neuhaus& Daniel, 2008). Prior comparisons of these two datasets, according to Lopez-Illescas et al. (2008), have not revealed a definite winner. Researchers claim that the benefit of one database over another is determined by what will be specifically studied, the academic field, and the time frame of the study. 2. Method The bibliometric method was used to discover the historical trends on pronunciation assessment in English as a second language. In general, Bibliometrics is the application of statistical analyses to articles, books, and other media of communication (Borgman, 1989). It encompasses a large variety of laws and methods (McBurney& Novak, 2002). In addition, when analyzed in-depth, it is an open-source science-mapping tool and is one of the most useful applications to import bibliographic data from SCOPUS, Web of Science (WOS), PubMed, Cochrane, through which bibliometric analysis can be performed and data matrices for co- citation, coupling, scientific collaboration analysis, and co-word analysis can be built (Aria &Cuccurullo, 2017). From the researchers’ perspective, Bibliometric methods estimate how much influence a certain research article has on future research generally by counting the number of times the article is cited (Cooper, 2015). After Garfield (1972) argued that the findings of citation analysis had immense potential for the management of library journal archives, many scholars have walked a long path. Apart from being the simplest and most straightforward indicator of a publication's influence (Milojevi et al., 2017), this statistic can also supply data regarding the influence and characteristics of existing articles, research groups, institutions, countries, and journals (Sangwal, 2013; Waltman, 2016). Since citation rates appear to correlate with expert ratings (Brito &Rodrguez-Navarro 2018), citations can be used to grade the significance of research findings. However, Haunschild and Bornmann (2016) stated that only standardized citation counts allow for comparing between different areas and periods of time. On the other hand, in order to analyze the downloaded data from Wos, Biblioshiny, which is a web-based app included in the bibliometrix package that analyzes the data downloaded from a bibliographic database, which is WoS in this study, is used. 2.1 Database Selection There are various databases including citations and search terms and queries. These databases may not only include journals but also books, reviews, and conference proceedings. In this study, we selected the Web of Science (WoS) database because it is considered globally and preferred by thousands of researchers. The supported databases in Bibliometrix are WoS and Scopus. In this study, we preferred the WoS database as it gives us global search opportunities, and through which, we believe the WoS database is the one used by academicians and researchers throughout the world, and we can reach the most available data. Web of Science (WoS) Core Collection including Science Citation Index Expanded (SCIE), Social Sciences Citation Index (SSCI), and Arts & Humanities Citation Index (A&HCI), are very well-known by the researchers and one of the most preferred indexes in academia (Liu et al. 2020). The data were converted into tables and graphs using the Bibliometrix R Package and Biblioshiny app. Analysis of bibliometric results begins with a basic description of the main statistics. Next, the investigation continues considering the indicators and information. In search results, we focused on general information about the articles, their abstracts and International Online Journal of Education and Teaching (IOJET) 2022, 9(4), 2057-2075. 2061 examined their subject matter as they provide the core of the data we were trying to unveil. Google Scholar was not used in this study because it doesn’t discriminate between academic citations and nonacademic citations. Finally, we could get enough data and ideas about publications of second language pronunciation assessment. The following types of information were collected from WoS such as titles, authors, keywords, and citations. 2.2 Search Terms/ Queries on Bibliometrics To retrieve the bibliographic data, first, we accessed the Web of Science website in order to investigate the bibliometric characteristics of publications about second language pronunciation. The relevant data were extracted through a query by using a search strategy including keywords, which are “assessing second language pronunciation”, “assessing foreign language pronunciation”, “assessing and evaluating second or foreign language pronunciation”, “evaluating second or foreign language pronunciation”. These terms were searched in “title”, “abstract” and “keywords” of the studies. The time span was not limited in this study, however, the researcher decided on what kind of documents to choose and decided on choosing research articles and conference proceedings. For the present study, the Prisma diagram strategy was applied, which is a Preferred Reporting Items for Systematic Reviews and meta-analyses (Moher et al., 2009). The main aim of the Prisma Statement is to help and support authors and researchers to improve the reporting of meta-analyses and systematic reviews (Aria &Cuccurullo, 2017). In addition, the search strategy developed by Aria and Cucurullo (2017) was handled, which includes six steps; 1. The WoS Sub-DBs were selected: Research articles and Conference Proceedings 2. All documents that contain the words “assessing second language pronunciation”, “assessing foreign language pronunciation”, “assessing and evaluating second or foreign language pronunciation”, “evaluating second or foreign language pronunciation” in the title, abstract, or in the keyword list were selected 3. The timespan was decided 4. Only documents in English were selected The detailed flow of the data gathering process is mentioned below: Höl 2062 Figure 1. The flow of information through the different phases of a systematic review (Moher, 2009) 2.3 Date of Data Extraction The meta-data were gathered on 1 September 2021 at Pamukkale University, Faculty of Education, Turkey. Publications were searched in WoS. A total of 118 documents including articles and proceeding papers were found using this search query. 2.4 Data Analysis Various bibliometric analysis tools were used to get the most useful data from the research articles, Firstly, all published papers indexed in WoS were detected with related keywords. Any kind of publishing related to assessing second language pronunciation was included in order to reach all data in WOS. We exported all documents as a plain text file from WoS. Secondly, through the R tools package, we ran the Biblioshiny (version 2.0) which shows us all information and statistics about publications. To open the Biblioshiny page, Rstudio was run and bibliometrix was installed from the tools section. we digited “library (bibliometrix)” and then “biblioshiny()”. In this way, the Biblioshiny web interface was opened automatically. And then, all documents we got from WoS are imported into Biblioshiny as a raw file. Lastly, we examined all the results one by one such as numbers, graphics, and charts utilizing MS Excel (V.18.0) Limitations This study included research articles on the WoS database. These findings of the present study were limited to WoS sub-categories related to assessing pronunciation in English as a second language. While there are many research contexts related to assessing pronunciation, International Online Journal of Education and Teaching (IOJET) 2022, 9(4), 2057-2075. 2063 in this study, it is limited to assessing pronunciation in the EFL context, and other fields of studies such as health were excluded from the study. No publication year filter was applied, no geographical or native language filters were applied for this study. 3.Findings 3.1 General Information and Historical Background The first aim of the present study was to find out an overall map of related studies from a general perspective. The findings reveal that 118 articles were published on assessing pronunciation in the EFL context between 1993 and 2021. It was found that the first publication about second language pronunciation assessment was published in 1993 in the journal Language Learning. The title of the first article related to assessing pronunciation in EFL context is “The Evaluation of Accent in the English of Dutchmen”. Table1 demonstrates the essential extracted information about the document types published. Table 1. General Information about the Document Types Description Results MAIN INFORMATION ABOUT DATA Timespan 1993:2001 Sources (Journals, Book, etc) 74 Documents 118 Average years from publication 5,78 Average citations per documents 9,788 Average citations per year pear doc 1,365 References 1 DOCUMENT TYPES Article 67 Article; book chapter 15 Article, early access 8 Article; proceedings paper 1 Proceedings paper 27 DOCUMENT CONTENTS Keyword plus (ID) 221 Author’s Keyword (DE) 359 AUTHORS Authors 273 Author Appearances 339 Authors of single-authored documents 24 Authors of multi-authored documents 249 AUTHORS COLLABORATION Single-authored documents 32 Documents per Author 0,432 Authors per Documents 2,31 Co-Author per Documents 2,87 Collaboration Index 2,9 Höl 2064 3.2 Second Language Pronunciation Publications It was also important for the present study to discover whether research in assessing second language pronunciation increases, stay stable or decrease, and if there is a change in the trend, in which year researchers were most prolific. Figure 2 demonstrates the annual publishing numbers of studies on assessing pronunciation in EFL context. This analysis period covers 28 years of scientific production, and statistics show an increase in publications over the years, and there is a critical spike in the number of publications appears in 2018 with 17 documents. The annual growth rate of studies is approximately 11%. These findings unveil that the number of studies increased slowly until 2005, however it has gained prominence and popularity after 2015. However, the most striking finding is that there is a jump between 2004 and 2019. . Figure 2. The Number of Documents Published per year. 3.3 Receiving Citations As an in-depth analysis, we tried to cover the citation analytics as it gives us the most preferred study. According to Pech and Delgado (2020), citation analytics is important in Scientometrics. It was discovered that the second language pronunciation assessment publications received citations from an average of 9.78 per study. The findings also reveal that Munro Mj is the most cited author (139 citations) and followed by Thomson RI (2015) (102 citations) which is the highest number of citations for articles. Table 2 shows the citation structure of second language pronunciation assessment publications. International Online Journal of Education and Teaching (IOJET) 2022, 9(4), 2057-2075. 2065 Table 2. The Citation Structure of Second Language Pronunciation Assessment Publications Paper Total Citations DOI Munro MI, 1999, LANG LEARN 139 10.1111/0023-8333.49.s1.8 Thomson RI, 2015, APPL LINGUIST 102 10.1093/applin/amu076 LinckJA, 2013, LANG LEARN 96 10.1111/lang.12011 Isaacs T, 2013, LANG ASSESS Q 72 10.1080/15434303.2013.769545 Saito K, 2017, APPL LINGUIST 52 10.1093/applin/amv047 Isaacs T, 2008, CAN MOD LANG REV 36 10.3138/cmlr.64.4.555 Harding L, 2012, LANG TEST 32 10.1177/0265532211421161 DlaskaA, 2008, SYSTEM 31 10.1016/j.system.2008.03.003 3.4 Countries and Universities As far as organizations are concerned, the top university is Concordia University in Canada with 9 publications. It is respectively followed by the University of Bristol with 7 publications. The other following universities are shown in Figure 3 with their publishing numbers. In addition, we tried to detect the most prolific countries in assessing pronunciation in English. Although many researchers have studied using citation analysis on comparing researchers, fields, institutions, and countries (Bornmann 2013; Fairclough &Thelwall 2015; Moed 2016; Radicchi& Castellano 2012; Rodrguez-Navarro & Brito 2018; Waltman 2016), very few research has been reported to examine on comparing citations of papers from variable databases. This type of comparison might be useful in various bibliometric systematic assessments related to the growth, development, and changes of a certain scientific field. In our examination about affiliations, we obtain the numbers of publications of the universities (The top 20 affiliations). Table 3. Countries’ Scientific Production Chart Region Freq Region Freq Region Freq USA 43 AUSTRALIA 4 OMAN 2 CANADA 25 GERMANY 4 ROMANIA 2 UNITED KINGDOM 24 INDIA 4 RUSSIA 2 CHINA 21 IRAN 4 SAUDI ARABIA 2 SPAIN 12 FINLAND 3 SWEDEN 2 FRANCE 10 SINGAPORE 3 ARGENTINA 1 PORTUGAL 10 SOUTH AFRICA 3 BAHRAIN 1 POLAND 8 AUSTRIA 2 ECUADOR 1 CHILE 7 DENMARK 2 ITALY 1 JAPAN 6 NETHERLANDS 2 QATAR 1 BRAZIL 5 NEW ZEALAND 2 SOUTH KOREA 1 Höl 2066 Figure 3.Most Relevant Affiliations 3.5 Authors’ Productivity On the grounds of information, the data from the WoS, 273 second language pronunciation assessment studies were published by the authors. The most prolific author is Isaac T. with 8 documents. It is interesting to notice that each article is written by two authors (2.87) and the collaboration index is 2,9. Documents had an average of 2.31 authors per publication. Table 4 and Figure 3 below give a summary of some authors and their productivity. 233 out of 273 studies were conducted by one author, showing that most authors have just one publication on assessing second language pronunciation. One author has eight studies and two authors have seven studies about the topic. Table 6 gives a summary of the productivity of authors. Table 4. Author Productivity Authors Articles Authors Articles ISAACS T 8 O'BRIEN MG 3 SAITO K 7 TSUNEMOTO A 3 TROFIMOVICH P 7 YOMA NB 3 TEPPERMAN J 4 BARAN-LUCARZ M 2 CROWTHER D 3 CHAN JYH 2 GALES MJF 3 CHEN NF 2 KENNEDY S 3 CHEN Y 2 MALININ A 3 DERWING TM 2 NARAYANAN S 3 ENGWALL O 2 International Online Journal of Education and Teaching (IOJET) 2022, 9(4), 2057-2075. 2067 Figure 3. Author Productivity 233 out of 273 studies were conducted by one author, showing that most authors have just one publication on assessing second language pronunciation. One author has eight studies and two authors have seven studies about the topic. Table 6 gives a summary of the productivity of authors. Table 6. Productivity of Authors Documents written Number of Authors 1 233 2 28 3 8 4 1 7 2 8 1 3.6 Most Influential Sources Another aim of the present study was to find out the sources in which assessing second language pronunciation studies were preferred and published by researchers. Articles have been published in different 74 sources. We obtained the sources listed in WoS, which is related to the second language pronunciation assessment with the number of publications. It was found that the most relevant source is Assessment in Second Language Pronunciation with 8 documents. Figure 4 shows the most influential sources in detail. Secondly, we tried to unveil Höl 2068 the source growth per year to compare the source and source growth. The annual source growth is illustrated in Figure 5. Figure 4. Sources of Publication Figure 5. Source Growth per Year 3.7 Major Themes and Most Frequent Words It was found in the study that the three most frequently used words are intelligibility (24 times), pronunciation (23 times), and judgment (22 times) are the mostly preferred keywords by the researchers. Figure 6 and 7 illustrates the most frequent words and top twenty themes in second language pronunciation assessment publications. As for the author’s keywords (DE), there are 359 keywords in total, which is illustrated in Figure 7. Researchers use multiple keywords in their articles, and it gives significant findings to determine the research trend. In addition, the most common title word is pronunciation with 43 occurrences. International Online Journal of Education and Teaching (IOJET) 2022, 9(4), 2057-2075. 2069 Figure 6. The Most Frequent Words and Top Twenty Themes Figure 7. Keyword Cloud on Assessing Pronunciation 3.8 Trend Topics per Year The analysis shows that the trend topics in researching assessing second language pronunciation varied throughout the years; while pronunciation, speech, and 2nd language were the trend topics in 2015, it shifted to intelligibility, judgments, foreign accent, experience and instruction in 2016. However, the major trend topics were fluency, perceptions and acquisition in 2017, and English, comprehensibility, language, proficiency, and accent in the following year. Figure 8 illustrates the most preferred trend topics by the researchers. Höl 2070 Figure 8. The Most Preferred Trend Topics 4. Conclusion and Discussion This bibliometric analysis tried to answer the questions about the publications, authors, citations, words, and also topics. In this study, the researcher examined second language pronunciation assessment publications using Web of Science. The study sought information on the articles, titles, abstracts, and also the phrases including “second language pronunciation assessment”, “assessing second language pronunciation”, “assessing foreign language pronunciation”, “assessing and evaluating second or foreign language pronunciation”, “evaluating second or foreign language pronunciation”. These terms were searched in “title”, “abstract” and “keywords” of the related studies in WoS.. This present study aims to contribute to the literature by unearthing two hidden fields in EFL context; assessment and pronunciation. Firstly, we can’t stand to say it is interesting to meet with a really low number of studies on this topic, it was really surprising for us. According to the statistics, there are a total of 118 publications in the WoS between 1993 and 2021 on the search query, which is “assessing pronunciation in EFL/ESL context”. We flaunt that this study is the first one exploring there are few studies on assessing second language pronunciation although learning and teaching a second or foreign language has taken huge steps in the last two decades, especially with the help of technology, it is really surprising to note that it is still a neglected field in second or foreign language education. As a subskill of one of the most important and productive skills, speaking, it is really noteworthy that the number of studies is inadequate. The first and the most important cause for this can be the general ignorance on assessing second language pronunciation. This may have various reasons. Firstly, it may seem gloomy to second/foreign language teachers and researchers; they may know and be aware of “what” to test, which is pronunciation in second language here, however, it may seem far away to them on “how” to test it. In the literature, it is still not clear how to assess the pronunciation, whether through computers, if so, how and through what kind of programs, or through a kind of human assessors, again, if so, through International Online Journal of Education and Teaching (IOJET) 2022, 9(4), 2057-2075. 2071 which steps and with which scale. This is a field that does not have a consensus among researchers. Some assess it through computers or some web tools, while others may have some hesitation on assessing through computers or web tools and may prefer to assess themselves as teachers or researchers, so this uncertainty may affect the number of studies conducted. It is hoped this study will give all stakeholders an idea to conduct some more research on assessing pronunciation in EFL/ESL context. From this perspective, it seems there are two questions to be answered by institutions: (1) Can a scale be developed to use in assessing second language pronunciation?, (2) How can in-service and pre-service EFL teachers be trained on assessing second language pronunciation? The other reason may be the native language of learners. This may also cause ambiguity. They have a native language that is really pronounced in a different way compared to the target language, which is English in this study. This may affect the quality and number of studies on assessing second language pronunciation. Secondly, our examination of citation patterns in second language pronunciation assessment publications reveals that the most cited publication has 139 citations and publications received citations from an average of 9.78 publications. From the analysis, we found that Isaacs T. is the author with the most articles. And the most frequently used word is intelligibility in publications. It shows that, rather than having native-like pronunciation skills for their students, researchers or/and teachers focus on the intelligibility level of pronunciation. It was also interesting that although researchers do not focus on perfectness in second language pronunciation, the number and the citation of research is really low, which is a sign for researchers to take urgent action in this field, and also a sign for a research field, which is neglected throughout the process. It was also revealed from the analysis that the most prolific university is Concordia University in Canada, and the majority of publications are from universities in Canada It is clear from the findings that Canada is a very important country to assess pronunciation as there are many newcomers to the country whether as students or immigrants and it is not surprising that it is the most fruitful country in this field. As a limitation, we focused only on Web of Science publications and their bibliometric analysis which limits the scope of our study. Investigating publications outside of WoS may provide another interesting picture of second language pronunciation assessment publications. Of course, these indicators are not the only indicators showing the quality of publications. Höl 2072 References AghaeiChadegani, A., Salehi, H., Yunus, M., Farhadi, H., Fooladi, M., Farhadi, M., & Ale Ebrahim, N. (2013). A comparison between two main academic literature collections: Web of Science and Scopus databases. Asian social science, 9(5), 18-26. 3(7). Retrieved from http://www.bio-diglib.com/content/3/1/7. Aria, M., Cuccurullo, C., & Aria, M. M. (2017). Package ‘bibliometrix’. Bachman, L. F. 1990. Fundamental considerations in language teaching. Oxford: Oxford University Press. Bachman, L. F., & Palmer, A. S. (1996).Language testing in practice.Oxford: Oxford University Press. Baker, A. (2013). Integrating fluent pronunciation use into content-based ESL instruction: Two case studies. Bakkalbassi, N., Bauer, K., Glover, J., & Wang, L. (2006). Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomedical Digital Libraries, Bar-Ilan, J., Levene, M., & Lin, A. (2007). Some measures for comparing citation databases. Journal of Informetrics, 1(1), 26-34. http://dx.doi.org/10.1016/j.joi.2006.08.001 Borgman, C. L. (1989). Bibliometrics and scholarly communication: editor's introduction. Communication Research, 16(5), 583-599. Bornmann, L. (2013). How to Analyze Percentile Citation Impact Data Meaningfully in Bibliometrics: The Statistical Analysis of Distributions, Percentile Rank Classes, and Top-Cited Papers. Journal of the American Society for Information Science and Technology, 64(3), 587–595. Brito, R., & Rodríguez-Navarro, A. (2018). Research assessment by percentile-based double rank analysis. Journal of Informetrics, 12(1), 315–329. Burnham, J. F. (2006). Scopus database: a review. Biomedical Digital Libraries, 3(1). http://dx.doi.org/10.1186/1742-5581-3-1 Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied linguistics, 1(1), 1-47. Cooper, I. D. (2015). Bibliometrics basics. Journal of the Medical Library Association: JMLA, 103(4), 217. Deis, L., & Goodman, D. (2005). Web of Science (2004 version) and Scopus. The Charleston Advisor, 6(3). Retrieved from http://www.charlestonco.com/comp.cfm?id=43 http://www.bio-diglib.com/content/3/1/7 http://dx.doi.org/10.1016/j.joi.2006.08.001 http://dx.doi.org/10.1186/1742-5581-3-1 http://www.charlestonco.com/comp.cfm?id=43 International Online Journal of Education and Teaching (IOJET) 2022, 9(4), 2057-2075. 2073 Derwing, T. M., & Munro, M. J. (2009). Putting accent in its place: Rethinking obstacles to communication. Language teaching, 42(4), 476-490. Fairclough, R., &Thelwall, M. (2015). More precise methods for national research citation impact comparisons. Journal of Informetrics, 9(4), 895–906. Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479. Graham, C., Buttery, P., & Nolan, F. (2017). Vowel characteristics in the assessment of L2 English pronunciation. Guz, A. N., &Rushchitsky, J. J. (2009). Scopus: A system for the evaluation of scientific journals. International Applied Mechanics, 45(4), 351-362. http://dx.doi.org/10.1007/s10778-009-0189-4 Harding, L. (2017). What do raters need in a pronunciation scale? The user’s view. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation assessment: Interdisciplinary perspectives (pp. 12–34). Bristol, UK and Blue Ridge Summit, PA: Multilingual Matters. Haunschild, R., &Bornmann, L. (2016). Normalization of Mendeley reader counts for impact assessment. Journal of Informetrics, 10(1), 62-73. Isaacs, T., &Trofimovich, P. (Eds.). (2016). Second language pronunciation assessment: Interdisciplinary perspectives. Multilingual Matters. Laguardia, C. (2005). E-views and reviews: Scopus vs. Web of Science. Library Journal, 15. Levis, J. M. (2005). Changing contexts and shifting paradigms in pronunciation teaching. Tesol Quarterly, 39(3), 369-377. Levis, J. M. (2006). Pronunciation and the assessment of spoken language. In R. Hughes (Ed.), Spoken English, TESOL and applied linguistics: Challenges for theory and practice (pp. 245–270). New York: Palgrave Macmillan. Li, J., Burnham, J. F., Lemley, T., & Britton, R. M. (2010). Citation Analysis: Comparison of Web of Science, Scopus, SciFinder, and Google Scholar. Journal of Electronic Resources in Medical Libraries, 7(3), 196-217. http://dx.doi.org/10.1080/15424065.2010.505518 Liu, W., Hu, G., & Tang, L. (2018). Missing author address information in Web of Science-An explorative study. Journal of Informetrics, 12(3), 985- 997. https://doi.org/10.1016/j.joi.2018.07.008 Lopez-Illescas, C., Moya-Anegon, F., &Moed, H. F. (2008). Coverage and citation impact of oncological journals in the Web of Science and Scopus. Journal of Informetrics, 2(4), 304-316. http://dx.doi.org/10.1016/j.joi.2008.08.001 Luis Luchini, P., & Mariel Ferreiro, G. (2018). Self-assessment and Negotiated Evaluation Among Teacher and Students in the Pronunciation Class: A Study http://dx.doi.org/10.1007/s10778-009-0189-4 http://dx.doi.org/10.1080/15424065.2010.505518 https://doi.org/10.1016/j.joi.2018.07.008 http://dx.doi.org/10.1016/j.joi.2008.08.001 Höl 2074 with Argentinean Learners. DIDACTICA-LENGUA Y LITERATURA, 30, 137- 156. Luoma, S. (2004). Assessing speaking. Cambridge: Cambridge University Press. McNamara, T. (1996). Measuring second language performance. Harlow, Essex: Pearson Education. McBurney, M. K., & Novak, P. L. (2002, September). What is bibliometrics and why should you care?. In Proceedings. IEEE international professional communication conference (pp. 108-114). IEEE. Milojević, S., Radicchi, F., & Bar-Ilan, J. (2017). Citation success index − An intuitive pair-wise journal comparison metric. Journal of Informetrics, 11(1), 223–231. Moed, H. F. (2016). Comprehensive indicator comparisons intelligible to non-experts: the case of two SNIP versions. Scientometrics, 106(1), 51–65. Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., &Prisma Group. (2009). Reprint— preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Physical therapy, 89(9), 873-880. Nagle, C., Sachs, R., &Zárate-Sández, G. (2020). Spanish teachers’ beliefs on the usefulness of pronunciation knowledge, skills, and activities and their confidence in implementing them. Language Teaching Research, 1362168820957037. Neuhaus, C., & Daniel, H. D. (2008). Data sources for performing citation analysis: an overview. Journal of Documentation, 64(2), 193-210. http://dx.doi.org/10.1108/00220410810858010 Pech, G., & Delgado, C. (2020). Assessing the publication impact using citation data from both Scopus and WoS databases: an approach validated in 15 research fields.Scientometrics, 125(2), 909-924. Pennington, M. C., & Richards, J. C. (1986). Pronunciation revisited. TESOL quarterly, 20(2), 207-225. Pennington, M. C., & Rogerson-Revell, P. (2019). Assessing Pronunciation. In English Pronunciation Teaching and Research (pp. 287-342). Palgrave Macmillan, London. Radicchi, F., & Castellano, C. (2012). A reverse engineering approach to the suppression of citation biases reveals universal properties of citation distributions. PLoS ONE, 7(3), e33833. Rodríguez-Navarro, A., & Brito, R. (2018). Double rank analysis for research assessment. Journal of Informetrics, 12(1), 31–41. Sangwal, K. (2013). Citation and impact factor distributions of scientific journals published in individual countries. Journal of Informetrics, 7(2), 487–504. http://dx.doi.org/10.1108/00220410810858010 International Online Journal of Education and Teaching (IOJET) 2022, 9(4), 2057-2075. 2075 Spring, R., &Tabuchi, R. (2021). Assessing the Practicality of Using an Automatic Speech Recognition Tool to Teach English Pronunciation Online. Journal of English Teaching through Movies and Media, 22(2), 93-104. Trofimovich, P., & Isaacs, T. (2017). Second language pronunciation assessment: A look at the present and the future. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation assessment: Interdisciplinary perspectives (pp. 259– 266). Bristol, UK and Blue Ridge Summit, PA: Multilingual Matters. Vieira, E. S., & Gomes, J. A. N. F. (2009). A comparison of Scopus and Web of Science for a typical university. Scientometrics, 81(2), 587-600. http://dx.doi.org/10.1007/s11192-009-2178-0 Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10, 365–391. http://dx.doi.org/10.1007/s11192-009-2178-0