Zorica Dodevska:tipska.qxd 1 Zorica Dodevska Institute for Artificial Intelligence Research and Development of Serbia, Novi Sad, Serbia Management: Journal of Sustainable Business and Management Solutions in Emerging Economies Forthcoming *Corresponding author: Zorica Dodevska, e-mail: zorica.dodevska@ivi.ac.rs Research Question: Bibliometric analysis of the literature on emerging markets aims to recognise the relationship between traditional metrics (i.e., citations) and alternative metrics (i.e., altmetrics) in this field. Motivation: This study is motivated by existing difficulties in measuring both the quality and societal effects of research papers. There are divided opinions among critics considering whether traditional metrics represent similar measures in comparison with altmetrics, or whether scientific and public attention are principally different categories. The study focuses on a specific field – emerging markets, considering that both opinions of scientists and the crowd are essential for driving appropriate societal changes. Understanding both metrics’ nature is crucial for their proper usage to support sustainable development efforts and link research project evaluation and financing in the field. Idea: The main idea is to examine the relationship between the number of citations and altmetric indicators and to determine the extent of overlapping individual papers/journals in the sample with the best ranking results in both categories. Data: The study uses data about articles on the topic of “emerging markets” exported from the Web of Science (WoS) database and expanded with altmetric indicators (either from providers Altmetric.com or PlumX, depending on the publisher). There are 3996 valid records collected during November 2019. Tools: This paper considers only altmetric indicators common to both providers. Analyses on the relationship of the number of citations and altmetric indicators (Pearson correlation and percentage of overlapping of top-ranked articles/journals) are performed across the entire sample or only considering selected records. Findings: Obtained results indicate that there is generally no significant relationship between observed metrics in circulation (except in the case of Mendeley). The same conclusion is reached by looking at 100 prominent individual papers in both categories in terms of overlapping, while the top 100 journals indicate a better overlap. The role of altmetrics is not to predict citation. Societal and academic impacts are rather different categories, and it is reasonable to consider the purpose of both metrics for initiating/enhancing development in emerging markets. Contribution: This expanded bibliometric study provides valuable information and orientation for researchers, journals, and academic institutions interested in emerging market growth, evaluation, and financially supporting related research projects. Keywords: articles on emerging markets, citations, altmetrics, Web of Science (WoS), Altmetric.com, PlumX JEL Classification: A23, I23 An Expanded Bibliometric Study of Articles on Emerging Markets DOI: 10.7595/management.fon.2022.0003 Abstract: 1. Introduction Emerging markets make a lot of the world economy, even 42 percent of the global GDP and 55 percent of the world population (Cavusgil, 2021). The following countries initially constitute emerging markets: Brazil, Russia, India, and China (O’Neill, 2001), while the acronym BRICS also includes South Africa (Westra, 2017). Depending on the source, emerging market economies comprise many other countries (e.g., Mexico, Turkey, etc.). Tending to become developed economies, they are on a ‘seesaw’ where fast growth is on one side and high risks on the other. Emerging markets present interdisciplinary research fields (Sudhir et al., 2015) and topics of interest of the general public (including, for example, potential investors, governments, com- panies, media, etc.). Bibliometric analysis of articles that deal with emerging-market themes can provide feedback information pri- marily in terms of channels and frequency of their past usage among scientists and the broader population. Therefore, this activity potentially can contribute to cognition regarding whether and to which extent scien- tific efforts influence driving changes. However, it is questionable whether and how the social value of re- search can be correctly perceived. Some of the open issues are non-standardised methods for measuring social impact (Tahamtan & Bornmann, 2020), initially unequal exposure of papers (for example, because of the Matthew effect – Drivas & Kremmydas, 2020), the way of interpreting bibliometric data (Szomszor et al., 2021). In traditional bibliometrics, researchers are at the point of confrontations of the following activities that have become part of academic life (Figure 1a): researchers face pressure to publish their work (the so-called “publish or perish” practice is a reality – Grimes et al., 2018); their publications pass academic evaluation through the number of acquired citations; all previously mentioned forces expressed quantification of re- search results. These imposed activities create constraints for researchers. Figure 1b shows the position of a researcher in measuring alternative metrics. As papers are available in the digital form today, their distri- bution, and therefore usage, go online. This opportunity allows daily tracking of public attention and alter- native metrics – known under the umbrella term ‘altmetrics’ (Haustein, 2016; Tunger et al., 2018). Researchers are encouraged to post their results on academic networks (e.g., ORCID, Publons, Research- Gate, etc.) to make them more visible. Numerous social media platforms enable their users to share pa- pers, comment on them, and discuss them. However, the potential benefits of new metrics (i.e., altmetrics) and their relationship with citations are not clear (Barthel et al., 2015; Costas et al., 2015). Figure 1: Position of a researcher: 1a. Traditional metrics; 1b. Alternative metrics The task of this research is to determine relations between citations and altmetrics in the literature of emerg- ing markets: • Can alternative metrics successfully predict article citations? The potential possibility of evaluating re- search projects based on altmetric indicators is significant for early-stage projects that do not yet have a chance to validate results through the academic community’s citations. • Do scientific and public attention converge in terms of overlapping top-ranked articles/journals? The answer provides an insight into the possibilities of scientists and the crowd to jointly initiate changes to achieve sustainable development in emerging economies. After introducing concepts of traditional bibliometrics and altmetrics (section 2), the paper describes the re- search method conducted to determine if there is a statistically significant correlation between traditional and non-traditional metrics in relevant articles on emerging markets (section 3). According to both metrics, section 4 contains obtained results – correlations and percentages of overlapping top articles/journals. The obtained results are discussed compared to other similar studies (section 5), while the conclusion is in section 6. 2. Traditional Bibliometrics and Altmetrics Prior to introducing the research papers’ metrics, we first need to formulate the related terminology and cri- teria consistently. Issues persist when distinguishing the following terms in the literature: outcome, output, impact, and effects of research papers (Bornmann & Haunschild, 2019). Figure 2 defines these terms as well the whole process from the outcome to effects. 2 Zorica Dodevska Forthcoming Figure 2: Measuring what? Outcome, output, impact, and effects of research papers (according to Bornmann & Haunschild, 2019) As an established measure of academic impact, citations only partially reflect research quality through the impact (Martin & Irvine, 1983; Bornmann & Haunschild, 2019) and relevance (Aksnes et al., 2019) in the scientific community, while they do not cover research importance and correctness. Besides, the connection between societal impact and the quality of research is less clear. Therefore, measuring the quality of re- search papers is an additional problem. 2.1 Traditional bibliometrics Traditional bibliometrics focuses on citation-based metrics for ranking journals by their 2-years impact fac- tor (IF). Other bibliometric indicators are also in use, such as 5-year IF, immediacy index, article influence score, source normalised impact per paper (SNIP), and SCImago journal rank (SJR) (Subochev et al., 2018). Further, there is the h-index for ranking scientists/researchers/academics and other author-based metrics (Da Silva & Dobránszki, 2018). Ranking academic institutions, such as universities (Johnes, 2018), is conven- tional, for which highly cited researchers are one of the indicators for academic ranking (Liu & Cheng, 2005). Traditional bibliometrics has established usage and recognised advantages. For example, according to Karanatsiou et al. (2017), it shows the actual impact and has a strong theoretical background. However, there are also some shortcomings: • Citation-based indicators reflect academic impact while funding agencies are interested in more ex- tensive types of effects (for example, patent metrics) – Thelwall et al. (2016). • Traditional bibliometrics imply long-time citation accumulation and, therefore, several years for eval- uation (Priem & Hemminger, 2010; Thelwall et al., 2016). • There are biases in different aspects of self- and team-citation (Ioannidis, 2015) and the need for fair ranking schemes for multi-author publications (Vavryčuk, 2018). • There are dependencies from citation databases (e.g., WoS, Google Scholar, Scopus, Microsoft Ac- ademic) that may provide different coverage via citations (Martín-Martín et al., 2020) and different re- sults (i.e., number of citations) for the same publication. This paper uses the WoS database for measuring citations, as many other studies (for example, Fang et al., 2020; Hou et al., 2020; Heydari et al., 2019; Banshal et al., 2018). Verily, WoS is one of the prime scientific citation searches and plat- forms with a long tradition (Li et al., 2018). • It is questionable whether highly cited papers guarantee remarkable scientific contributions, as we tend to expect. Citation behaviour can also be motivated by non-scientific reasons (Bornmann & Da- niel, 2008). Moreover, papers can have high citation scores on account of criticism or useful reviews instead of state-of-the-art research (Thelwall et al., 2016). • There are systematic biases in the sense of differing citation norms for different fields, so it is advis- able to compare only articles from the same area (and preferably the same year) if normalisation is not an option (Thelwall et al., 2016). • Citations are related to subsequent research project evaluations, and they are not applicable in the project’s early phases. Therefore, additional metrics are required. 2.2 Altmetrics Altmetrics are alternative metrics that, depending on a provider, count the number of mentions (e.g., policy sources, news, blogs), social media activities, captures from reference managers and social bookmarking, etc. Altmetric providers can include data about citations (for example, Altmetric.com provides ‘Dimensions’ which do not contribute to the Altmetric Attention Score, while PlumX observes several citation databases and takes the maximum recorded value for the number of citations). Each provider has some specifics. Alt- metric.com has defined weighting coefficients for each type of mention, as well as provides demographic data. PlumX does not give a final score but tracks separate metrics by categories (also, it includes the ‘Usage’ category). Crossref Event Data (CED) collects events from a specific list of data sources. Detailed information about these three providers’ differences by individual sources or separate metrics leads to con- 3 Management: Journal of Sustainable Business and Management Solutions in Emerging Economies Forthcoming clusions that there is a seriously higher number of articles mentioned on Twitter, according to Altmetric.com, in comparison with CED and PlumX (Ortega, 2018). Problems of heterogeneous sources, dependencies from a data provider, and the quality of provided altmetric data are notable (Haustein, 2016). Some authors express concerns about various research metrics without standardised measurement rules (e.g., Buttlier & Buder, 2017). Altmetrics measure research impact per day, not per year (Barnes, 2015), since altmetric providers collect data online in real-time and use automated algorithms to process them. Instant feedback is an advantage in comparison with citations. Altmetrics attract the general population’s attention (Banshal et al., 2018) and record public reaction or research exposure (Barnes, 2015). According to the present idea of collective in- telligence (Barnes, 2015), it is reasonable to assume that research people think necessary will draw much more attention. 3. Research Method The starting point for the research is the WoS database. A search of the database on the topic “emerging markets” was carried out with the following filters, aiming at incorporating more relevant scientific literature: document type – articles; Web of Science Index – Social Sciences Citation Index, Science Citation Index Ex- panded. Obtained WoS records exported on 11 November 2019 include articles from 1996 – 2019. During the next couple of weeks, the author eked out the base with additional data. The initial article database is expanded with altmetric data either from Altmetric.com or PlumX, gathering from detailed pages, using Digital Object Identifier (DOI) numbers. From 5192 initial records, the database was reduced to 3996 records (because of the elimination of articles without or with incorrect DOI numbers and articles for which a publisher does not provide/display altmetric data on its pages). The number of articles by providers is the following: • Altmetric.com – 2042 articles; • PlumX – 1954 articles. Table 1 shows the top 10 journals (of the total 650) by the number of articles/records in the database. They together make 26.62% of the total number of records. Table 1: Top 10 journals by the number of articles Further analyses use the number of citations and numbers of unified altmetric data for each article. Namely, data for each common indicator from both altmetric providers, Altmetric.com and PlumX, were unified in sin- gle columns. Therefore, the study observes the following parameters for each article in the sample: • WoS number of citations – the number of citations according to the WoS database; • Mentions – the sum of mentions by all recorded sources (Figure 3 shows the sources). Since the providers have different categories of mentions, here all individual mentions by all sources are summed up without previous weighting; • Twitter – the number of social media interactions from Twitter; • Facebook – the number of social media interactions from Facebook; • Mendeley – the number of Mendeley captures; • CiteULike – the number of CiteULike captures. 4 Zorica Dodevska Forthcoming Journal Name Number of Articles Percentage Number of Articles Emerging Markets Finance and Trade 195 4.88% Emerging Markets Review 144 3.60% Journal of International Money and Finance 138 3.45% Journal of Banking & Finance 108 2.70% Physica A-Statistical Mechanics and Its Applications 99 2.48% Journal of Business Research 94 2.35% International Business Review 80 2.00% Journal of International Business Studies 73 1.83% Research in International Business and Finance 70 1.75% Applied Economics 63 1.58% Total 1,064 26.62% 5 Management: Journal of Sustainable Business and Management Solutions in Emerging Economies Forthcoming Criterion N Relationship between Pearson Correlation Strength All records 3996 WoS number of citations Mendeley 0.467** moderate Mentions 0.176** very weak CiteULike 0.152** very weak WoS number of citations 50 442 WoS number of citations Mendeley 0.562** moderate CiteULike 0.203** weak Figure 3: Representation of Mentions by sources for all articles The Altmetric Attention Score, introduced by Altmetric.com, which weights mention sources and social media sources, is not used in analyses because of the inability to compare with PlumX, which does not use weighting. 4. Results Table 2 shows only statistically significant results for two criteria: all records and, conditionally speaking, only highly cited articles in the sample with the number of citations higher or equal to 501. There are mod- erate correlations between the WoS number of citations and Mendeley in both criteria (in the latter criterion, it is more significant than in the former). Other captured correlations are very weak (between WoS number of citations and Mentions/CiteULike) for the former criterion and weak (between WoS number of citations and CiteULike) for the latter criterion. Table 2 does not include correlations that are not statistically significant in this research, i.e., between WoS number of citations and Twitter/Facebook for the former criterion, and be- tween WoS number of citations and Mentions/Twitter/Facebook for the latter criterion (i.e., selected articles). Table 2: Correlations between WoS number of citations and some altmetric indicators **. Correlation is significant at the 0.01 level (2-tailed). Other results show a weak correlation between Twitter and Mendeley for all records (it equals 0.204) and a moderate correlation between Twitter and Mentions for selected articles (it equals 0.439). The relationship between Mendeley and CiteUlike is very week, and it equals 0.112/0.160 for the former/latter criterion, re- spectively. 1 WoS number of citations ≥ 50 is also used in other fields – e.g., for quantitative analyses in disability-related fields (Ahmed et al., 2016). Similarly, several hydrology journals have approximately 10% highly cited articles with citations more numerous than 50 for ten years of obser- vation (Clark & Hanson, 2017). In this paper, the percentage is similar – it equals 11.06% for citations ≥ 50. 6 Zorica Dodevska Forthcoming Table 3 shows the top 10 journals’ overlapping by the sum of WoS number of citations and the top 10 jour- nals by cumulative altmetric indicator – the sum of Mentions, Twitter, and Facebook, without Mendeley and CiteULike2. There are three journals (shaded in Table 3) that overlap in both categories, while the top 100 journals show 56% of the same overlapping. Table 3 : Overlapping of top 10 journals * Top 10 journal on the basis of the number of articles in the database Regarding the overlapping of individual articles, only two articles overlap between the top 100 according to the number of citations and the top 100 according to the cumulative altmetric indicator. shows the de- pendence of the overlapping of articles on a top selected part of the sample (the overlap for all records equals 100%). Figure 4: Dependence of overlapping of articles on a top selected part of the sample 2 Mendeley and CiteULike do not participate in the cumulative altmetric indicator in this paper in the same way as they do not participate in the Altmetric Attention Score. The difference is that the weighting of individual altmetric indicators is not done here as it is done by Altmet- ric.com to encompass the data from PlumX, where no weighting is used. Top 10 journals by citations Top 10 journals by cumulative altmetric indicator Journal Name WoS number of citations Journal Name (Mentions + Twitter + Facebook) Journal of Financial Economics 7683 Emerging Markets Review* 1029 Journal of Finance 4977 Journal of International Financial Markets Institutions & Money 572 Journal of International Business Studies* 4316 Research in International Business and Finance 419 Journal of Banking & Finance* 4029 Journal of World Business 406 Strategic Management Journal 3821 Journal of Business Research* 397 Journal of International Economics 3574 International Review of Financial Analysis 386 Journal of International Money and Finance* 3450 Journal of International Money and Finance* 342 Review of Financial Studies 3078 Journal of International Economics 305 Physica A-Statistical Mechanics and Its Applications* 2266 Bulletin of Indonesian Economic Studies 282 Journal of World Business 2113 Geoforum 233 5. Discussion Policy sources are the most representative mentions in the research field about emerging markets, followed by news, blogs, Wikipedia, and patents. This result suggests that the topic matches the five most weighted sources listed according to Altmetric.com (2021). In comparison with the result of less than 0.5% of the WoS indexed papers belonging to different WoS subject categories mentioned in policy sources (i.e., policy-re- lated documents) at least once (Haunschild & Bornmann, 2017), this percent is higher in this study. It equals 3.35% in the field of emerging markets. The obtained results do not show statistically significant correlations between the WoS number of citations and the altmetric indicators in general. This especially goes for social media sources (Facebook and Twit- ter). Mentions and CiteULike show very weak correlations with the WoS number of citations, while only Mendeley shows a moderate correlation (0.467). In observing only selected papers (with at least 50 citations), correlation with Mendeley is moderate (0.562) while it is weak with CiteULike. Therefore, it is not advisable to use altmetrics as non-traditional metrics to predict citations. Altmetrics are an imperfect tool in the sense of foretelling future citations (Barnes, 2015). Also, Costas et al. (2015) conclude that “altmetrics do not re- flect the same concept of impact as citations” (p. 2003). Altmetrics rather give evidence about the public in- terest or attention (Tahamtan & Bornmann, 2020). The positive result in Mendeley’s case is not accidental – Mendeley reader counts for journal articles represent their useful early impact indicators (Thelwall, 2018). Bornmann and Haunschild (2018a) point out that “citation-based metrics and readership counts are significantly more related to quality, than tweets” (p. 1/12). Tweets do not show a positive result in this study, although there is a claim that those collected in the first three days after article publishing can predict highly cited articles (Eysenbach, 2011). The top 100 articles, according to the WoS number of citations and the cumulative altmetric indicator, show minimal overlapping of only two articles (2%). Similar disjoint results are provided by Banshal et al. (2018), who have compared the top 100 Altmetric 2016 papers and the top 100 WoS citation papers – only 12 arti- cles overlap. It should be remembered that 48% of the top 100 Altmetric 2016 papers come from medical literature, while this paper deals with different research field. The percentage of overlapping top-ranked jour- nals according to the WoS number of citations and the cumulative altmetric indicator in this paper is higher – it equals 30% (in the top 10) and 56% (in the top 100). 7 Management: Journal of Sustainable Business and Management Solutions in Emerging Economies Forthcoming Conclusion The paper presents an original and expanded bibliometrics study conducted in a specific field, i.e., emerging markets. It includes relevant scientific articles and data from three sources: one citation database and two altmetric providers. The gen- eral conclusion is that the role of altmetrics is not to predict citations. Therefore, altmetrics are not a reliable indicator for the early evaluation of research projects in this field. Besides, scientific and public preferences do not converge in the top- ranked articles on emerging markets, while their interests in observing top journals are better matched. Generally, the academic impact is closer to the quality of publications than the societal impact if we represent them through altmetrics, i.e., ”the relationship of altmetrics and quality is not as strong as the relationship between citations and qual- ity” (Bornmann & Haunschild, 2018b, p.1010). Furthermore, altmetric potentials are in linking research literature and ac- tions in the society. Still, altmetrics may capture some different aspects of societal impact, the so-called “unknown attention” (Bornmann et al., 2019). The positive correlation of citations and academic readers (such as Mendeley) is not surprising because researchers who write articles use them. Simultaneously, the subject of societal mentions is not restricted to only scientific literature (and not only directly). Unofficial contents rather come into sight of the wider population, which is not inclined to reference. Altmetrics and citations are rather more complementary than similar measures, so it is correct that altimetric providers show both data on their pages, providing a complete picture of scientific and public attention, which does not overlap. The dif- ferent natures of these metrics also imply their different usage. In the context of funding research projects, it is not advis- able to use altmetrics for their evaluation. Altmetrics may have different roles that need to be investigated in the future, for example, in crowdfunding (Morell et al., 2020). As one way of alternative funding in emerging markets, this activity is rising (Cumming & Zhang, 2016). It relies on attracting public attention for research work since funders are interested in a “broader impact” (Altmetric YouTube Channel, 2015). In general, activities that would unite the scientific and ‘non-scientific’ efforts and knowledge are desirable and necessary for the further growth of emerging markets (e.g., in the domain of open innovations – Kafouros & Forsans, 2012). Other situations may include overcoming ongoing crises (e.g., consequences of COVID-19 on emerging markets, discussed by Arellano et al., 2020; Uddin et al., 2020; Topcu & Gulal, 2020; etc.). REFERENCES [1] Ahmed, A., Adam, M., Ghafar, N. A., Muhammad, M., & Ebrahim, N. A. (2016). Impact of article page count and number of authors on citations in disability related fields: a systematic review article. Iranian journal of public health, 45(9), 1118–1125. [2] Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. Sage Open, 9(1), 2158244019829575. DOI: 10.1177/2158244019829575 [3] Altmetric YouTube Channel. (2015, Jul 15). Altmetrics in practice [Video]. YouTube. https://www.youtube.com/watch?v=Fglec-1LScc&feature=emb_rel_pause [4] Altmetric.com (2021, September 21). How is the Altmetric Attention Score calculated? Retrieved from https://help.altmetric.com/support/solutions/articles/6000233311-how-is-the-altmetric-attention-score- calculated- [5] Arellano, C., Bai, Y., & Mihalache, G. P. (2020). Deadly Debt Crises: COVID-19 in Emerging Markets (No. w27275). National Bureau of Economic Research. DOI: 10.3386/w27275 [6] Banshal, S. K., Basu, A., Singh, V. K., & Muhuri, P. K. (2018, January). Scientific vs. public attention: a comparison of top cited papers in WoS and top papers by Altmetric Score. In International Workshop on Altmetrics for Research Outputs Measurements and Scholarly Information Management (pp. 81-95). Springer, Singapore. DOI: 10.1007/978-981-13-1053-9_7 [7] Barnes, C. (2015). The use of altmetrics as a tool for measuring research impact. Australian academic & research libraries, 46(2), 121-134. DOI: 10.1080/00048623.2014.1003174 [8] Barthel, S., Tönnies, S., Köhncke, B., Siehndel, P., & Balke, W. T. (2015, June). What does Twitter meas- ure? Influence of diverse user groups in altmetrics. In Proceedings of the 15th ACM/IEEE-CS Joint con- ference on digital libraries (pp. 119-128). DOI: 10.1145/2756406.2756913 [9] Bornmann, L., & Daniel, H. D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of documentation, 64(1), 45–80. DOI: 10.1108/00220410810844150 [10] Bornmann, L., & Haunschild, R. (2018a). Do altmetrics correlate with the quality of papers? A large- scale empirical study based on F1000Prime data. PloS one, 13(5), e0197133, 12 pages. DOI: 10.1371/journal.pone.0197133 [11] Bornmann, L., & Haunschild, R. (2018b). Normalization of zero-inflated data: An empirical analysis of a new indicator family and its use with altmetrics data. Journal of informetrics, 12(3), 998-1011. DOI: 10.1016/j.joi.2018.01.010 [12] Bornmann, L., & Haunschild, R. (2019). Societal Impact Measurement of Research Papers. In W. Glän- zel, H. F. Moed, U. Schmoch & M. Thelwall (Eds.), Springer Handbook of Science and Technology In- dicators (pp. 609-632). Springer, Cham. DOI: 10.1007/978-3-030-02511-3_23 [13] Bornmann, L., Haunschild, R., & Adams, J. (2019). Do altmetrics assess societal impact in a compara- ble way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF). Journal of informetrics, 13(1), 325-340. DOI: 10.1016/j.joi.2019.01.008 [14] Buttliere, B., & Buder, J. (2017). Personalizing papers using Altmetrics: comparing paper ‘Quality’ or ‘Im- pact’ to person ‘Intelligence’ or ‘Personality’. Scientometrics, 111(1), 219-239. DOI: 10.1007/s11192- 017-2246-9 [15] Cavusgil, S. T. (2021). Advancing knowledge on emerging markets: Past and future research in per- spective. International Business Review, 101796. DOI: 10.1016/j.ibusrev.2021.101796 [16] Clark, M. P., & Hanson, R. B. (2017). The citation impact of hydrology journals. Water Resources Re- search, 53(6), 4533-4541. DOI: 10.1002/2017WR021125 [17] Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive compa- rison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Associa- tion for Information Science and Technology, 66(10), 2003-2019. DOI: 10.1002/asi.23309 [18] Cumming, D., & Zhang, Y. (2016). Alternative investments in emerging markets: A review and new trends. Emerging Markets Review, 29, 1-23. DOI: 10.1016/j.ememar.2016.08.022 8 Zorica Dodevska Forthcoming One limitation of this study is the representation of many zeros in the data, without using any approach to overcome this data flaw. One such technique is, for example, the Mantel-Haenszel quotient indicator (Bornmann & Haunschild, 2018b). Other lim- itations include work with altmetric data that can change daily and the existence of ‘unequal’ metrics among used providers. The relationship between studied metrics and preprint publishing is one of the possible research directions for future research. Acknowledgment This research study has been supported by the research grants No. 451-03-9/2021-14/200066 of the Serbian Ministry of Ed- ucation, Science and Technological Development, related to Research and Development Institute Lola. [19] Da Silva, J. A. T., & Dobránszki, J. (2018). Multiple versions of the h-index: Cautionary use for formal aca- demic purposes. Scientometrics, 115(2), 1107-1113. DOI: 10.1007/s11192-018-2680-3 [20] Drivas, K., & Kremmydas, D. (2020). The Matthew effect of a journal's ranking. Research Policy, 49(4), 103951. DOI: 10.1016/j.respol.2020.103951 [21] Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and cor- relation with traditional metrics of scientific impact. Journal of medical Internet research, 13(4), e123. DOI: 10.2196/jmir.2012 [22] Fang, Z., Costas, R., Tian, W., Wang, X., & Wouters, P. (2020). An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics. Scientome- trics, 124(3), 2519-2549. DOI: 10.1007/s11192-020-03564-9 [23] Grimes, D. R., Bauch, C. T., & Ioannidis, J. P. (2018). Modelling science trustworthiness under publish or perish pressure. Royal Society open science, 5(1), 171511. DOI: 10.1098/rsos.171511 [24] Haunschild, R., & Bornmann, L. (2017). How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics, 110(3), 1209-1216. DOI: 10.1007/s11192-016-2237-2 [25] Haustein, S. (2016). Grand challenges in altmetrics: heterogeneity, data quality and dependen- cies. Scientometrics, 108(1), 413-423. DOI: 10.1007/s11192-016-1910-9 [26] Heydari, S., Shekofteh, M., & Kazerani, M. (2019). Relationship between altmetrics and citations a study on the highly cited research papers. DESIDOC Journal of Library & Information Technology, 39(4), 169. DOI: D10.14429/djlit.39.4.14204 [27] Hou, J., Yang, X., & Chen, C. (2020). Measuring researchers’ potential scholarly impact with structural variations: Four types of researchers in information science (1979–2018). PloS one, 15(6), e0234347. DOI: 10.1371/journal.pone.0234347 [28] Ioannidis, J. P. (2015). A generalized view of self-citation: Direct, co-author, collaborative, and coercive induced self-citation. Journal of psychosomatic research, 78(1), 7-11. DOI: 10.1016/j.jpsychores.2014.11.008 [29] Johnes, J. (2018). University rankings: What do they really show?. Scientometrics, 115(1), 585-606. DOI: 10.1007/s11192-018-2666-1 [30] Kafouros, M. I., & Forsans, N. (2012). The role of open innovation in emerging economies: Do compa- nies profit from the scientific knowledge of others?. Journal of World Business, 47(3), 362-370. DOI: 10.1016/j.jwb.2011.05.004 [31] Karanatsiou, D., Misirlis, N., & Vlachopoulou, M. (2017). Bibliometrics and altmetrics literature re- view. Performance Measurement and Metrics, 18(1), 16–27. DOI: 10.1108/PMM-08-2016-0036 [32] Li, K., Rollins, J., & Yan, E. (2018). Web of Science use in published research and review papers 1997– 2017: A selective, dynamic, cross-domain, content-based analysis. Scientometrics, 115(1), 1-20. DOI: 10.1007/s11192-017-2622-5 [33] Liu, N. C., & Cheng, Y. (2005). The academic ranking of world universities. Higher education in Eu- rope, 30(2), 127-136. DOI: 10.1080/03797720500260116 [34] Martin, B. R., & Irvine, J. (1983). Assessing basic research: some partial indicators of scientific progress in radio astronomy. Research policy, 12(2), 61-90. DOI: 10.1016/0048-7333(83)90005-7 [35] Martín-Martín, A., Thelwall, M., Orduna-Malea, E., & López-Cózar, E. D. (2020). Google Scholar, Micro- soft Academic, Scopus, Dimensions, Web of Science, and OpenCitations' COCI: a multidisciplinary comparison of coverage via citations. Scientometrics, 36 pages, DOI: 10.1007/s11192-020-03690-4. [36] Morell, M. F., Hidalgo, E. S., & Rodríguez, E. (2020). Goteo. org civic crowdfunding and match-funding data connecting Sustainable Development Goals. Scientific Data, 7(1), 1-10. DOI: 10.1038/s41597-020- 0472-0 [37] O'Neill, J. (2001). Building better global economic BRICs. Retrieved from http://www.elcorreo.eu.org/IMG/pdf/Building_Better_Global_Economic_Brics.pdf [38] Ortega, J. L. (2018). Reliability and accuracy of altmetric providers: a comparison among Altmetric. com, PlumX and Crossref Event Data. Scientometrics, 116(3), 2123-2138. DOI: 10.1007/s11192-018- 2838-z [39] Priem, J. & Hemminger, B. H. (2010). Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web, First Monday, 15(7). Available at https://firstmonday.org/ojs/index.php/fm/article/view/2874/2570/ [40] Subochev, A., Aleskerov, F., & Pislyakov, V. (2018). Ranking journals using social choice theory methods: A novel approach in bibliometrics. Journal of Informetrics, 12(2), 416-429. DOI: 10.1016/j.joi.2018.03.001 [41] Sudhir, K., Priester, J., Shum, M., Atkin, D., Foster, A., Iyer, G., ... & Wood, W. (2015). Research oppor- tunities in emerging markets: an inter-disciplinary perspective from marketing, economics, and psyc- hology. Customer Needs and Solutions, 2(4), 264-276. DOI: 10.1007/s40547-015-0044-1 [42] Szomszor, M., Adams, J., Fry, R., Gebert, C., Pendlebury, D. A., Potter, R. W., & Rogers, G. (2021). In- terpreting bibliometric data. Frontiers in Research Metrics and Analytics, 5, 30. DOI: 10.3389/frma.2020.628703 9 Management: Journal of Sustainable Business and Management Solutions in Emerging Economies Forthcoming 10 Zorica Dodevska Forthcoming [43] Tahamtan, I., & Bornmann, L. (2020). Altmetrics and societal impact measurements: Match or mi- smatch? A literature review. El profesional de la información (EPI), 29(1). DOI: 10.3145/epi.2020.ene.02 [44] Thelwall, M. (2018). Early Mendeley readers correlate with later citation counts. Scientometrics, 115(3), 1231-1240. DOI: 10.1007/s11192-018-2715-9 [45] Thelwall, M., Kousha, K., Dinsmore, A., & Dolby, K. (2016). Alternative metric indicators for funding scheme evaluations. Aslib Journal of Information Management. 68(1), 2–18. DOI: 0.1108/AJIM-09-2015- 0146 [46] Topcu, M., & Gulal, O. S. (2020). The impact of COVID-19 on emerging stock markets. Finance Re- search Letters, 36, 101691. DOI: 10.1016/j.frl.2020.101691 [47] Tunger, D., Clermont, M., & Meier, A. (2018). Altmetrics: State of the Art and a Look into the Future. In M. Jibu & Y. Osabe (Eds.), Scientometrics (pp. 123-134). London, UK: IntechOpen. DOI: 10.5772/in- techopen.76874 [48] Uddin, G. S., Yahya, M., Goswami, G. G., Ahmed, A., & Lucey, B. M. (2020). Stock market contagion of covid-19 in emerging economies. Available at SSRN 3573333. [49] Vavryčuk, V. (2018). Fair ranking of researchers and research teams. PloS one, 13(4), e0195509, 17 pages. DOI: 10.1371/journal.pone.0195509 [50] Westra, R. (2017). From development to BRICS: The policy magical mystery tour. In R. Westra (Ed.), The Political Economy of Emerging Markets: Varieties of BRICS in the Age of Global Crises and Austerity (pp. 3-31). Routledge: Taylor & Francis Group. Received: 2020-01-27 Revision requested: 2020-06-01 Revised: 2021-03-08 (3 revisions) Accepted: 2021-05-23 Zorica Dodevska Institute for Artificial Intelligence Research and Development of Serbia, Novi Sad, Serbia zorica.dodevska@ivi.ac.rs Zorica Dodevska is a research assistant at the Institute for Artificial Intelligence Research and Development of Serbia, Novi Sad. While working at the Research and Development Institute Lola, Belgrade, she wrote this paper. She is a doctoral candidate at the Faculty of Organizational Sciences, University of Belgrade. Zorica’s narrow scientific field is modelling business systems and business decision making. Her research interests include fair ranking, multi-criteria decision making, quantitative models, preferential dependencies, artificial intelligence, machine learning, voting, bibliometrics, simplified robotic models, augmented reality, and project management. About the Author << /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /All /Binding /Left /CalGrayProfile (Dot Gain 20%) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (U.S. Web Coated \050SWOP\051 v2) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Warning /CompatibilityLevel 1.4 /CompressObjects /Tags /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages true /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /DetectCurves 0.0000 /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedOpenType false /ParseICCProfilesInComments true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 1048576 /LockDistillerParams false /MaxSubsetPct 100 /Optimize true /OPM 1 /ParseDSCComments true /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveDICMYKValues true /PreserveEPSInfo true /PreserveFlatness true /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Apply /UCRandBGInfo /Preserve /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true ] /NeverEmbed [ true ] /AntiAliasColorImages false /CropColorImages true /ColorImageMinResolution 300 /ColorImageMinResolutionPolicy /OK /DownsampleColorImages true /ColorImageDownsampleType /Bicubic /ColorImageResolution 300 /ColorImageDepth -1 /ColorImageMinDownsampleDepth 1 /ColorImageDownsampleThreshold 1.50000 /EncodeColorImages true /ColorImageFilter /DCTEncode /AutoFilterColorImages true /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /ColorImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 300 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /GrayImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict << /K -1 >> /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (None) /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False /CreateJDFFile false /Description << /ARA /BGR /CHS /CHT /CZE /DAN /DEU /ESP /ETI /FRA /GRE /HEB /HRV /HUN /ITA /JPN /KOR /LTH /LVI /NLD (Gebruik deze instellingen om Adobe PDF-documenten te maken voor kwaliteitsafdrukken op desktopprinters en proofers. De gemaakte PDF-documenten kunnen worden geopend met Acrobat en Adobe Reader 5.0 en hoger.) /NOR /POL /PTB /RUM /RUS /SKY /SLV /SUO /SVE /TUR /UKR /ENU (Use these settings to create Adobe PDF documents for quality printing on desktop printers and proofers. Created PDF documents can be opened with Acrobat and Adobe Reader 5.0 and later.) >> /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ << /AsReaderSpreads false /CropImagesToFrames true /ErrorControl /WarnAndContinue /FlattenerIgnoreSpreadOverrides false /IncludeGuidesGrids false /IncludeNonPrinting false /IncludeSlug false /Namespace [ (Adobe) (InDesign) (4.0) ] /OmitPlacedBitmaps false /OmitPlacedEPS false /OmitPlacedPDF false /SimulateOverprint /Legacy >> << /AddBleedMarks false /AddColorBars false /AddCropMarks false /AddPageInfo false /AddRegMarks false /ConvertColors /NoConversion /DestinationProfileName () /DestinationProfileSelector /NA /Downsample16BitImages true /FlattenerPreset << /PresetSelector /MediumResolution >> /FormElements false /GenerateStructure true /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles true /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /NA /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /LeaveUntagged /UseDocumentBleed false >> ] >> setdistillerparams << /HWResolution [2400 2400] /PageSize [623.622 850.394] >> setpagedevice