SKIN November 2020 Volume 4 Issue 6 Copyright 2020 The National Society for Cutaneous Medicine 497 ORIGINAL RESEARCH Analyzing the Relationship Between Altmetric Score and Literature Citations in the Dermatology Literature Meredith Thomley, BS1, Ana Preda-Naumescu, BS1, Carter J. Boyd, MD, MBA1*, Tiffany Mayo, MD2 1School of Medicine, University of Alabama at Birmingham, Birmingham, AL 2Assistant Professor, Department of Dermatology, University of Alabama at Birmingham, Birmingham, AL *equal contribution as primary authors Scientific research furthers our wealth of medical knowledge and can enact clinical change. The question stands of how the impact of a single article can be determined. Various metrics seek to describe this impact. The publishing journal’s impact factor (IF) and the article’s citation count have traditionally served as the quantitative measures of an article’s impact.1 These two metrics have limitations with the former yielding a generalization marker based on the journal rather than the article, and the latter making it difficult for authors and peers to see the true impact of a publication until years later, when substantial citation information has been accrued.2,3 ABSTRACT Background: Standard bibliometric methods used in dermatologic research include impact factor and citations. The Altmetric score is an adjunctive measure of article impact. Objectives: The purpose of this study is to examine the breadth of societal impact made by scientific articles in dermatology and investigate a correlation between an article’s impact factor and citations, with its Altmetric score. Methods: We reviewed 15 dermatology journals with the highest impact factors and analyzed the 10 most cited articles from 2013 and 2016 within those journals. We studied the articles’ Altmetric scores, number of citations, and social media mentions. Using Microsoft Excel, we performed statistical analysis with Pearson correlation coefficients and descriptive statistics. Results: Analysis revealed a significant positive relationship between citation count and Altmetric scores for articles published in 2013 (p=0.0009) and 2016 (p=0.003). Impact factor was also significantly associated with Altmetric scores across both years (p=0.002, p=0.0005). Conclusions: Altmetric score weakly corresponded with citation count and journal impact factor across cohorts. We conclude that Altmetric scores serve as an additional measurement of article impact in dermatology, though they are insufficient as a replacement for traditional measures at this time. INTRODUCTION SKIN November 2020 Volume 4 Issue 6 Copyright 2020 The National Society for Cutaneous Medicine 498 The platform Altmetric first published its rating scale, the Altmetric score, in 2010.4 This algorithm gives a numerical score to articles’ combined influence across mixed media platforms, including mentions from Twitter to Facebook, new outlets to public policy documents, video sources to Reddit, and others.5 Altmetric incorporates the digital distribution and publicity for a scientific article, grading each article based on individual mentions across platforms. The mentions are converted into a total index measure of attention and impact, useful for standardization of bibliometric data collection. The score does not measure the quality of the research, the researcher, or the attention an article has accrued, but it is useful for measuring the article’s visibility and overall reach, with a higher score indicating a greater overall reach.4 To say a higher score is better, or alternatively worse, than a lower score is subjective- the score measures attention which can itself be ‘good or bad.’ While there are not cutoffs such as a maximum or minimum possible score, it is always a whole number.4 We propose the use of the Altmetric score as a timely and accurate proxy measure for article impact, correlating with the traditional metrics of IF and citations. In a society with the world of information in the palm of one’s hand, journal impact factor and an article’s citation count cause a delay in data that is contrary to modern standards. Social media platforms and the internet at large are granted immediate access to research articles by a simple web search. By drawing on alternative metric ratings, a more rapid picture of the impact of a published research article can be established. The authors selected the top 15 journals by impact factor in dermatology for inclusion in this study. The list was gathered by searching “Dermatology” in Elsevier’s Citescore in the Scopus database.7 These journals include: Journal of Investigative Dermatology, Pigment Cell & Melanoma Research, Journal of the American Academy of Dermatology, JAMA Dermatology, Acta Derma- Venereologica, Experimental Dermatology, British Journal of Dermatology, Contact Dermatitis, Journal of Dermatological Science, Journal of the European Academy of Dermatology, Wound Repair and Regeneration, American Journal of Clinical Dermatology, Dermatologic Surgery, Seminars in Cutaneous Medicine and Surgery, and Journal of Dermatology. We extracted data on Twitter presence of each of these journals, noting whether or not the journals had accounts, and if so, how long have they been active. This study also determined journal IF, acquired from Journal Citation Reports (JCR), across two cohorts: 2013 and 2016.6 2013 was chosen to yield a six-year interim to present day, the citations after this allotted time correlate with 90% total citations. Data from 2016 allows for better comparison to a modern Altmetric algorithm, though citation count three-years post- publication is 75% predictive of total citations.7 The ten most cited articles in each journal were analyzed for bibliometric data; the sets of ten were identified by utilizing Elsevier’s Scopus.8 150 total articles were analyzed. The Altmetric Attention score was calculated by using the Altmetric bookmarklet, available for download through the Altmetric website. Altmetric scores were recorded for each article across 2013 and 2016. Specific social media and digital dispersion data were collected on the following platforms: Twitter, Facebook, blogs, policy sources, Wikipedia, Reddit, videos, Google+, and research highlights.4 METHODS SKIN November 2020 Volume 4 Issue 6 Copyright 2020 The National Society for Cutaneous Medicine 499 All data was gathered and analyzed via Microsoft Excel; statistical significance was determined by p-value <0.05. Pearson’s correlation coefficient (r) was used to establish correlation between variables; the coefficient of determination (R2) was used to measure the amount of variance in these correlations. This study largely modeled its methods, procedures, and analysis from examples in pediatric surgery, general surgery, and urology literature, also comparing Altmetric score to citation and impact factor.7,9,10 From the 15 dermatology journals with the highest impact factor based on the JCR, the 10 most highly cited articles published in 2013 and 2016 were reviewed (Table 1). Thus, 150 articles were reviewed from each year. In 2013, the cumulative total of citations for all articles was 13,819 and the total Altmetric Attention Score was 2,397. The median number of citations for each individual article was 62 with a range from 16- 854 (standard deviation 80.1). The median Altmetric Attention scores for each individual article was 3 with a range from 0-396 (standard deviation 51.7). Citation number and journal impact factor demonstrated a significant positive correlation in the 2013 cohort (r = 0.463, p<0.0001). Analysis revealed a significant positive relationship between citation count and Altmetric Attention scores for articles published in 2013 (r=0.267, p=0.0009) and a significant positive relationship between Altmetric Attention scores and journal impact factor for the same cohort of articles (r=0.255, p=0.002) (Figure 1 A, B). R2 was low for both correlations: R2=0.072 for citations and R2=0.065 for impact factor. Journals were also studied in isolation to calculate the correlation coefficient between citation number and Altmetric Attention score (Table 2). In 2013, only Acta Dermato- Venereologica (r=0.891, p=0.0005) and Seminars in Cutaneous Medicine and Surgery (r=0.985, p<0.0001) had significant correlations between their articles’ citation count and Altmetric Attention scores. The remainder were not significant, though two others approached statistical significance. To compare the evolution across time, data was also collected in 2016. Total citation count for 2016 was 6,717 compared to the 13,819 for 2013, a 51.4% decline. The total Altmetric Attention score was 2,397 in 2013 but climbed by 60.3% to a score of 3,843 in 2016. Median number of citations in 2016 was 31.5 ranging from 8-425 (standard deviation 42.9), whereas the median Altmetric score in 2016 was 2.5 with a range of 0-509 (standard deviation 83.6). Altmetric ratings were significantly correlated with citation count for the 2016 cohort (r=0.244, p=0.003). Impact factor was significantly associated with Altmetric Attention scores as well (r=0.283, p=0.0005). R2 was minimal for both, 0.060 for citation count and 0.080 for journal impact factor in 2016. The journals from 2016 were likewise studied individually to calculate the correlation coefficient between citation count and Altmetric score (Table 2). Five of the 15 journals had significant correlations between their articles’ citation count and article Altmetric Attention score in 2016. These journals were Journal of Investigative Dermatology (r=0.935, p<0.0001), Journal of the American Academy of Dermatology (r=0.797, p=0.006), Experimental Dermatology (r=0.727, p=0.02), Journal of Dermatological Science (r=0.843, p=0.0002), and Journal of Dermatology (r=0.723, p=0.017). Nine of the 15 journals had Twitter accounts at the time of review (Table 1). Average age of the RESULTS SKIN November 2020 Volume 4 Issue 6 Copyright 2020 The National Society for Cutaneous Medicine 500 Table 1. Categorization of dermatology journal bibliometric and Altmetric data. Journal Citations (median [range]) Altmetric Score (median [range]) Journal Impact Factor (2013) Journal Impact Factor (2016) Age of Twitter Account (y) Journal of Investigative Dermatology 160.5 [120- 854] 8.5 [1-277] 6.372 6.287 7.33 Pigment Cell & Melanoma Research 55 [46-164] 2 [0-17] 5.641 5.17 n/a Journal of the American Academy of Dermatology 151 [107-276] 12.5 [1-396] 5.004 7.002 4.08 JAMA Dermatology 129 [91-203] 28.5 [2-323] 4.970 5.817 0 Acta Dermato- Venereologica 40.5 [32-144] 1 [0-12] 4.244 3.653 8.25 Experimental Dermatology 52.5 [47-115] 1 [0-23] 4.115 2.532 2.67 British Journal of Dermatology 172 [101-233] 5 [1-85] 4.1 3.605 6.17 Contact Dermatitis 62 [48-70] 4 [0-11] 3.624 4.335 0.5 Journal of Dermatological Science 57 [40-172] 0 [0-4] 3.335 3.733 n/a Journal of the European Academy of Dermatology and Venereology 90.5 [74-196] 4 [1-51] 3.105 3.528 0.42 Wound Repair and Regeneration 62 [37-99] 1 [0-16] 2.768 3.041 n/a American Journal of Clinical Dermatology 53.5 [37-142] 7.5 [1-26] 2.519 2.755 n/a Dermatologic Surgery 47 [41-62] 3 [1-13] 2.467 2.351 n/a Seminars in Cutaneous Medicine and Surgery 22 [16-205] 2 [0-81] 2.402 1.317 1.92 Journal of Dermatology 40 [35-51] 1 [0-12] 2.354 2.094 n/a Twitter accounts was 3.5 years. Acta Dermato-Venerologica had the oldest Twitter account (established in 2011), while JAMA Dermatology just recently established a Twitter account in 2019 around the time of this study. Older Twitter accounts were not significantly associated with increasing correlations between Altmetric Attention score and bibliometrics in both cohorts: 2013 (r=0.012, p=0.783) and 2016 (r=0.0002, p=0.973) (Figure 2). Dissecting the components comprising the Altmetric Attention score in 2013 revealed that Twitter, news outlets, and Facebook were the top three platforms in which articles received mentions. This rank order for article mentions remained the same SKIN November 2020 Volume 4 Issue 6 Copyright 2020 The National Society for Cutaneous Medicine 501 Figure 1. Altmetric comparisons with citation count and journal impact factor by year. A. Altmetric significantly correlates with citation count for articles published in 2013 (p = 0.0009). B. Altmetric demonstrated a significant positive association with journal impact factor for articles published in 2013 (p = 0.002). C. Altmetric Attention scores have a significant positive correlation with citation count for articles published in 2016 (p = 0.003). D. Altmetric Attention scores have a significant positive correlation with journal impact factor for articles published in 2016 (p = 0.0005). R2 – A. 0.0715; B. 0.0651; C. 0.0594; D. 0.0798 in 2016 (Table 3). Twitter was the clear leader for mentions received in both years with 180% more mentions than the second most utilized platform in 2013, though this margin diminished slightly in 2016 to 145%. Between the two years, Twitter (85.7%), news outlets (109.7%), Reddit (100%), and Google + (84.6%) saw increases in mentions. All of the remaining platforms’ mentions decreased in 2016 compared to 2013 (Table 3). To date, the influence of an article has been based on both the IF of the journal and the number of citations. However, because citation information accrues over time, the delayed citation counts yield an interim during which the actual impact of an article cannot be yet determined.2,3 For a more modern approach, the Altmetric score takes into account the overall popularity of an article in real time by examining its mixed media influence. This is done based off of a variety of measures, such as Twitter, Facebook, Wikipedia mentions, Google+, news outlets, etc. We posited that the Altmetric score could predict and parallel the total impression of an article by acting as a proxy for IF and citation count in the top dermatology journals. This study is the first, to the authors’ knowledge, to examine these various bibliometric methods in the field of dermatologic research. Our findings indicate a positive correlation between the modern platform, Altmetric scores, with previously standard methods, IF and citations, in both year cohorts. However, the correlation was determined by low R2 coefficients which DISCUSSION C. D. A. B. SKIN November 2020 Volume 4 Issue 6 Copyright 2020 The National Society for Cutaneous Medicine 502 Table 2. Summary of journal Twitter accounts and comparisons between bibliometrics and Altmetrics for the journals of study in 2013 and 2016. Journal Year Twitter Created Correlation Coefficient between number of citations and Altmetric score (2013) P-value (2013) Correlation Coefficient between number of citations and Altmetric score (2016) P-value (2016) Journal of Investigative Dermatology 2012 0.02 0.95 0.93 <0.0001 Pigment Cell & Melanoma Research n/a 0.11 0.77 0.005 0.99 Journal of the American Academy of Dermatology 2015 0.0006 0.99 0.80 0.006 JAMA Dermatology 2019 0.58 0.08 -0.29 0.42 Acta Dermato- Venereologica 2011 0.89 0.0005 -0.11 0.75 Experimental Dermatology 2016 -0.15 0.69 0.73 0.02 British Journal of Dermatology 2013 0.47 0.17 -0.17 0.63 Contact Dermatitis 2019 -0.18 0.63 -0.28 0.42 Journal of Dermatological Science n/a 0.59 0.073 0.84 0.0002 Journal of the European Academy of Dermatology and Venereology 2019 -0.28 0.44 0.24 0.50 Wound Repair and Regeneration n/a -0.23 0.51 0.54 0.11 American Journal of Clinical Dermatology n/a -0.006 0.99 0.28 0.44 Dermatologic Surgery n/a -0.01 0.97 -0.20 0.57 Seminars in Cutaneous Medicine and Surgery 2017 0.98 <0.0001 n/a n/a Journal of Dermatology n/a -0.06 0.87 0.72 0.02 suggests that citation count and journal impact factor are not entirely explained by article Altmetric Attention score. While Altmetric Attention scores may be broadly predictive of the impact of an article, they are not an all-inclusive bibliometric for journal impact and citation count. Altmetric scores show attention or engagement with an article, whether that be positive or negative, as opposed to the quality of the article’s impact as far as its implications for traditional scientific research. In order to fully evaluate the scientific and societal contributions of an article, a comparison between Altmetric and the more traditional methods can yield complementary perspectives and a better gage of overall impact. SKIN November 2020 Volume 4 Issue 6 Copyright 2020 The National Society for Cutaneous Medicine 503 Figure 2. Relationship between the Twitter account age and the correlation coefficients between citation number and Altmetric Attention score. Longer-established Twitter accounts were not associated with increased correlation between Altmetric Attention score and bibliometric factors for the dermatology journals in 2013 (A, p = 0.95) or 2016 (B, p = 0.81). R2 – A. 0.0115; B. 0.0002 Table 3. Classification of social media utilization factoring into the cumulative Altmetric score for journals by year. Source Article "mentions" in 2013 Article "mentions" in 2016 Percent Change (%) Twitter 512 951 85.74 Facebook 159 139 -12.57 Blog 30 5 -83.33 Policy Source 20 9 -55.00 News Outlets 185 388 109.7 Wikipedia 28 14 -50 Reddit 2 4 100.00 Videos 23 2 -91.30 Patents 87 7 -91.95 Google + 13 24 84.62 Research Highlight platforms 20 18 -10.00 A. B. SKIN November 2020 Volume 4 Issue 6 Copyright 2020 The National Society for Cutaneous Medicine 504 When studying journals in isolation, 2 of the 15 showed a significant correlation between citation number and Altmetric Attention score in 2013, while 5 of the 15 demonstrated a significant correlation between citation number and Altmetric score in 2016. With the continued growth of social media and digital involvement in research publicity over time, it could be that Altmetric Attention scores will follow this trend towards becoming more representative, rather than simply complementary, in describing a journals’ impact. However, the minority of journals achieved significant correlation in this way, suggesting that the Altmetric algorithm does not entirely explain impact for top dermatology journals. Additionally, Twitter was demonstrated to be the key platform to increasing engagement of dermatology research articles. Twitter exceeded all other social media platforms in its mentions, in both 2013 and 2016. The majority of journals studied in this case had a Twitter presence for an average of 3.5 years, with accounts created between 2011 and 2019. Interestingly, the older Twitter accounts did not display increased correlations between Altmetric and traditional impact metrics. Moving forward, further studies of true Twitter activity and engagement could be more indicative of correct correlation, rather than simply length of account life. With Twitter, news outlets, and Facebook contributing largest to Altmetric score, authors and publishers might draw upon this information to concentrate marketing efforts to augment article impact in the future. These three platforms prove to be popular amongst dermatology researchers and publishers. Furthermore, it is important to note that across the three-year-time period, Twitter, news outlets, Reddit, and Google+ all displayed increasing trends in mentions, indicative of the expanding online presence for scientific research. The total Altmetric Attention score for the 2016 cohort of articles was 60.3% higher than in 2013, whereas article citation counts were 51.4% lower in 2016 than 2013. The fewer citations by the 2016 cohort is attributable to the decrease in time for these articles to accrue citations. The increased volume of Altmetric score coincides with the ever-growing use of media incorporation into article impact. As the evolution of media continues to infiltrate the world of scientific research, we predict the weak correlation between traditional measures and Altmetric scores to grow stronger. Article “impact” describes an amalgam of factors, including journal notability, citation count, media mentions, and web dissemination. The benefit to utilizing the Altmetric score is the speed at which one can sum up these various contributing factors to quantify a research article’s success. Though, based on our findings, it cannot wholly describe the final impact of an article in the field of dermatology, it can contribute to the overall picture of how the article is performing in present day, based on social media standards. Evaluating the impact of a scientific article accurately is important for authors and readers alike. By quantifying the number of mentions, shares, reposts, and citations into a standardized metric, a publication can be more appropriately graded, compared, and applied clinically. Bibliometric data has traditionally been based on journal IF and citation accumulation; however, alternative metric ratings have emerged as a way to incorporate social media and online presence into the overall measurement of an article’s impact. The Altmetric score, draws on media CONCLUSION SKIN November 2020 Volume 4 Issue 6 Copyright 2020 The National Society for Cutaneous Medicine 505 platform information, which traditional metric patterns exclude. Altmetric method provides a quicker analysis of impact than the previous metrics could provide via long-term citation accumulation and positively, yet weakly, correlates with previous methods of measuring an article’s impact, such as IF and citations. Altmetric data may be used as a supplementary measure of article success, but not as a substitution to journal IF and citations in top dermatologic journals. Conflict of Interest Disclosures: None Funding: None Corresponding Author: Meredith Thomley, BS Department of Dermatology University of Alabama at Birmingham Faculty Office Tower 1107 510 20th Street South Birmingham, AL 35249 Phone: 205-937-8135 Email: thomley@uab.edu References: 1. Durieux V, Gevenois P. Bibliometric Indicators: Quality Measurements of Scientific Publication. Radiology. 2010;255(2):342-351. 2. Bornmann L LL. Scientometrics in a changing research landscape: bibliometrics has become an integral part of research quality evaluation and has been changing the practice of research. EMBO Rep. 2014;15:1228e1232. 3. Wang J. Citation time window choice for research impact evaluation. Scientometrics. 2012;94(3):851-872. 4. Altmetric. Altmetric Website. https://www.altmetric.com. Accessed August 2019. 5. Elmore S. The Altmetric Attention Score: What Does It Mean and Why Should I Care?. Toxicol Pathol. 2018;46(3):252-255. 6. Journal Citation Reports, Clarivate Analytics. Available at: https://incites.clarivate.com. Accessed August 2019. 7. Chang J, Desai N, Gosain A. Correlation Between Altmetric Score and Citations in Pediatric Surgery Core Journals. Journal of Surgical Research. 2019;243:52-58. 8. Scopus Sources, Elsevier. Available at: https://www.scopus.com/sources. Accessed August 2019. 9. Nocera, A. P., Boyd, C. J., Boudreau, H., Hakim, O., & Rais-Bahrami, S. (2019). Examining the Correlation Between Altmetric Score and Citations in the Urology Literature. Urology. doi: 10.1016/j.urology.2019.09.014 10. Mullins, C. H., Boyd, C. J., & Corey, B. L. (2020). Examining the Correlation Between Altmetric Score and Citations in the General Surgery Literature. Journal of Surgical Research, 248, 159–164. Retrieved January 13, 2020, from 10.1016/j.jss.2019.11.008 https://www.scopus.com/sources.%20Accessed%20August%202019 https://www.scopus.com/sources.%20Accessed%20August%202019