Hypothesis, Vol. 37, No. 1, 2025 Reply to Bakker et al.: Assessing the Accuracy of the Scite Citation Classification System Requires the Same Definitions to be Used for Training as for Testing Sean C. Rife, PhDa, Joshua M. Nicholson, PhDb, Ashish Uppala, MBAc, Domenic Rosati, MSd aHead of Academic Relations at Research Solutions, Co-founder of Scite, and Associate Professor of Psychology, Murray State University, Murray, KY, https://www.orcid.org/0000-0002-6748-0841 , srife@researchsolutions.com bChief Strategy Officer at Research Solutions and Co-founder of Scite, Brooklyn, NY, https://www.orcid.org/0000-0002-1111-1828 , jnicholson@researchsolutions.com cHead of Engineering and Product, Portal Innovations, LLC, Chicago, IL, https://www.orcid.org/0000-0001-8748-1465 , ashish.uppala.93@gmail.com dHead of AI, Research Solutions and Doctoral Candidate, Dalhousie University, Brooklyn, NY, https://orcid.org/0000-0003-2666-7615 , drosati@researchsolutions.com, Social Media Cite as: Rife SC, Nicholson JM, Uppala A, Rosati D. Reply to Bakker et al.: Assessing the Accuracy of the Scite Citation Classification System Requires the Same Definitions to be Used for Training as for Testing. Hypothesis. 2025;37(1). doi:10.18060/28018 Rife, Nicholson, Uppala, Rosati. All works in Hypothesis are licensed under a CC BY-NC 4.0 DEED Attribution-NonCommercial 4.0 International. Authors own copyright of their articles appearing in Hypothesis. Readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. For any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-6748-0841 https://orcid.org/ mailto:srife@researchsolutions.com https://www.orcid.org/0000-0002-1111-1828 https://orcid.org/ mailto:jnicholson@researchsolutions.com https://www.orcid.org/0000-0001-8748-1465 https://orcid.org/ mailto:ashish.uppala.93@gmail.com https://orcid.org/0000-0003-2666-7615 https://orcid.org/ mailto:drosati@researchsolutions.com https://creativecommons.org/licenses/by-nc/4.0/ Hypothesis, Vol. 37, No. 1, 2025 Abstract Bakker, Theis-Mahon, and Brown1 recently presented an analysis of citations in scite.ai, a citation analysis platform. They concluded that the Scite algorithm inaccurately classified citations, particularly those they regarded as supporting previous work. While we strongly believe that independent assessments of Scite’s classifications are valuable, we argue that Bakker et al.’s assessment is incorrect, and that their conclusions are due to their definition of what constitutes supporting or contrasting citations. Additionally, Bakker, et al. restricted their analyses to citations of retracted works in systematic literature reviews, which artificially limits the types of statements that could be considered supporting or contrasting a specific claim. In our reply, we document the rationale for Scite’s classification scheme. We also provide examples of how Scite classifies different types of citations, as well as how these classifications differ from those presented in Bakker et al. In a recent issue of Hypothesis, Bakker, Theis-Mahon, and Brown1 present an analysis of citation classifications by scite.ai, a platform that facilitates citation analyses using machine learning. They conclude that the accuracy of Scite’s classifications of citations is low, based on their own coding of citations. While there is value in this type of analysis, we believe there is a fundamental difference in how citation classifications are being coded and assessed; hence, the discrepancy from their analysis and our own. First, we wish to highlight some positive aspects of Bakker et al.’s1 work. As co-founders of Scite and practicing researchers, we appreciate any attempt to assess the accuracy of our citation classifications and the overall utility of the product suite Scite provides. We have published our own assessment and in-depth treatment of how Scite works elsewhere2, yet outside investigations are free from the biased attachment that comes with developing a new tool and are useful in the continual development and improvement of Scite. Additionally, we think the study of retracted publications is an interesting (and understudied) bibliometric topic; in fact, we have done our own investigation of citations of retracted publications3 and uncovered a number of concerning phenomena (e.g., authors citing retracted works while apparently unaware that they have been retracted). This is not mere dicta on our part - we genuinely believe efforts like those of Bakker et al.1 are valuable, and wish to see more of them in the future. However, Bakker et al.’s1 work has a critical flaw, in that it does not appear to assess what it claims it is assessing. Bakker et al. report that a citation was classified as supporting if it was included in a review “without indication of its retracted status or significant critique of the research quality,” and classified as contrasting if it was “described as retracted, or if concerns about the publication or underlying research were discussed” (p. 3). While these are perfectly valid attributes one may attach to a dataset of citation statements, they differ markedly from the annotations in Scite’s training dataset. In contrast, Scite’s classification schema is very different. As we note2: scite focuses on the authors’ reasons for citing a paper. (...) We consider that for capturing the reliability of a claim, a classification decision into supporting or contrasting must be backed by scientific arguments. The evidence involved in our assessment of citation intent is directed to the factual information presented in the 2 Hypothesis, Vol. 37, No. 1, 2025 citation context, usually statements about experimental facts and reproducibility results or presentation of a theoretical argument against or agreeing with the cited paper. (p. 888) This approach, which was part of the training given to annotators who created the labels in Scite’s training data differs markedly from that of Bakker et al.1 in two critical ways: first, noting or failing to note that a cited paper has been retracted does not indicate that the citing paper is presenting supporting or contrasting evidence. It simply looks at whether the authors are aware of the retraction or not, or whether or not they meaningfully engage with the cited work. Indeed, most debates or direct refutations in science have little to do with retractions; for example, there are over 64,000 retractions in the scholarly literature at the time of this writing3, while there are over 6.4 million contrasting citations in the Scite database. Second, while important, retractions are distinct from disagreements and distinct from looking at supporting or contrasting evidence. For example, in another independent study looking at disagreements in the literature, there was no mention of retractions or use of retractions in the definition of what is a scientific disagreement4. To provide concrete examples, Table 1 (reprinted from Nicholson et al.2) shows a sample of citations and how they are classified, as well as an explanation of each classification. We should note that Scite’s approach to citation classification is more precise than sentiment analysis (the automatic classification of statements as generally positive or generally negative), in that it focuses specifically on whether or not new evidence supports or contrasts a claim. As such, statements that refer to another paper with a positive or negative valence are deliberately classified by Scite as mentioning (see, for example, the fifth example in Table 1). 3 Hypothesis, Vol. 37, No. 1, 2025 Table 1. Examples of Citation Classifications Citation statement Classification Explanation “In agreement with previous work (Nicholson et al., 2015), the trisomic clones showed similar aberrations, albeit to a lesser extent (Supplemental Figure S2B).” Supporting “In agreement with previous work” indicates support, while “the trisomic clones showed similar aberrations, albeit to a lesser degree (Supplemental Figure S2B)” provides evidence for this supporting statement. “In contrast to several studies in anxious adults that examined amygdala activation to angry faces when awareness was not restricted (Phan, Fitzgerald, Nathan, & Tancer, 2006; Stein, Goldin, Sareen, Zorrilla, & Brown, 2002; Stein, Simmons, Feinstein, & Paulus, 2007), we found no group differences in amygdala activation.” Contrasting “In contrast to several studies” indicates a contrast between the study and studies cited, while “we found no group differences in amygdala activation” indicates a difference in findings. “The amygdala is a key structure within a complex circuit devoted to emotional interpretation, evaluation and response (Stein et al., 2002; Phan et al., 2006).” Mentioning This citation statement refers to Phan et al. (2006) without providing evidence that supports or contrasts the claims made in the cited study. “In social cognition, the amygdala plays a central role in social reward anticipation and processing of ambiguity [87]. Consistent with these findings, amygdala involvement has been outlined as central in the pathophysiology of social anxiety disorders [27], [88].” Mentioning Here, the statement “consistent with these findings” sounds supportive, but, in fact, cites two previous studies: [87] and [27] without providing evidence for either. Such cites can be valuable, as they establish connections between observations made by others, but they do not provide primary evidence to support or contrast the cited studies. Hence, this citation statement is classified as mentioning. “For example, a now-discredited article purporting a link between vaccination and autism (Wakefield et al., 1998) helped to dissuade many parents from obtaining vaccination for their children.” Mentioning This citation statement describes the cited paper critically and with negative sentiment but there is no indication that it presents primary contrasting evidence, thus this statement is classified as mentioning. 4 Hypothesis, Vol. 37, No. 1, 2025 In contrast, Bakker et al.’s methodology classifies citations based on methodological discussion and acknowledgement of a target article’s retraction. This results in many citations that Scite (justifiably, we believe, given the criteria outlined in Nicholson et al.) classifies as mentioning being classified as supporting. (A previous study5, which describes the data used in Bakker et al. in greater detail, does not provide access to the specific examples used in Bakker et al. As such, we present randomly-selected examples of citations to retracted works in systematic reviews.) Consider the following citation6 in Gatto (2020; emphasis added)7, which Scite classifies as mentioning: “Interestingly, recent studies have demonstrated that magnesium treatment protects cognitive function and synaptic plasticity by inhibiting GSK-3β in sporadic Alzheimer’s disease (AD) model rats (Xu et al, 2014). As such, Mg 2+ ions are a critical factor in controlling synapse density and plasticity, showing a reduction in Aβ-plaques and cognitive deficits in APPswe/PS1dE9 mice, a transgenic mouse model of AD (Li et al, 2014). However, the effects of dietary Mg 2+ deficiency on learning and memory are not followed by changes in the spine density and morphology of hippocampal neurons of rodent models (Serita et al, 2019), and further examination of this topic is undoubtedly needed. (p. 573)” Since this citation does not mention that Li et al. is retracted, nor does it contain a significant methodological critique, Bakker et al. would classify it as supporting. Another example can be found in a review by Zhang et al. (20218; emphasis added), citing Kaur (2014)9, which Scite also classifies as mentioning: “Some cross-sectional studies have shown a close association between thyroid disease and metabolic disorders (i.e., metabolic syndrome [MetS] and its components) (11–13). MetS is characterized by a cluster of abnormal metabolic parameters consisting of insulin resistance, central obesity, type 2 diabetes, impaired glucose tolerance, hyperinsulinemia, and dyslipidemia (14). The global prevalence of MetS is between 11.6% and 62.5% (15). (p. 2)” Again, because Kaur does not mention the fact that Zhang et al. was retracted, Bakker et al. would classify this citation as supporting. There are also instances where Bakker et al. would presumably classify a citation as contrasting while Scite would classify it as mentioning. For example, one paper10 cites Tsukumo et al. (2007; emphasis added)11, stating: “A total of 37% of the retrieved articles had a positive-citation pattern; meanwhile, 63% had a negative-citation pattern. The most cited article with a negative-citation-pattern was published in 2007 and was retracted in 2016 [24]. Thus far, it has received a total of 490 citations and of these, 58 were from after the retraction of the article. (p. 9)” Because the citing paper does not present findings contrary to Tsukumo et al., Scite classifies this citation as mentioning. However, because it acknowledges the retraction of the target article, Bakker et al. would classify this citation as contrasting. In summary, we think that while Bakker et al.’s1 efforts are laudable, their coding system bears no resemblance to that of the Scite training dataset. We therefore strongly disagree with 5 Hypothesis, Vol. 37, No. 1, 2025 Bakker et al.’s conclusion that the accuracy of Scite’s classifications is low. While we would all benefit from outside assessments of Scite’s accuracy, Scite included, this will have to come from future works that rely on a classification scheme that is the same, or at least similar, to that of Scite. Until such a paper is published, we recommend that readers consult our previous work2 for information on the accuracy of Scite’s classifications. Disclosure of Conflicts All authors are employees/contractors of and shareholders in Research Solutions, Inc., the company that owns Scite. References 1. Bakker C, Theis-Mahon N, Brown SJ. Evaluating the accuracy of scite, a smart citation index. Hypothesis: Research Journal for Health Information Professionals. 2023 Sep 13;35(2). doi:10.18060/26528 2. Nicholson JM, Mordaunt M, Lopez P, Uppala A, Rosati D, Rodrigues NP, Grabitz P, Rife SC. Scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. Quant Sci Stud. 2021 Nov 5;2(3):882-98. doi:10.1162/qss_a_00146 3. scite.ai: scite Search; [reviewed 2024 May 21; cited 024 May 21]. Available from: https://scite.ai/search?hasRetraction=true&mode=all 4. Lamers WS, Boyack K, Larivière V, Sugimoto CR, van Eck NJ, Waltman L, Murray D. Meta-Research: Investigating disagreement in the scientific literature. Elife. 2021 Dec 24;10:e72737. doi:10.7554/eLife.72737 5. Brown SJ, Bakker CJ, Theis-Mahon NR. Retracted publications in pharmacy systematic reviews. J Med Libr Assoc. 2022 Jan 1;110(1):47. doi:10.5195/jmla.2022.1280 6. Li W, Yu J, Liu Y, Huang X, Abumaria N, Zhu Y, Huang X, Xiong W, Ren C, Liu XG, Chui D. Elevation of brain magnesium prevents synaptic loss and reverses cognitive deficits in Alzheimer’s disease mouse model. Mol Brain. 2014 Dec;7(1):1-20. doi:10.1186/s13041-014-0065-y 7. Gatto RG. Molecular and microstructural biomarkers of neuroplasticity in neurodegenerative disorders through preclinical and diffusion magnetic resonance imaging studies. J Integr Neurosci. 2020 Sep 30;19(3):571-92. doi:10.31083/j.jin.2020.03.165 8. Zhang C, Gao X, Han Y, Teng W, Shan Z. Correlation between thyroid nodules and metabolic syndrome: a systematic review and meta-analysis. Front Endocrinol. 2021 Sep 16;12:730279. doi:10.3389/fendo.2021.730279 9. Kaur J. A comprehensive review on metabolic syndrome. Cardiol Res Pract. 2014 Oct;2014. doi:10.1155/2014/943162 6 https://scite.ai/ https://scite.ai/search?hasRetraction=true&mode=all Hypothesis, Vol. 37, No. 1, 2025 10. Stavale R, Ferreira GI, Galvão JA, Zicker F, Novaes MR, Oliveira CM, Guilhem D. Research misconduct in health and life sciences research: A systematic review of retracted literature from Brazilian institutions. PLoS One. 2019 Apr 15;14(4):e0214272. doi: 10.1371/journal.pone.0214272 11. Tsukumo DM, Carvalho-Filho MA, Carvalheira JB, Prada PO, Hirabara SM, Schenka AA, Araujo EP, Vassallo J, Curi R, Velloso LA, Saad MJ. Loss-of-function mutation in Toll-like receptor 4 prevents diet-induced obesity and insulin resistance. Diabetes. 2007 Aug 1;56(8):1986-98. doi:10.2337/db06-1595 7