Book review: Measuring research: What everyone needs to know The International Journal of Information, Diversity, & Inclusion, 3(4), 2019 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v3i4.32996 IJIDI: Book Review Sugimoto, C. R., & Larivière, V. (2018). Measuring research: What everyone needs to know. New York, NY: Oxford University Press. ISBN 9780190640125. 149 pp. $16.95 US. Reviewer: Norda A. Bell, York University, Canada Book Review Editor: Norda A. Bell, York University, Canada Keywords: bibliometrics; citation indexes; impact factors; inequality; research impact; research measurement Publication Type: book review “The allocation of resources on the basis of highly correlated metrics can exacerbate the Matthew effect and create barriers for underrepresented populations to receive due rewards and resources... However, it is a privilege of the elite to both create and reject indicators. Those who are well served by indicators generate cumulative advantage that propels them into the future. It is often those who are marginalized or are displaced by indicators who would benefit most from, but have the least opportunity to generate new metrics. The inequalities created and perpetuated by indicators must be responsibly addressed by the scientific community.” (p. 129) easuring Research: What Everyone Needs to Know (2018) by Cassidy R. Sugimoto and Vincent Larivière draws our attention to a rarely discussed aspect of diversity and inclusion in academic environments—inherent inequalities in research metrics that afford advantages to some groups while under-privileging others. The limitations of data and the biases of tools used to measure research are part and parcel of scholarly communication. Measuring research impact is not without its challenges, complexities, and controversies, especially in a neoliberal academic environment with a reward system largely reliant on metrics. Tenure and promotion, awards, funding, and even further recognition are based on citation counts or bibliometrics. However, traditional metrics serve some groups better than others. With the introduction of more accessible metric tools and the removal of analysts from the data and research evaluation, how can researchers, funders, and administrators interpret research indicators meaningfully and responsibly in an era of ranking, metrics, and performance evaluation in higher education? This book aims to describe “the ways in which these indicators are constructed, their strengths and weaknesses, and how they should be interpreted and used” (p. 1), focusing largely on bibliometrics and problematizing some of the ways research indicators are being used as proxies for research impact or research quality. In four chapters, each with headings in the form of questions, the authors attempt to outline what everyone needs to know about measuring research. Chapter 1 (“The Basics”) lays out the historical and theoretical foundations of measuring research as well as addressing such questions as “Why measure research?”, “What is an indicator?” and “What are the data sources for measuring research?”. Readers learn bibliometric data are skewed; a minority of researchers (20%) publish the majority of research (80%) (p. 11). Inherent inequality and disparities exist in an academic system which rewards the already prestigious and well-cited scholars (known as the Matthew effect, or the “rich get M https://jps.library.utoronto.ca/index.php/ijidi/index https://jps.library.utoronto.ca/index.php/ijidi/index Measuring Research: What Everyone Needs to Know The International Journal of Information, Diversity, & Inclusion, 3(4), 2019 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v3i4.32996 richer”) and penalizes female scholars, junior researchers, and precarious academic workers who tend to be less cited (the Matilda effect, or “the poor get poorer”). Chapter 2 (“The Data”), provides a general overview of citation indexes and covers the development of key databases Web of Science (WoS) and Scopus, as well as the freely available Google Scholar Citations. However, accessibility might be an issue for readers from international and smaller sized institutions or libraries that may not have access to these subscription databases. The authors compare these three databases with plenty of statistics on their size and coverage of materials as well as their strengths and limitations. The answer to the question, “What are the cultural biases of data sources?” will be of interest to IJIDI readers. It is refreshing to read that “Data sources for research indicators are not value-neutral. Rather, they are the products of historical, political, economic, and social contexts” (p. 38). It is important to understand these contexts. Biases in coverage by language and geography abound with 95% of WoS indexed papers written in English and the majority (nearly 25%) of authors are from the U.S. in both Scopus and WoS. A comparative analysis of WoS’ and Scopus’ coverage of authors by country, by the language of publication, and by discipline finds that Scopus has better non-Western coverage and Chinese language papers than WoS. Despite a lack of transparency regarding its data, Google Scholar Citations shows some evidence of this English language-bias as well. This chapter also highlights the problem of important national or local journals (e.g., in health or education) that may not have a high international impact, but are nonetheless of great significance domestically; it also describes the challenges of disciplinary classification and definitions across WoS and Scopus. In addition to language and geographic biases, disciplinary and format biases exist. Science and biomedicine are better represented than social sciences and humanities publications. Similarly, journal articles are better covered than other formats, such as books and conference papers. Better representation leads to more visibility of research and, inevitably, to more citations of particular works than others. How does one translate research activity into measurable units (or indicators)? Chapter 3 (“The Indicators”) is the heart of the book, with 23 questions centred on specific research indicators and metrics, including the journal impact factor, Eigenfactor Score, SCimago journal rank, h- index, and altmetrics (or alternative metrics); it reviews how each one is constructed and how to approach interpreting them. Interpretation can be challenging as impact indicators were traditionally based on bibliometric data from citation indexes. What is problematic when relying on this data (or counting citations) as a proxy for research impact is that often the results are dependent on the type of data used, which can be inherently biased, flawed, and incomplete. Furthermore, researchers are often not measuring what they think they are measuring. The authors demonstrate how difficult it is to actually “capture and operationalize” (p. 64) impact. Through very detailed examples and discussions, we learn that citations are indicators of usage, not necessarily impact or quality. Similarly, altmetrics, or alternative metrics, do not measure impact, but rather attention. The authors emphasize the importance of understanding what concepts are actually being measured and address some myths around measuring impact. The final chapter (“The Big Picture”) is thought-provoking as it examines the current scholarly communication landscape and the role of stakeholders such as academics, administrators, and funding agencies, and the adverse effects of measurement; in particular, it draws our attention to the fact that, in an incentivized world of academia, “research measurement has cultivated in scholars a ‘taste for rankings’ over a ‘taste for science’” (p. 127). The authors caution that 107 https://jps.library.utoronto.ca/index.php/ijidi/index https://jps.library.utoronto.ca/index.php/ijidi/index Measuring Research: What Everyone Needs to Know The International Journal of Information, Diversity, & Inclusion, 3(4), 2019 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v3i4.32996 measuring research at the individual level (author) is dangerous given the limitations of data and biases of tools. The authors also implore the scientific community to assess new data and tools and become knowledgeable of the inner workings of the indicators used, their limitations, and especially what they actually measure. I believe that this is especially important for those who are disadvantaged by traditional metrics: women, racialized groups, precarious academics, researchers from the Global South, those who publish in languages other than English, and in academic disciplines outside of science, technology, engineering, and health or medicine (STEM). Unfortunately, given the proprietary nature of many of these tools (e.g., WoS, Scopus, CiteScore, Journal Citation Reports, and Altmetrics) access to them for examination are limited to only those affiliated with large institutions or libraries, and/or those residing in Western countries, in short, those who can afford access to these tools; this leaves readers and researchers in the Global South and/or in smaller, cash-strapped institutions, disadvantaged. Surprisingly, the authors, hailing from the U.S. and Canada, address neither this limitation nor issues of access to these tools. “How do scholars measure research impact around the world?” would have been an interesting additional question to pose in the final chapter, especially for IJIDI readers. This is a very useful introductory text. After reading this book, readers will have an informed and in-depth knowledge of the history, structure, and limitations of WoS and Scopus, and various research metrics (both traditional and alternative). However, the readability of the text was an issue; the book is text-heavy, especially if you read it cover-to-cover. However, readers looking for quick answers to specific questions and reading this book selectively may not encounter the same challenge. There are many comparisons of the WoS and Scopus coverage throughout the book, but these are often textual. A chart providing an overview of these comparisons, perhaps with the addition of Google Scholar, would have been useful for improving readability, as would have using colours. However, this may have impacted the cost of the book (a reasonable $16.95 US for the paperback). The list of further readings is also useful for readers wishing to read up more on some of the fascinating studies and works mentioned in the main text. Overall, this book serves its purpose as an introductory text to research measurement. Issues of diversity and equality in scholarly communication and biases in well-established tools are seldom addressed, and this book fills the need. This text is highly recommended for library students, librarians, and scholars interested in understanding scholarly communication and research metrics at a foundational level. It is equally important for all who are inherently disadvantaged by metrics. It is vital to have a basic understanding of how these tools are constructed, what research is “counted,” and what is invisible. Once this is known, underrepresented groups, namely women, racialized persons, Global South authors, precarious academics, authors writing in languages other than English, and non-STEM researchers can counter the narratives told by traditional metrics to help mitigate some of these disadvantages. Norda Bell (nordam@yorku.ca) is Scholarly Publishing Librarian at York University. She is subject specialist for social work, human rights and equity studies, and linguistics & applied linguistics. Her current research focus includes diversity and equity in libraries, scholarly communications, and the representation of visible minorities in online spaces (Wikipedia and Wikidata). 108 https://jps.library.utoronto.ca/index.php/ijidi/index mailto:nordam@yorku.ca mailto:nordam@yorku.ca https://jps.library.utoronto.ca/index.php/ijidi/index