52 Format Aside: Applying Beall’s Criteria to Assess the Predatory Nature of Both OA and Non-OA Library and Information Science Journals Joseph D. Olivarez, Stephen Bales, Laura Sare, and Wyoma vanDuinkerken* Jeffrey Beall’s blog listing of potential predatory journals and publishers, as well as his Criteria for Determining Predatory Open-Access (OA) Publishers are often looked at as tools to help researchers avoid pub- lishing in predatory journals. While these Criteria has brought a greater awareness of OA predatory journals, these tools alone should not be used as the only source in determining the quality of a scholarly journal. Employing a three-person independent judgment making panel, this study demonstrates the subjective nature of Beall’s Criteria by applying his Criteria to both OA and non-OA Library and Information Science jour- nals (LIS), to demonstrate that traditional peer-reviewed journals could be considered predatory. Many of these LIS journals are considered as top-tier publications in the field and used when evaluating researcher’s publication history for promotion and tenure. Introduction While the open access (OA) movement is growing throughout academia and has gained traction in some disciplines, there is still skepticism on the part of research- ers who question the quality of OA publishing and may associate OA journals with “predatory” journals. Jeffrey Beall defines “predatory” journals as those that “exploit the gold open-access model for their own profit” while not engaging in a rigorous peer review process.1 Beall developed a tool on his blog to help researchers identify potential predatory journals and publishers. This tool is a list of potential predatory journals and publishers, as well as the Criteria for Determining Predatory Open-Access Publishers. Joseph D. Olivarez is Serials Cataloging/Metadata Librarian, Stephen Bales is Humanities and Social Sciences Librarian, Laura Sare is Government Information Librarian, and Wyoma vanDuinkerken is Director of the Joint Library Facility, all at Texas A&M University Libraries; e-mail: jolivare@library. tamu.edu, sbales@library.tamu.edu, lsare@library.tamu.edu, wvanduin@library.tamu.edu. ©2018 Joseph D. Olivarez, Stephen Bales, Laura Sare, and Wyoma vanDuinkerken, Attribution-NonCommercial (http:// creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. doi:10.5860/crl.79.1.52 mailto:jolivare@library.tamu.edu mailto:jolivare@library.tamu.edu mailto:sbales@library.tamu.edu mailto:lsare@library.tamu.edu mailto:wvanduin@library.tamu.edu http://creativecommons.org/licenses/by-nc/4.0/ http://creativecommons.org/licenses/by-nc/4.0/ https://doi.org/10.5860/crl.79.1.52 Format Aside 53 The authors of this interpretive analysis (hereafter referred to as the “analysts”) believe that users need to be cautioned about the subjective nature of the Criteria. They recommend discretion because the criteria used are so general that an evaluator might label a scholarly journal as “predatory” when it is not or may disregard the article as being without merit solely because it is published in a journal found on Beall’s List. Promotion and tenure committees may even look to this list when reviewing tenure candidate publications, thereby influencing future research.2 Although Beall noted that such committees need to “decide for themselves how importantly or not to rate articles published in these journals in the context of their own institutional standards and/or geocultural locus,”3 there is the potential to misconstrue and misapply the Criteria in terms of its objectivity. Indiscriminate application could affect promotion and tenure decisions, especially those decisions concerning faculty members in emerging fields of scholarship where needs for publishing outlets are met with OA journals.4 In this study, the analysts argue that the individual elements of Beall’s Criteria are so general in design that evaluators might broadly apply them as measures of illegiti- macy to any refereed academic journal, regardless of publishing model. By applying Beall’s Criteria to a list of both OA and non-OA well-regarded refereed academic LIS journals, this study demonstrates the subjective nature of his Criteria. While the Criteria may serve as an aid to researchers’ selecting publication outlets, this Criteria is pres- ently not adequate for making such determinations because its elements focus more on the publisher and/or journal attributes rather than the quality of articles published by the outlet. Literature Review The rising cost of journal subscriptions at the end of the 20th century created a “serials crisis” rendering libraries unable to afford to purchase all of the scholarly literature needed to support the research and educational missions of their institutions. How- ever, by the 1990s, the World Wide Web became increasingly available to the general population and enabled the proliferation of online journals and, in turn, the growth of the OA movement. According to Suber, OA scholarship represents an attempt to circumvent traditional obstacles to access.5 This practice of making content freely available to access online is significantly different from the traditional for-profit publication model that libraries were required to deal with to purchase access to scholarly information. Some publishers have transformed themselves from being content providers to their readers to being service providers to their authors. These publishers charge authors a fee to publish in their journals as a way of offsetting publication costs.6 Charging publication fees to authors is not a new concept. Many print journals have been charging authors page charges to offset the cost of publishing.7 Shen and Björk contended that basic journal practices and elements of article publication, including peer review, have not changed appreciably.8 Because some OA journals charge a fee, despite having no print costs to cover, the practice of OA journals charging fees has opened up debate about questions of qual- ity, and whether authors are simply paying to have their work published (rather than publishing quality work). For example, a survey by Swan and Brown that compared the opinions of authors who had published in OA journals and those who had not yet published in OA showed that non-OA authors perceived OA journals “to be of lower reputation and prestige.”9 It can only be assumed that OA authors did not perceive OA journals the same way. Coonin and Younce also found, among social sciences and humanities researchers, the impression that “[OA] may be a less acceptable model, whether because research is often undertaken without substantial grant funding, or 54 College & Research Libraries January 2018 because (to some) it may sound a bit too close to ‘vanity press’ publishing.”10 John Bohannon highlighted his argument concerning OA quality in October 2013 when he performed his “sting operation” on OA journals. Bohannon composed a fraudulent paper about a lichen molecule that seemed to have cancer-fighting properties and submitted the paper to 304 OA journals. Even though he wrote the paper in a way that journal referees should have rejected it immediately, 157 OA journals accepted the paper.11 Davis pointed out that this high acceptance rate was not surprising, since “nearly half of the publishers were listed on Beall’s List of predator publishers,” further lending validity to Beall’s List as an authoritative standard.12 However, according to Bohannon, “Nearly half of all of the fee-charging open access publishers listed in the DOAJ [Directory of Open Access Journals] failed this test.”13 Because of this, the DOAJ is performing a reevaluation and cleanup of journals listed in their directory, having removed 3,300 of approximately 10,000 titles.14 Jeffrey Beall argues that there are some publishers (not all) manipulating this “author-pay model” for their own profit.15 Beall writes in his blog that these “fly-by- night” publishers operate as vanity presses for the sole purpose of generating income and do not follow professional publishing practices by disregarding peer review and digital preservation, and by spamming professional e-mail lists to solicit article submissions.16 Pickler et al. support Beall’s belief that predatory publishers engage in deceptive practices, including making false assertions about the publication’s status within a profession and providing inadequate or bogus information concerning the publisher or editorial board.17 According to Beall, in 2013 there were 225 publishers and 126 stand-alone journals he identified as “predatory,” while in 2015 this number increased to 693 publishers and 507 stand-alone journals.18 This indicated that the number of questionable journals and publishers is on the rise, causing much negative publicity for the OA movement. In an attempt to shed light and encourage discussion about this problem, Beall continues to develop a set of Criteria for Determining Predatory Open-Access Publishers to help researchers determine if a journal is predatory. He uses these criteria to update continually a list of publishers that he believes are “potential, possible, or probable predatory scholarly open-access publishers.”19 In addition to having developed the Criteria, Beall hosts a website on which he provides a list of publishers and stand-alone journals that he considers to have failed one or more of these criteria. Beall’s Criteria recognizes the (1) Code of Conduct for Journal Publishers and (2) Principles of Transparency of Best Practice in Scholarly Journals, both published by the Committee on Publication Ethics (COPE), as foundational documents.20 COPE, which adapted the latter docu- ment from one used by the DOAJ, uses it for the purpose of membership evaluation.21 It is important to note, however, that COPE membership is open to journal editors regardless of whether or not their journal is OA, and that these two documents are not written specifically to be applied to OA publishers.22 Beall’s Criteria for Determining Predatory Open-Access Publishers is currently in its third edition. It contains 27 criteria divided into four categories: editor and staff, business management, integrity, and other. Beall provides individual criteria under these categories but does not include further explanations or definitions of the criteria themselves. Beall also includes a category titled Poor Journal Standards that consists of 26 additional criteria. However, since, according to Beall, these additional criteria “are considered to be reflective of poor journal standards,” but “do not equal predatory criteria,”23 the researchers chose not consider them in this study. A number of researchers have criticized Beall’s List. Shen and Björk, for example, examined Beall’s List by using “a multistage stratified sampling method to take a look into the predatory publisher and journals on Beall’s List and generated their Format Aside 55 development trend over time.”24 They found that the problems caused by predatory journals are rather limited and regional, seeing the problem as created by institutions in developing countries trying to compete with the academic excellence viewpoint focusing on “how often one publishes” rather than on “what is published,” creating a market for dubious journals. Other researchers like Crawford believe Beall’s lists are subjective resources.25 Berger and Cirasella support Crawford’s findings, stating, “Imperfect English or a predominantly non-western editorial board does not make a journal predatory.”26 Despite these criticisms, the Criteria Beall uses to create his list is a good foundation for librarians to use when thinking about and identifying preda- tory journals and has been used by research as the foundation of critical evaluation of OA journals.27 Research Questions The purpose of this study was to assess the subjectivity of Beall’s Criteria for Determining Predatory Open-Access Publishers, the same criteria by which he determines his lists. This task was accomplished by applying the 27 criteria to a collection of LIS journals, both OA and non-OA, which are generally considered to be of high quality because of their inclusion in a recognized source for identifying academic journal rankings. Again, the analysts did not consider Beall’s category area of criteria titled Poor Journal Standards in this study. They were not looking to declare any of these LIS journals as “predatory.” On the contrary, they have published in several of the LIS journals from the collection. Instead, the analysts investigated whether respected and long-established journals would pass Beall’s Criteria as if they were newly emerging journals without an estab- lished reputation. The analysts asked the following questions during this assessment: 1. How many members of the population fail one or more of Beall’s Criteria? 2. What are the percentages of failure for Beall’s Criteria by area (editor and staff, business management, integrity, and other)? 3. What are the average ages of the failed journals? 4. What type of publisher (association, commercial, or university press) had the largest number of failed journals? 5. To what degree did the analysts agree in terms of their decision-making when applying Beall’s Criteria? 6. What do the previous questions tell us about Beall’s List as a tool for evaluating scholarly journals? Methodology To analyze the subjectivity of Beall’s Criteria, the analysts examined a collection of scholarly journals in the field of LIS to determine if evaluators of the individual titles could perceive them as having predatory journal tendencies when applying Beall’s Criteria. The collection of journals tested against Beall’s Criteria came from the InCites Journal Citations Reports (JCR) section in Thomson Reuters’ Web of Science database. The JCR provided a category, information science & library science, consisting of 87 entries, seven of which were OA journals. At the time of retrieval, these 87 entries comprised the entire population of the database’s information science & library sci- ence topic category. At the time of this study, the most recent year of JCR listings compiled was 2014. It was not the analysts’ intention to analyze the JCR collection to make statistically generalizable claims concerning the applications of Beall’s Criteria to any larger population of LIS journals. The analysts selected the JCR list because it provides a convenient ranked collection of journals that also represents an assumed level of quality in the LIS community through its ranking of journals having Thomson Reuters Impact Factors.28 56 College & Research Libraries January 2018 The analysts implemented an interpretive approach consisting of a three-person panel making independent judgments to determine whether a journal from the population of JCR LIS journals qualifies to be categorized as a “potential, possible, or probable predatory journal” according to Beall’s Criteria for Determining Predatory Open-Access Publishers.29 Other researchers have made use of panels of expert judges in both quantitative and qualitative research. Saljo, for example, used both quantita- tive and qualitative judging techniques to study learning approach and outcomes.30 Douglas, Douglas, McClelland, and Davies used a panel of three independent judges to determine levels of student satisfaction and dissatisfaction in U.K. higher educa- tion. Use of this technique allowed for making claims concerning the reliability of the judging process and the outcomes.31 During the course of a one-week period, the panel rated the JCR LIS entries sepa- rately. The three analysts are all academic librarians possessing ALA-accredited mas- ter’s degrees in LIS and 22 years’ total combined experience as professional librarians. They have authored a combined total of 23 refereed publications in peer-reviewed journals—four of which are published in journals listed in the JCR collection—and have themselves refereed 16 total articles for academic journals. The analysts were cognizant that their dual role as both authors and judges increased the potential for bias in any resulting analysis. They recognized, however, that bias is an inherent aspect of interpretive research and that conscious reflection throughout the research process is a method for mitigating such bias. To minimize bias, the analysts took part in a briefing session concerning the importance of reflexivity prior to beginning the analysis. The analysts applied the 27 Criteria for Determining Predatory Open-Access Publishers to each journal title. They performed this task independently of each other, with no ad- ditional instructions for how to apply the Criteria beyond those given in Beall’s Criteria itself. That is, they were each required to interpret and apply the Criteria themselves. None of the three analysts had previously served as either editors or board members for any of the journals under consideration. There was no time limit for applying the Criteria to journals during the evaluation process. There was no discussion allowed concerning possible interpretations of Beall’s Criteria, or as to whether or not a journal or its publisher met any of the 27 criteria of the Criteria. There was no discussion concerning the quality of any of the journals being reviewed. The only discussion allowed among the analysts during this period of review was only what was needed to determine and agree upon the identity of a particular listed journal if there was any doubt as to that identity (for example, if more than one journal exists with the same title). Analysts recorded a YES or NO judgment for each of the 27 criteria, with YES mean- ing that their analysis of the journal indicated that it fit the description of a “potential, possible, or probably predatory publisher” according to the particular criteria, and NO meaning that it did not sufficiently meet the individual analyst’s interpretation of the particular criterion. All three of the analysts were unaware of each others’ decisions throughout the entire judging process. Following completion of the qualitative judgment of the 87 entries, the three ana- lysts’ decisions were tabulated for each journal. Individual criterion were assigned an overall YES designation (meeting the criteria for a “potential, possible, or probable predatory journal”) or a NO designation (failing to meet the criteria for a “potential, possible, or probably predatory journal”) depending on which of the two possible outcomes received the majority of votes (that is, two or three votes out of three). The analysts recorded the number of YES versus NO votes for each journal for further analysis. Format Aside 57 Results Of the 87 journal titles analyzed, the analysts removed six entries from the study due to journal title duplication, title changes, or the determination that the entry did not actually represent a peer-reviewed academic journal. The remaining 81 entries counted as individual journal titles (see appendix for a list of the 81 journals evaluated). Of these 81 journals examined, 45 of the journals failed at least one of Beall’s Criteria (again, this determination of failure required at least two of three YES votes on the part of the analysts). A closer look at these 45 journals in table 1 shows that journals can and have failed more than one criteria in the study, with 18 of the 45 journals (45%) failing more than one criteria. Looking closer at the types of publishers: association, commercial, or university press, we see that all types of publishers failed at least one criterion (see table 2). In cases where journal publishers were affiliated with an academic institution, the analysts classified those as university presses. Commercial presses published the majority of failed journals (52%). It should be noted that the 17 instances of failed journals does not represent 17 different commercial presses, but that cer- tain presses had multiple failed journal titles. Table 3 shows the number of criteria failed by age of journal, demonstrating that all of the journals that failed a criterion are at least 10 years old. The median age of journals in the ISI example is 32 years. Beall divides his Criteria into four criteria areas. Table 4 highlights the percentage of total failed journals in each criteria area. The criteria area for editor and staff, which has nine specific criteria, had the largest failure percentage (77.78%). The criteria area for integrity, which encompasses seven specific criteria, was not the cause for any journal failure. Table 5 illustrates the number of journals that could be perceived as predatory by the top five failing criteria. The most failed criterion was under the criteria area business management: “Has no policies or practices for digital preservation, meaning that if the journal ceases operations, all of the content disappears from the internet.” Twenty-two journals failed this criterion. Three out of five of the top five most failed criteria fell into the editor and staff criteria area. TABLE 1 Journals That Failed One or More Criteria Number of Journals Number of Criteria Failed 27 1 10 2 4 3–6 4 7–9 TABLE 2 Types of Publishers That Failed One or More Criteria Type of Publisher Number of Journals That Failed Association Press 8 Commercial Press 17 University Press 8 TABLE 3 Age and Number of Journals that Failed One or More Criteria Age of the Journal Number of Journals Total Number of Criteria Failed 50–150 years old 9 28 30–49 years old 18 21 10–29 years old 18 49 0–9 years old 0 0 58 College & Research Libraries January 2018 Interanalyst Agreement Statistics The analysts calculated interanalyst agreement statistics for individual criterion as well as the criteria areas. They excluded two individual criteria from the ranking because they had too many missing cells to get an accurate agreement percentage rating. These ex- cluded criteria were “The publisher falsely claims to have its content indexed in legitimate abstracting and indexing services or claims that its content is indexed in resources that are not abstracting and indexing services” (criteria area: integrity) and “Publish papers that are not academic at all, e.g. essays by laypeople, polemical editorials, or obvious pseudo-science” (criteria area: other). All three analysts agreed either YES or NO 87.5 percent of the time. According to table 6, the most disagreement occurred in the criteria area of editor and staff (76%). The highest area of agreement was integ- rity (98%). TABLE 6 Criteria Ranked From Least Referee Agreement to Most Criteria Area % of Agreement editor and staff 76% business management 86% integrity 98% other 90% TABLE 5 Number of Failing Journals in the Top Five Failed Criteria Failed Criteria Criteria Area Number of Failing Journals Has no policies or practices for digital preservation, meaning that if the journal ceases operations, all of the content disappears from the internet. business management 22 No single individual is identified as any specific journal’s editor. editor and staff 17 No academic information is provided regarding the editor, editorial staff, and/or review board members (such as institutional affiliation). editor and staff 13 There is little or no geographical diversity among the editorial board members, especially for journals that claim to be international in scope or coverage. editor and staff 12 Have a “contact us” page that only includes a web form or an e-mail address, and the publisher hides or does not reveal its location. other 11 TABLE 4 Percentage of Total Failed Journals by Criteria Criteria Area (Number of Criteria in Area) % of Total Failed Journals editor and staff (9) 77.78% business management (6) 33.34% integrity (7) 0% other (6) 16.67% Format Aside 59 Table 7 lists the individual criteria in order of percentage of agreement. The highest percentage of disagreement occurred for the criterion, “the publisher’s owner is identi- fied as the editor of each and every journal published by the organization” (criteria area: editor and staff). TABLE 7 Individual Criteria in Order of Percentage of Agreement Individual Criteria Criteria Area Analysts Agree % The publisher’s owner is identified as the editor of each and every journal published by the organization. editor and staff 0 Have a “contact us” page that only includes a web form or an e-mail address, and the publisher hides or does not reveal its location. other 57 Has no policies or practices for digital preservation, meaning that if the journal ceases operations, all of the content disappears from the internet. business management 60 There is little or no geographical diversity among the editorial board members, especially for journals that claim to be international in scope or coverage. editor and staff 76 No single individual is identified as any specific journal’s editor; No academic information is provided regarding the editor, editorial staff, and/or review board members (such as institutional affiliation). editor and staff editor and staff 81 Provides insufficient information or hides information about author fees, offering to publish an author’s paper and later sending an unanticipated “surprise” invoice. business management 84 The journal does not identify a formal editorial / review board. Demonstrates a lack of transparency in publishing operations. editor and staff business management 86 Evidence exists showing that the editor and/or review board members do not possess academic expertise to reasonably qualify them to be publication gatekeepers in the journal’s field. The editorial board engages in gender bias (i.e., exclusion of any female members). editor and staff editor and staff 88 The journals have an insufficient number of board members,(e.g., 2 or 3 members),have concocted editorial boards (made up names), name scholars on their editorial board without their knowledge or permission or have board members who are prominent researchers but exempt them from any contributions to the journal except the use of their names and/ or photographs. editor and staff 87 Does not allow search engines to crawl the published content, preventing the content from being indexed in academic indexes. business management 92 60 College & Research Libraries January 2018 TABLE 7 Individual Criteria in Order of Percentage of Agreement Individual Criteria Criteria Area Analysts Agree % Two or more journals have duplicate editorial boards (i.e., same editorial board for more than one journal). Provides insufficient information or hides information about author fees, offering to publish an author’s paper and later sending an unanticipated “surprise” invoice. editor and staff business management 94 The publisher asks the corresponding author for suggested reviewers and the publisher subsequently uses the suggested reviewers without sufficiently vetting their qualifications or authenticity. (This protocol also may allow authors to create faux online identities to review their own papers.) integrity 95 The publisher dedicates insufficient resources to preventing and eliminating author misconduct, to the extent that the journal or journals suffer from repeated cases of plagiarism, self-plagiarism, image manipulation, and the like. Use boastful language claiming to be a “leading publisher” even though the publisher may only be a startup or a novice organization. integrity other 97 Begins operations with a large fleet of journals, often using a common template to quickly create each journal’s homepage. The name of a journal is incongruent with the journal’s mission The name of a journal does not adequately reflect its origin (such as a journal with the word “Canadian” or “Swiss” in its name when neither the publisher, editor, nor any purported institutional affiliate relates whatsoever to Canada or Switzerland). In its spam e-mail or on its website, the publisher falsely claims one or more of its journals have actual (Thomson-Reuters) impact factors, or advertises impact factors assigned by fake “impact factor” services, or it uses some made-up measure (such as view factor), feigning/claiming an exaggerated international standing. Republish papers already published in other venues/outlets without providing appropriate credits Operate in a Western country chiefly for the purpose of functioning as a vanity press for scholars in a developing country (such as using a mail-drop address or P.O. box address in the United States, while actually operating from a developing country). Provide minimal or no copyediting or proofreading of submissions. business management integrity integrity integrity other other other 99 Format Aside 61 Discussion What do the data tell us about Beall’s List as a tool for evaluating both OA and tra- ditional model scholarly journals? First, might evaluators consider these established, well-regarded LIS journals as possible predatory journals? According to the analysts’ application of Beall’s Criteria, yes, even traditional model journals might classify as predatory. Again, the panel was not applying the criteria to identify LIS journals from the JCR collection as predatory (and to avoid any misreading of method, this research report does not directly identify by name individual journals with how they fared against any of the criteria). The primary motivation of the exercise was to determine the validity of Beall’s Criteria and whether these journals, which are arguably considered high quality and high impact, would fail some elements of Beall’s Criteria. Consequently, the present authors question if Beall’s List suggests a possible bias against OA journals and favors traditional model journals in a way that is not defensible, which previous research has also addressed.32 As a result, the authors find it necessary to consider why Beall’s Criteria aims predominantly at OA publishers when established journals based on traditional models can fail at a high rate. Future research might analyze a list of OA journals generally considered to be of high quality to compare traditional and OA models according to Beall’s Criteria. Second, although Beall acknowledges the subjectivity of his Criteria, going as far as to state that his lists consist solely of his opinion,33 this study affirms this subjectiveness and raises questions concerning their application to publisher and journal evaluation. Due to the rate of disagreement, the use of a three-person, independently working panel of analysts highlights the subjectiveness of Beall’s Criteria. This calls into ques- tion its indiscriminate use by librarians who provide the list to students and faculty as a resource without explaining the limitations of the list and of the Criteria. This is especially important regarding promotion and tenure committees across academic institutions who may use Beall’s Criteria as a way to disqualify articles published in journals that the committee determined fail some Criteria but are not truly predatory. Evaluators should consider a variety of resources when determining the quality of a journal, publisher, or article. Such tools might include JCR, the DOAJ, the Open Access Scholarly Publishers Association, Cabell’s Directory of Publishing Opportunities, and subject specialist knowledge. The present authors recognize that there are predatory journals in the OA publish- ing model. However, it would be prudent to caution researchers to understand that, as noted by Berger and Cirasella, there are also low-quality journals found in the traditional publishing model.34 Therefore, the present authors recommend that librar- ians remain cognizant of these liabilities, but they agree with the conclusion of Smith that librarians should stress to users that OA and traditional journals deserve equal consideration when making decisions concerning quality.35 Beall’s Criteria serves an important purpose as a tool for information literacy and evaluation. However, since Beall ties it directly to his predatory journal and publish- ers lists, the Criteria becomes counterproductive to the two objectives of information literacy and evaluation. Because evaluators often interpret the lists as determinative of a journal or publisher’s predatory status, this impedes the application of the Criteria because it represents the subjective employment of the Criteria by Beall himself. The indiscriminate use of the Criteria and the lists derived from it effectively obscures this subjective nature and alienates the reader from the process. Additionally, the title of Beall’s benchmarking tool, Criteria for Determining Predatory Open-Access Publishers, is misleading to the reader. The title’s orientation toward only OA journals potentially biases readers against OA publishers and journals. In addition, Beall’s use of “Criteria for Determining” in the title of his evaluation tool further muddies the subjective nature 62 College & Research Libraries January 2018 of the Criteria. According to the Oxford English Dictionary, to “determine” something means “To end in (a termination, conclusion, or result)” as well as “To bring to an end a dispute, controversy, or doubtful matter; to conclude, settle, decide, fix.”36 The present authors argue that, while the Criteria may serve as an aid to this end, it is presently not adequate for making such determinations and they question the limited scope of the Criteria. Information literacy and evaluation goes beyond the structure of the journal and its publishing practices to include the assessment of the actual information content. This is extremely important for the work of promotion and tenure committees, as they need to assess the actual information content rather than simply relying on a subjective criterion such as “There is little or no geographical diversity among the editorial board members, especially for journals that claim to be international in scope or coverage.” Beall’s Criteria does not adequately account for the subject knowledge and subject evaluation expertise necessary for determining the quality of the published content; instead, it focuses on traditional publication standards as interpreted through the lens of a librarian (specifically the lens of Beall and all of his possible biases). Therefore, while the authors of this study are not devaluating Beall’s Criteria, they advocate the development of additional criteria that focus on subject content, par- ticularly if evaluators might use such a list of criteria as a means for making decisions regarding promotion and tenure across university departments. Conclusion This study demonstrates the subjective nature of the Criteria by which Beall constructs his lists. Furthermore, it highlights the finding that well-regarded academic journals, whether OA or not, can be considered as possible predatory journals, even when LIS professionals apply the Criteria. The researchers recommend placing emphasis upon training evaluators to apply the Criteria itself, as opposed to relying on a list of journals derived from the Criteria’s application developed under one person’s opinion. The problem with relying on any list is that when an information specialist presents a list to a reader, the reader might view this list uncritically, assuming it to be 100 percent correct. The reader may rely solely on the list, even with the author’s “warnings,” and not consider the information content presented within the journal itself. However, as this study has demonstrated, even librarians disagree about the interpretation and application of the Criteria, and this study’s three analysts did not even consider the quality of the content found within the pages of the journals. After all, scholarly com- munication is ultimately driven by the information within the journal. The researchers put forth that only experts in a specific field of study can truly identify fraudulent information in their field of expertise, whether or not that journal finds itself on any predatory journal list. However, if a list of potential predatory publishers and journals is given, an evalu- ation process needs to be developed that is transparent to the reader so that the reader will know and understand why certain criteria did not pass muster and, therefore, why the journal failed the evaluation. Beall’s lists are deficient in this area, as he provides only names and links to journal publisher websites that have failed. Identifying the reasons a journal failed would shine light on the Criteria and would help the reader to reconsider if the journal is truly predatory, possibly causing them to further consider the information within the journal. This study shows the subjective nature of Beall’s Criteria, as well as the subjective nature of the lists created from the application of the Criteria. Such criteria provide a starting point for a discussion on predatory aspects of academic publishing. Neverthe- less, as librarians, our duty is to refrain from offering up these lists as the final word on predatory journals. Rather, it is our responsibility to (1) use such lists and criteria as Format Aside 63 tools for teaching faculty to be proactive about evaluating what journals to publish in and (2) to ensure that newer journals, which are often OA, are not disqualified unfairly from consideration as part of quality scholarly output. This latter point is especially important if promotion and tenure committees use Beall’s Criteria without consider- ing the subjective nature of their application, or without also including supplemental evaluative measures of journal quality. The authors of this paper concur with Berger and Cirasella that librarians “are key stakeholders in scholarly and professional con- versations reimagining various aspects of scholarly communication,”37 and the present study emphasizes this role in the area of predatory publishing. 64 College & Research Libraries January 2018 Appendix: InCites Journal Citation Reports Information Science & Library Science Category 1 African Journal of Library Archives and Information Science 2 Aslib Journal of Information Management 3 ASLIB Proceedings 4 Australian Academic & Research Libraries 5 Australian Library Journal 6 Canadian Journal of Information and Library Science: La Revue Canadienne ses sciences de l’information et de bibliothéconomie 7 College & Research Libraries 8 Database for Advances in Information Systems 9 Electronic Library 10 Ethics and Information Technology 11 European Journal of Information Systems 12 Government Information Quarterly 13 Health Information and Libraries Journal 14 Informacao & Sociedade-Estudos 15 Informacios Tarsadalom 16 Information & Culture 17 Information & Management (Elsevier) 18 Information and Organization 19 Information Development 20 Information Processing & Management 21 Information Research: An International Electronic Journal 22 Information Society 23 Information Systems Journal 24 Information Systems Research 25 Information Technology & People 26 Information Technology and Libraries 27 Information Technology for Development 28 International Journal of Computer-Supported Collaborative Learning 29 International Journal of Geographical Information Science 30 International Journal of Information Management 31 Investigacion Bibliotecologica 32 Journal of Academic Librarianship 33 Journal of Computer-Mediated Communication 34 Journal of Documentation Format Aside 65 35 Journal of Global Information Management 36 Journal of Global Information Technology Management 37 Journal of Health Communication 38 Journal of Information Science 39 Journal of Information Technology 40 Journal of Informetrics 41 Journal of Knowledge Management 42 Journal of Librarianship and Information Science 43 Journal of Management Information Systems 44 Journal of Organizational and End User Computing 45 Journal of Scholarly Publishing 46 Journal of Strategic Information Systems 47 Journal of the American Medical Informatics Association 48 Journal of the Association for Information Science and Technology 49 Journal of the Association for Information Systems 50 Journal of the Medical Library Association 51 Knowledge Management Research & Practice 52 Knowledge Organization 53 Law Library Journal 54 Learned Publishing 55 Library & Information Science Research 56 Library Collections Acquisitions & Technical Services 57 Library Hi Tech 58 Library Journal 59 Library Quarterly 60 Library Resources & Technical Services 61 Library Trends 62 Libri 63 Malaysian Journal of Library & Information Science 64 MIS Quarterly 65 MIS Quarterly Executive 66 Online Information Review 67 portal: Libraries and the Academy 68 Profesional de la Informacion 69 Program-Electronic Library and Information Systems 70 Reference & User Services Quarterly 71 Research Evaluation 66 College & Research Libraries January 2018 72 Restaurator-International Journal for the Preservation of Library and Archival Material 73 Revista Espanola de Documentacion Cientifica 74 Scientometrics 75 Serials Review 76 Social Science Computer Review 77 Social Science Information sur les sciences sociales 78 Telecommunications Policy 79 Telematics and Informatics 80 Transinformacao 81 Zeitschrift fur Bibliothekswesen und Bibliographie Note: The journals are listed in alphabetical order after six entries were removed from the study due to duplication, ceased publication, or the perception that the entry did not represent a peer-reviewed academic journal. Notes 1. Jeffrey Beall, “Predatory Publishing: Overzealous Open-Access Advocates Are Creating an Exploitative Environment, Threatening the Credibility of Scholarly Publishing,” The Scientist 26, no. 8 (2012), available online at www.the-scientist.com/?articles.view/articleNo/32426/title/ Predatory-Publishing [accessed 1 December 2016]. For a description of the degrees of OA, includ- ing “gold,” see Peter Suber’s “Open Access Overview,” available online at http://legacy.earlham. edu/~peters/fos/overview.htm [accessed 1 December 2016]. 2. Martin S. Fiebert, “A Look at Open Access Publication and Beall’s List of ‘Predatory’ Jour- nals,” Global Journal of Interdisciplinary Social Sciences 3, no. 4 (2014): 5–6. 3. Jeffrey Beall, “List of Publishers,” previously available online but has since been removed. See https://www.insidehighered.com/news/2017/01/18/librarians-list-predatory-journals-report- edly-removed-due-threats-and-politics [accessed 11 December 2017]. 4. Diane Harley, Sophia Krzys Acord, Sarah Earl-Novell, Shannon Lawrence, and C. Judson King, “Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines” (UC Berkeley: Center for Studies in Higher Education, 2010), available online at https://escholarship.org/uc/item/15x7385g [accessed 1 December 2016]. 5. Peter Suber, “Open Access Overview,” available online at http://legacy.earlham.edu/~peters/ fos/overview.htm [accessed 24 January 2016]. 6. Charlotte Haug, “The Downside of Open-Access Publishing,” New England Journal of Medicine 368, no. 9 (2013): 791–93, doi:10.1056/NEJMp1214750. 7. Sally Morris, “The True Costs of Scholarly Journal Publishing,” Learned Publishing 18, no. 2 (2005): 115–26. 8. Cenyu Shen and Bo-Christer Björk, “‘Predatory’ Open Access: A Longitudinal Study of Article Volumes and Market Characteristics,” BMC Medicine 13, no. 1 (2015): 1, doi:10.1186/s12916- 015-0469-2. 9. Alma Swan and Sheridan Brown (Key Perspectives, Ltd.), “Joint Information Systems Committee ( JISC) and the Open Society Institute (OSI) Journal Authors Survey Report” (2004), available online at www.jisc.ac.uk/media/documents/themes/infoenvironment/acf655.pdf [ac- cessed 1 December 2016]. 10. Bryna Coonin and Leigh Younce, “Publishing in Open Access Journals in the Social Sci- ences and Humanities: Who’s doing it and Why?” in Pushing the Edge: Proceedings of the Fourteenth Annual Meeting of the Association of College and Research Libraries, March 12-15, ed. Dawn Mueller (Chicago: American Library Association, 2009), 85-94. 11. John Bohannon, “Who’s Afraid of Peer Review,” Science 342, no. 6154 (2013): 60-65. 12. Phil Davis, “Post Open Access Sting: An Interview with John Bohannon,” The Scholarly Kitchen, available online at http://scholarlykitchen.sspnet.org/2013/11/12/post-open-access-sting- an-interview-with-john-bohannon/ [accessed 25 January 2016]. 13. Ibid. http://www.the-scientist.com/?articles.view/articleNo/32426/title/Predatory-Publishing http://www.the-scientist.com/?articles.view/articleNo/32426/title/Predatory-Publishing http://legacy.earlham.edu/~peters/fos/overview.htm http://legacy.earlham.edu/~peters/fos/overview.htm https://www.insidehighered.com/news/2017/01/18/librarians-list-predatory-journals-reportedly-removed-due-threats-and-politics https://www.insidehighered.com/news/2017/01/18/librarians-list-predatory-journals-reportedly-removed-due-threats-and-politics https://escholarship.org/uc/item/15x7385g http://legacy.earlham.edu/~peters/fos/overview.htm http://legacy.earlham.edu/~peters/fos/overview.htm https://doi.org/10.1056/NEJMp1214750 http://www.jisc.ac.uk/media/documents/themes/infoenvironment/acf655.pdf http://scholarlykitchen.sspnet.org/2013/11/12/post-open-access-sting-an-interview-with-john-bohannon http://scholarlykitchen.sspnet.org/2013/11/12/post-open-access-sting-an-interview-with-john-bohannon Format Aside 67 14. “DOAJ News Service: News, Updates & Developments from DOAJ,” available online at https://blog.doaj.org/ [accessed 11 December 2017]. 15. Jeffrey Beall, “Predatory Publishers Are Corrupting Open Access,” Nature 489, no. 7415 (2012): 179, doi:10.1038/489179a. 16. Jeffrey Beall, “Scholarly Open Access,” available online at http://scholarlyoa.com [accessed 25 January 2016]. 17. Rita Pickler, Jane Noyes, Lin Perry, Brenda Roe, Roger Watson, and Mark Hayter, “Authors and Readers Beware the Dark Side of Open Access,” Journal of Advanced Nursing (2014), doi:10.1111/ jan.12589. 18. Jeffrey Beall, “Beall’s List of Predatory Publishers 2015,” previously available online but has since been removed. See https://www.insidehighered.com/news/2017/01/18/librarians-list- predatory-journals-reportedly-removed-due-threats-and-politics [accessed 11 December 2017]. 19. Ibid. 20. Committee on Publication Ethics, “Principles of Transparency and Best Practice in Scholarly Publishing,” available online at http://publicationethics.org/files/Principles%20of%20Transpar- ency%20and%20Best%20Practice%20in%20Scholarly%20Publishing.pdf [accessed 22 January 2016] and “Code of Conduct,” Code of Conduct, available online at http://publicationethics.org/ files/Code%20of%20conduct%20for%20publishers%20FINAL_1_0_0.pdf [accessed 22 January 2016]. 21. Committee on Publication Ethics, “Principles of Transparency and Best Practice in Scholarly Publishing.” 22. Committee on Publication Ethics, “About COPE,” available online at http://publicationeth- ics.org/about [accessed 22 January 2016]. 23. Ibid. 24. Shen and Björk, “‘Predatory’ Open Access,” 1. 25. Ibid, 28–31. 26. Walt Crawford, “Ethics and Access 1: The Sad Case of Jeffrey Beall,” Cites & Insights 14, no. 4 (2014): 1–14; Monica Berger and Jill Cirasella, “Beyond Beall’s List: Better Understanding Predatory Publishers,” College & Research Libraries News 76, no. 3 (2015): 132–35. 27. Nerissa Nelson and Jennifer Huffman, “Predatory Journals in Library Databases: How Much Should We Worry?” Serials Librarian 69, no. 2 (2015): 169–92, doi:10.1080/036152 6X.2015.1080782. 28. James Testa, “The Thomson Reuters Journal Selection Process,” available online at http:// wokinfo.com/essays/journal-selection-process/ [accessed 10 February 2016]. 29. Jeffrey Beall, Criteria for Determining Predatory Open-Access Publishers, 3rd ed. (2015), avail- able online at https://scholarlyoa.files.wordpress.com/2012/11/criteria-2012-2.pdf [accessed 1 December 2016]. 30. Roger Saljo, “Learning Approach and Outcome: Some Empirical Observations,” Instructional Science 10 (1981): 47–65, doi:10.1007/BF00124566. 31. Jacqueline Ann Douglas, Alexander Douglas, Robert James McClelland, and John Davies, “Understanding Student Satisfaction and Dissatisfaction: An Interpretive Study in the UK Higher Education Context,” Studies in Higher Education 40, no. 2 (2015): 329–49, doi:10.1080/03075079.20 13.842217. 32. Crawford, “Ethics and Access 1: The Sad Case of Jeffrey Beall,” 1–14. 33. Kristen Wilson, “Librarian vs. (Open Access) Predator: An Interview with Jeffrey Beall,” Serials Review 39, no. 2 (2013): 125-28); Jeffrey Beall, “Disclaimer,” available online at https://jef- freybeall.com/disclaimer/ [accessed 11 December 2017]. 34. Monica Berger and Jill Cirasella, “Beyond Beall’s List: Better Understanding Predatory Publishers,” College & Research Libraries News 76, no. 3 (2015): 132–35. 35. Beverly Dowdy, David Crotty, Beth Bernhardt, Paige Hall Smith, and Jan Mayo, “Chal- lenges and Opportunities of Open Access: A Panel Discussion,” Serials Review 40 (2014): 188–90. 36. Oxford English Dictionary Online, s.v. “determine,”available online at www.oed.com/ view/Entry/51244?redirectedFrom=determine [accessed 7 April 2016]. 37. Berger and Cirasella, “Beyond Beall’s List,” 135. https://blog.doaj.org/ http://scholarlyoa.com https://doi.org/10.1111/jan.12589 https://doi.org/10.1111/jan.12589 https://www.insidehighered.com/news/2017/01/18/librarians-list-predatory-journals-reportedly-removed-due-threats-and-politics https://www.insidehighered.com/news/2017/01/18/librarians-list-predatory-journals-reportedly-removed-due-threats-and-politics http://publicationethics.org/files/Code%20of%20conduct%20for%20publishers%20FINAL_1_0_0.pdf http://publicationethics.org/files/Code%20of%20conduct%20for%20publishers%20FINAL_1_0_0.pdf http://publicationethics.org/about http://publicationethics.org/about http://wokinfo.com/essays/journal-selection-process/ http://wokinfo.com/essays/journal-selection-process/ https://scholarlyoa.files.wordpress.com/2012/11/criteria-2012-2.pdf https://doi.org/10.1007/BF00124566 http://www.oed.com/view/Entry/51244?redirectedFrom=determine http://www.oed.com/view/Entry/51244?redirectedFrom=determine _GoBack