Automated Fact-Checking to Support Professional Practices: Systematic Literature Review and Meta-Analysis American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 1 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC THE PROMISE AND CHALLENGES OF AUTOMATED FACT-CHECKING FOR PROFESSIONAL PRACTICES Kari Anne Olsen, Stian Andersen, Kristine Berg University of Bergen, Norway Abstract:The prevalence of disinformation, misinformation, and propaganda online has been on the rise, with multiple political, societal, and cultural implications. Automated fact-checking (AFC) has increasingly garnered interest in research over the past years as a way to address this problem. AFC systems use natural language processing and machine learning techniques to identify and verify claims made online. While AFC systems have shown promise in some areas, they still face challenges, such as the ability to handle complex claims and the need for human supervision. Keywords: Automated fact-checking (AFC), Disinformation, Misinformation, Propaganda, Natural language processing (NLP) Introduction The prevalence of disinformation, misinformation, and propaganda online has been on the rise, accompanied by multiple political, societal, and cultural implications (Kalsnes, 2018). As such, automated fact- checking (AFC) has increasingly garnered interest in research over the past years (Guo et al., 2022). These different forms of information disorder are not new to the world’s history (Kalsnes, 2018; Posetti & Matthews, 2018), but online social interactions contribute to their large-scale accessibility and visibility, considering the central role played by social media and platforms (Guess et al., 2020). Even though the verification of information has always been a part of editorial processes, it was popularized as a prominent subgenre in recent years (Singer, 2021), referring to “good journalism” (Singer, 2018) or “unbiased journalism” (Graves, 2018). At the same time, the fact-checking community also encompasses political activists or fact-checkers acting on behalf of social progress (Mena, 2019). Although they share the same idea of defending and practicing a form of accountability reporting, these actors may view this ideal differently (Graves, 2018). Such heterogeneity is challenging for AFC tool designers, particularly in defining the requirements or needs of potential end users whose various organizational contexts and professional standards contribute to shaping their practices. In this article, we define the end user as a fact-checker employed either in a fact-checking organization in a newsroom model or in a newsroom alongside or not alongside other journalistic activities. These professionals assimilate their identity and professional practices into journalism, considering that factchecking is a genre of journalism (Cavaliere, 2020; Graves & Cherubini, 2016). Therefore, the professional values that frame their practices are linked to the traditional ethical values of journalism—accuracy, fairness, and objectivity (Frost, mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 2 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC 2015)—that support the social responsibility of the journalists regarding the audiences to whom they are accountable (Bardoel & d’Haenens, 2004). These principles are strongly connected to the transparency defended in the ethical principles of the International Fact-Checkers Network (IFCN; Mena, 2019) and the European Fact- Checking Standards Network, which bring together fact-checkers to promote excellence in fact-checking. Considering the speed at which misinformation and disinformation spread, AFC can be fast and effective tools not only to find claims worth fact-checking, relevant previously fact-checked claims, or supporting evidence but also to translate or summarize content (Nakov et al., 2021). The four stages of manual fact-checking are extracting statements, constructing appropriate questions, obtaining answers from relevant sources, and reaching a verdict using these answers (Vlachos & Riedel, 2014). Concretely, they encompass four primary tasks: monitoring media and capturing content, detecting claims, checking claims, and publishing content (Konstantinovskiy, Price, Babakar, & Zubiaga, 2021). These activities require the conjugation of critical thinking and know-how related to online investigative journalism and multimedia forensics. The boundary object (BO) theory allows us to approach AFC tools as technological artifacts that exist between adjacent communities of designers and end users. According to this theory, as boundary objects, AFC tools “negotiate meaning to understand and articulate connections and disconnections between communities, cultures, and information infrastructures” (Huvila, Anderson, Jansen, Mckenzie, & Worrall, 2017, p. 1812). The BO theory can be seen as a valuable instrument to highlight the issues that researchers face when technology and journalism intertwine because it involves distinct occupational and social worlds that do not necessarily share the same values or visions of journalism (Lewis & Usher, 2016; Sirén-Heikel, Kjellman, & Lindén, 2022). From the point of view of the approach to the relationship between design and use (or between developers and fact- checkers), these issues are potentially linked to the work of interpreting what fact-checking is, that is, in terms of objectives or in terms of process. Therefore, it would involve more than an accurate understanding or transfer of knowledge from fact-checkers to designers because interdisciplinary interactions transcend differences in meaning between disciplines (Akkerman & Bakker, 2011; Fox, 2011; Trompette & Vinck, 2009). This systematic literature review maps the field of AFC and its progress over the past five years to follow the recent developments in artificial intelligence (AI) in this field. It identifies how AFC tools, as boundary objects, connect or disconnect adjacent communities on either side of the technological artifact. In other words, how do they adapt to journalism and infuse professional practices? The objective is to provide a comprehensive multidisciplinary state of the art that considers a holistic and sociotechnical approach to studying AFC from a journalistic perspective to nourish future works. Therefore, two research questions are investigated: RQ1: To what extent does research on AFC tools consider end users, and how? RQ2: As boundary objects, how do AFC tools connect and disconnect with the social world of journalism? Method A systematic literature review is a means of collecting and synthesizing previous research, providing an overview of areas covered by the research, and demonstrating evidence on a meta-level (Snyder, 2019). Its main objective is to answer specific questions by relying on rigorous and explicit methods to ensure the work’s transparency, transferability, and replicability (Thomas & Harden, 2008). Although systematic literature reviews mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 3 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC are commonly used in the medical and computer sciences, they can also be utilized in social sciences to provide an overall picture of the evidence in a topic area to guide future research (Petticrew & Robert, 2006, p. 21). Meta- analysis refers to statistical techniques to obtain overall estimations or a synthesis of published research works (Çoğaltay & Karadağ, 2015; Shelby & Vaske, 2008). In this study, the meta-analysis was mobilized through descriptive statistics to highlight the corpus characteristics and provide quantified evidence to complement the systematic literature review (Davis, Mengersen, Bennett, & Mazerolle, 2014; Mengist, Soromessa, & Legese, 2020). From a methodological perspective, this systematic literature review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to ensure transparent and complete reporting (Liberati et al., 2009). It consists of a checklist of 27 items that frame the method, the writing of a systematic review report, and a flow diagram that shapes information retrieval and selection (Page et al., 2021). The tool Parsifal, available online (Durier da Silva, Bicharra Garcia, & Matsui Siqueira, 2022), was used to define the research planning (objectives, research questions, keywords, search strings, sources, and selection criteria) as well as to identify the articles published in Scopus. The tool was also used to import all the collected data as spreadsheets containing the articles’ metadata to identify duplicate entries. The data collection was carried out via Google Scholar, Semantic Scholar, and Scopus between March 14 and 31, 2022. The search terms were approached through three complementary queries to refine the quality of the corpus and limit the phenomena of noise (linked to unexpected results) and silence (referring to the absence of expected results). This iterative process followed an inverted pyramid model, that is, going from the general to the specific, considering that Google Scholar was the most general database used and Scopus was the most specific: • Fact-checking AND “machine learning” (Google Scholar) • Automated AND fact-checking AND journalism (Semantic Scholar) • (“machine learning” OR automated) AND fact-checking (Scopus) These three queries returned 918 results. Inclusion and exclusion criteria were applied to refine the corpus selection. Published peer-reviewed articles; open-access articles published on arXiv, which are published after moderation but are not peer-reviewed; book chapters; and proceedings related to AFC were included because the first objective of this review was to get a broad overview of the research works. Duplicate entries were removed along with all the articles that fell into the scope of the exclusion criteria: non-English texts, articles dated before 2017, undated articles, articles unrelated to AFC, and articles not available either in a PDF format or online. Dissertations were excluded from the review due to their extensive length, which is not comparable to research articles, and their limited accessibility and availability. Moreover, because their authors could publish their work in peer-reviewed journals, this exclusion criterion was also intended to limit the number of duplicate research. After the processes, the main corpus consisted of 267 articles used for a state of the art (Figure 1). mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 4 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Figure 1. Corpus selection process. To fit the scope of this study, the selection criteria were addressed using two complementary methods: (1) close reading of all of the collected abstracts and, in cases of uncertainty, additional close reading of the full texts, which concerned half of the collected corpus, and (2) distant reading of the abstracts through text mining and text analysis techniques using the programming language R (Ramage, Rosen, Chuang, Manning, & Mcfarland, 2009; Welbers, Van Atteveldt, & Benoit, 2017) and dedicated packages to proceed a meta-analysis and n-grams frequencies (tidyverse, tidytext, TM, quanteda, highcharter), topic modeling (LDA) and clustering (textmineR). This process allowed us to define a first subcorpus of 72 articles that considered users or journalism, as well as a second subcorpus of 21 articles that referred to users and journalism (Figure 1). The R packages previously mentioned were also used for the meta-analysis of the main corpus. The meta- analysis combined a deductive and an inductive approach (Grimmer, Roberts, & Stewart, 2021; Molina & Garip, 2019) to support discoveries considering research questions that globally refer to the challenges of AFC in the social world of journalism. In addition, we have created a database including a unique identifier, the title of the article, the abstract, the field of research, the type of article, the year of publication, and the number of citations. We added two columns to identify articles related to users and journalism or newsrooms. The datasets and the source code of all these operations are available on GitHub (https://github.com/laurence001/AFC_SLR). The limitations of the research strategy are related to the level of accuracy provided by distant reading. Indeed, it is recognized that topic modeling is not suitable for advanced data relationships and performs poorly when documents do not have a sufficient length (Vayansky & Kumar, 2020). Clustering is also challenging for finding similarities between data points and grouping similar ones into the same cluster (Qaddoura, Faris, & Aljarah, 2020). Consequently, these results were primarily used to support human analysis. In addition, it is challenging to claim the completeness of a corpus queried through databases during a short period. Nevertheless, the examination of the references in the articles collected indicates the representativeness of the corpus on recent research works on AFC. mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 5 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Results The presentation of the results is divided into three parts: (1) a meta-analysis of the corpus to identify the state of the art of AFC technologies as well as the general challenges of AFC in journalism, (2) a review of the first subcorpus related to the uses and users of AFC tools to answer RQ1, and (3) an analysis of the second subcorpus identifying the challenges of developing AFC tools in a journalistic context to address RQ2. This three-level analysis aims to provide evidence to fuel the discussion about how research should consider end users more when designing AFC tools. Meta-Analysis and Challenges for Journalism Most of the articles collected were published in 2019, 2020, and 2021 (respectively 22.47%, 22.10%, and 29.96%). However, it does not mean that AFC gained particular interest over this period, considering the time required for the research and the time needed for reviewing and publishing, although preprint articles represent 20.52% of the main corpus. This corpus also includes 51.49% of articles, 46.64% of proceedings, and 1.87% of book chapters. Computer science was the main research area covered (80.90%), followed by social computing (9.74%), and the articles related to social science or journalism studies represented less than 7% of the corpus. It is not surprising that computer science is the most represented field given that the development of AFC tools involves using specific technologies such as machine learning, natural language processing, and knowledge graphs (Gallofré Ocaña & Opdahl, 2020; Lakshmanan, Simpson, & Thirumuruganathan, 2019). In addition, social computing and information science are closely related to computer science, and journalism studies and social science intersect. AFC covers four main functionalities that assist and support human fact-checkers: finding claims, detecting already fact-checked claims, evidence retrieval, and verification (Nakov et al., 2021). Based on this typology, automated analysis of the abstracts of the main corpus demonstrated that the current state of the art in AFC technologies primarily covers claim detection (34.33%), followed by claim verification (14.18%) and evidence retrieval (8.20%). The technologies developed in AFC are most frequently oriented to machine learning (supervised and unsupervised), followed by natural language processing and knowledge graphs. Blockchain technologies are marginally represented (0.75%). AFC technologies are essentially text focused. Only 14 articles were dedicated to images and/or videos (5.24%). They included datasets of images and/or videos (e.g., Papadopoulou, Zampoglou, Papadopoulos, & Kompatsiaris, 2019; Zlatkova, Nakov, & Koychev, 2019), supervised machine learning to detect deceptive images or for image classification (Boididou et al., 2018; Reis & Benevenuto, 2021), the assessment of image forensics services (Katsaounidou, Gardikiotis, Tsipas, & Dimoulas, 2020; Nakov et al., 2021), social-computing solutions for user-generated content verification (Middleton, Papadopoulos, & Kompatsiaris, 2018), and deepfake detection (Hoque, Ferdous, Khan, & Tarkoma, 2021). Four articles focused on multimodal solutions to detect misleading texts and images (Abdelnabi, Hasan, & Fritz, 2021; Dhankar, Zaïane, & Bolduc, 2022; Gao, Hoffmann, Oikonomou, Kiskovski, & Bandhakavi, 2022; Yang et al., 2018). One article was dedicated to verifying audio content associated with video (Vryzas, Katsaounidou, Vrysis, Kotsakis, & Dimoulas, 2022). The full-text search results indicated that 129 articles were about experimental-stage systems (35%), and 179 articles mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 6 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC were about frameworks (48.5%). In addition, the authors of 45 articles provide online the source code of their fact-checking system (60%) and/or the datasets they built (62.2%) through a GitHub project page. Based on the abstracts’ content, datasets are the most prominent topic because it was found in 40% of the corpus. This is not surprising because AFC tools rely on data for classification and prediction purposes: it is how machine-learning works. Several articles focused more specifically on creating training datasets that are specific to one particular news context (e.g., on the Syrian war, U.S. politics, or COVID19). They are mainly published in English, except for one multilingual article and three others in Spanish, Arabic, and Czech. They rely on several different classification systems such as “True,” “False,” or “Halftrue” (Wang, 2017); “Unproven” (Kotonya & Toni, 2020); and “Contradiction,” “Compatible,” or “Unrelated” (Sepúlveda-Torres, Bonet-Jover, & Saquete, 2021). They also encompass variable amounts of data; for instance, the FEVER dataset—which consists of 185,445 claims generated by altering sentences extracted from Wikipedia—comprises 185,000 rows (Thorne, Vlachos, Christodoulopoulos, & Mittal, 2018), whereas AraStance has more than 4,000 rows. It consists of a multicountry and multidomain dataset of Arabic stance detection for fact-checking, based on claim-article pairs from a diverse set of sources comprising three factchecking websites and one news website (Alhindi, Alabdulkarim, Alshehri, Abdul-Mageed, & Nakov, 2021). The reliability of dataset labeling (or annotation) required for supervised tasks is challenged by crowdsourcing as mentioned in the abstracts of 11 articles. At the same time, Wikipedia was quoted as a source for data extraction in five out of 12 articles, and it was mentioned in 110 articles (41.2% of the corpus). Training journalistic tools with Wikipedia poses several issues because the content is generated by unknown users (Umarova & Mustafaraj, 2019), and encyclopedic texts differ from journalistic ones. From the broader perspective of machine-learning research, it has been pointed out that the crowd is not always made up of experts and that human biases can interfere with tasks’ accuracy and outcomes’ reliability (Lease, 2011; Miceli, Posada, & Yang, 2021). The performance of the AFC systems relying on machine learning can be approached through the F1 score that is commonly used to evaluate the precision and recall of a classification model. Hence, it is tackled as an error rate indicator. We found 92 articles (35.20% of the corpus) referring to this score, which ranged between 5% (Hui Xian Ng & Carley, 2021) and 99.6% (Ebadi, Choo, & Rad, 2022). Although the F1 score is not the only indicator to measure the performance of a model, it remains generic and can be considered a weak estimator for the uncertainty that can remain in the outcomes (Kläs & Vollmer, 2018). The performance of a model can also be tackled through the detection of overfitting, which occurs when the model is trained for too long and, as a result, comes to reflect the specifics of the training data rather than the general characteristics of the underlying domain. Consequently, the model fails to find a general predictive rule (Dietterich, 1995). In this corpus, the overfitting of the machine-learning model was pointed out in 24 articles. This does not mean that the research has failed, but it illustrates the challenges in developing efficient and scalable AFC tools. As boundary objects, AFC systems represent a clear division between their designers and factcheckers. This finding is confirmed in the second part of the analysis, which focuses on the relationship between AFC systems and the social world of journalism. However, some attempts to connect technologies to work mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 7 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC environments and practices have also been observed, particularly when considering keeping the human fact- checker in the loop. Uses and End Users of AFC Systems The proportion of articles considering end users and/or journalism is relatively low, representing 20.22% and 14.61% of the corpus, respectively. This highlights that attempts by developers in AFC are driven by the goal to provide a technical solution to the social problem of information disorder. According to this approach, the growing importance of information disorder creates a need for new tools to speed up the process of detecting and verifying claims (e.g., Azevedo, 2018; Çarik & Yeniterzi, 2021; Du, Bosselut, & Manning, 2022; Kar, 2020). However, we found 72 articles directly or indirectly related to the sociotechnical dimension of AFC technologies, that is, 26% of the main corpus, comprising the first subcorpus devoted to the uses and end users. Several articles highlighted the time-consuming aspects of human fact-checking activities and the challenge that human fact-checkers cannot keep up with the amount of misinformation and the speed at which it spreads (e.g., Adair, Li, Yang, & Yu, 2017; Gencheva, Koychev, Màrquez, Barrón-Cedeño, & Nakov, 2019; Hanselowski & Gurevych, 2017; Jiang, Baumgartner, Ittycheriah, & Yu, 2020). Scholars also emphasized that human fact-checking requires expertise (Jaroucheh, Alissa, Buchanan, & Liu, 2020) and, in a certain way, human intuition and creativity that can’t be automated (Nakov et al., 2021). Furthermore, the relationship between AFC tools and human users remains problematic (Borges, Martins, & Calado, 2019) due to limitations related to credibility issues for automated systems and scalability issues for human factchecking (Nakov et al., 2021). When researchers consider uses or end users, the focus is mostly on technological solutions to speed up the AFC process or assist human fact-checkers (e.g., Brand, Roitero, Soprano, & Demartini, 2021; Majithia et al., 2019). However, the fact-checkers toolbox is also explored (Bañon Castellón, 2021; Nygren, Guath, Axelsson, & Frau-Meigs, 2021; Svahn & Perfumi, 2021), compared (Školkay & Filin, 2019), classified (Nakov et al., 2021), and evaluated (Komendantova et al., 2021; Picha Edwardsson, Al-Saqaf, & Nygren, 2021). Another perspective is related to the benefits of human-computer interactions (Miranda et al., 2019; Shi, Bhattacharya, Das, Lease, & Gwizdka, 2022; Yang et al., 2019). These approaches can be understood in terms of challenges and opportunities that concern both users and developers (e.g., Demartini, Mizzaro, & Spina, 2020; Gallofré Ocaña & Opdahl, 2020). They encourage a hybrid process where humans are augmented by the use of AI, broadening a technical solution into a sociotechnical one, which is likely to result in the reconfiguration of professional practices (Diakopoulos, Trielli, & Lee, 2021). This concerns not only the AFC system’s functionalities but also how the tool was designed, considering the users’ requirements (Nguyen et al., 2018) or the transparency of the model in facilitating human interactions (Rony, Hoque, & Hassan, 2020). Making an AFC system transparent depends on its explicability. Transparency concerns both the upstream components of the tool (Reis, Correia, Murai, Veloso, & Benevenuto, 2019)—for example, the explanation of the model and the information sources used (Gencheva et al., 2019) or why the expert assigned that label to a training dataset (Berendt et al., 2021)—its interface (Katsaounidou et al., 2020); and the results it provides (Denaux & Gomez-Perez, 2020; Middleton et al., 2018). Therefore, the use of AI tools is first and foremost a matter of trust (Demartini et al., 2020). Providing explanations not only helps humans perform fact-checking but mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 8 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC also can foster trust toward the tool or improve AFC systems (Gad-Elrab, Stepanova, Urbani, & Weikum, 2019). However, this can be difficult when the system relies on black-box methods, such as deep neural networks (Saeed & Papotti, 2021). Contextual information about the application domain is another topic that has characterized this first subcorpus. It implies some shared expertise between coders and end users (Berendt et al., 2021). Providing context with the results contributes to a better understanding of a claim (Vizoso, Vaz-Álvarez, & López-García, 2021; Vryzas et al., 2022) and suggests trustworthiness in the tool and its results (Middleton et al., 2018). In addition, the lack of knowledge of the application domain may interfere with the correctness and the quality of a training dataset’s annotations (Singh, Das, Li, & Lease, 2021). It was also pinpointed that human fact-checking also requires human judgment and sensitivity (Graves, 2018). Therefore, research in AFC also considers contextbased approaches (Boididou et al., 2018; Fairbanks, Fitch, Bradfield, & Briscoe, 2020; Gencheva et al., 2019), which can be interpreted as another form of an attempt to link boundary objects with the knowledge and practices of a given professional community (Trompette & Vinck, 2009). AFC and Journalism The analysis of the second subcorpus of 21 articles (7.86% of the main corpus) demonstrated that the most prominent perspective is about developing tools (42.85%) or supporting journalistic practices (52.38%), mainly for claims or stance detection (33.33%). Nearly half of the articles (42.86%) presented prototypes (e.g., Adair et al., 2017; Horne, Dron, Khedr, & Adali, 2018), of which five consisted of experiments (Adler & Boscaini-Gilroy, 2019; Boididou et al., 2018; Jiang et al., 2020; Masood & Aker, 2018; Vryzas et al., 2022). In addition, one article focused on fact-checkers’ needs by exploring the tools they use and how AFC tools can support them (Nakov et al., 2021), and four articles—all related to journalism studies—were dedicated to journalism practices within newsrooms (Bañon Castellón, 2021; de Haan, Van Den Berg, Goutier, Kruikemeier, & Lecheler, 2022; Diakopoulos, 2020; Picha Edwardsson et al., 2021). The world of journalism is less considered in research on AFC. However, when it is considered, the complementarity between the journalist and the tool is the most frequently underlined aspect. That is mainly because fact-checking is a time-consuming process that still requires human input either to assess the validity of a claim or to adapt to dynamic news contexts by extending the daily collection of the source, for instance, as news information collection is a continuous process that occurs in a moving reality (Berendt et al., 2021; Hassan et al., 2017). Several articles also stressed that human-machine complementarity is still needed because AFC systems are not strong enough to give rise to fully automated solutions (Komendantova et al., 2021). The question of the fact-checkers’ user needs was tackled either through a sociotechnical prism (Diakopoulos et al., 2021) or by considering the cultural background and attitudes of journalists who may be skeptical toward technology and who generally lack algorithmic culture (de Haan et al., 2022). Still, from a journalistic point of view, other challenges are related to the “lack of collaboration between researchers and practitioners in terms of defining tasks and developing datasets for automated systems” (Nakov et al., 2021, p. 5), the lack of time to learn how to use digital tools (Picha Edwardsson et al., 2021), the lack of consideration of the specific needs that can be expressed by fact-checkers, and the lack of transparency of some fact-checking mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 9 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC services (Komendantova et al., 2021). Although addressing some of these issues might be seen as a fruitful path to better intersect AFC systems’ users and designers, all cannot be technically solved, mainly because they also rely on social and organizational variables that involve human decisions. Designing an AFC system is also challenging because of the complexity of defining central concepts such as “claim” and “verification” (e.g., Konstantinovskiy et al., 2021). Other issues can also be related to the limited availability of training datasets or to the overall quality of some existing ones, including the aspects related to the quality of their annotations (e.g., de Haan et al., 2022). Systems must also be adapted to the complexity of the newsroom’s workflows, where the multiplication of information channels is required to speed up the processes and provide newsworthy insights (Gallofré Ocaña & Opdahl, 2020). Discussion and Conclusion Regarding RQ1, related to the integration of end-user perspectives, the findings indicated that most research focuses on providing technological solutions to a social problem without embedding end users’ views or needs. As machine-learning systems rely on training data, quality issues appear crucial, specifically when annotations are based on crowdsourcing by nonexperts. Annotations are inherently prone to errors even when control procedures are set for correction (Northcutt, Athalye, & Mueller, 2021). They are also language, domain, and context dependent, making them less adaptive or reusable. In addition, the datasets used for machine learning have various sizes, rely on binary classes for the labeling, and tend to be unbalanced, which is not ideal for training (Zeng, Abumansour, & Zubiaga, 2021). Because they are likely to rely on various annotation process types, the question is also that of developing a standardized annotation system (Zeng et al., 2021). Such diversity also complicates both comparing approaches and gauging their improvement over time. At the same time, there is no consensus on the best classification strategies and sets of features for AFC tools developed for detection (Silva, Santos, Almeida, & Pardo, 2020). Nonetheless, researchers have underlined that the datasets developed for natural language processing purposes are more valuable, cover different domains, and help progress research in automated claim validation (Zeng et al., 2021). Furthermore, automating detection and selection processes is also challenging because the criteria are likely to vary from one fact-checking organization to another, although common patterns are observed, such as prominence within the opinion or debate, virality, and measures of social engagement (Micallef, Armacost, Memon, & Patil, 2022). Working with data from datasets based on Wikipedia raises the same risk of potential quality issues as any other user-generated content in terms of accuracy or reliability because there are no guarantees about the users’ expertise. In this regard, quality control of the datasets and their maintenance over time also appeared as two other obstacles to efficiency, and data quality has “a huge impact on the efficiency, accuracy and complexity of machine learning tasks” (Gupta et al., 2021, p. 1). In addition, the use of crowd workers raises “practical and ethical issues, such as funding and remuneration” (Berendt et al., 2021, p. 11). Although tremendous efforts are dedicated to gathering and annotating datasets for machine-learning tasks, the lack of relevant ones could be a constraint on developing AFC tools (Pathak & Srihari, 2019). mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 10 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC At the same time, the lack of vision about the fact-checkers’ needs and requirements appears problematic as several fact-checking tools are developed without considering them (Komendantova et al., 2021). The explicability of the system contributes to building a relationship of confidence between the user and the tool, and it plays a central role in using AI tools (Stray, 2019). In a context where fact-checkers vigorously promote values of transparency, being able to explain algorithms at work and the results they provide Katsaounidou et al., 2020) can be considered a lever for trust, or an instrument to demystify tools that might be considered black boxes (e.g., Bartneck, Lutge, Wagner, & Welsh, 2021; de Haan et al., 2022; Zhou, Hu, Li, Yu, & Chen, 2019). From the BO theory perspective addressed in RQ2, AFC tools might exist between the adjacent communities of researchers and fact-checkers by involving the latter in the process when it comes to assessing the performance and the usability of the tool, for instance because this is likely to help improve it (Miranda et al., 2019). This implies a transfer of knowledge from one professional community to another, although the computational process results from a work of interpretation (Huvila et al., 2017). The social aspects of fact- checking also play a role in the adoption of the technological artifact, including in terms of its perceived advantages (Fox, 2011). Specific training in identifying manipulated images and deep fakes, along with the ease of use and integration of AFC tools into editorial systems, are additional outcomes from journalism studies to enhance the integration of AFC tools in newsrooms (Katsaounidou et al., 2020; Picha Edwardsson et al., 2021; Vizoso et al., 2021). AFC tools are helpful to detect falsehoods, but they do not eliminate the need for human intervention (Bañon Castellón, 2021). Moreover, AFC tools should meet the users’ trust (Nakov et al., 2021), which can be achieved only if both the system and its outcomes are trustworthy too. When professional fact-checkers use their human expertise and take charge of the labeling of training datasets for machine-learning-based systems (Berendt et al., 2021), this “human in the loop” perspective is a means to achieve connection between the communities surrounding the BO. Integrating the users’ knowledge also improves a system’s transparency and enhances its trustworthiness (Nguyen et al., 2018) because it also emphasizes on the social components of technology. Ultimately, “automated fact-checking works well in some cases,” but it “still needs improvement prior to widespread use” (Lazarski, Al-Khassaweneh, & Scotts Howard, 2021, p. 1). Reconnecting adjacent professional communities from either part of the BO may also be achieved by paying acute attention to the users’ beliefs and the possibility “to infuse their views and knowledge into the system” (Shi et al., 2022, p. 315). Furthermore, the necessary compatibility of a technological artifact with journalistic ideals and values was also stressed in the context of the diffusion of news automation within newsrooms (Diakopoulos, 2019). However, in journalism, psychological and cultural barriers challenge the effectiveness of AFC tools, although technology can make fact-checking easier and faster (Cazalens, Lamarre, Leblay, Manolescu, & Tannier, 2018). We can draw a parallel with previous qualitative research on the possibilities of AI technologies within newsrooms where human journalists intend to keep the lead in these new forms of human-machine collaborations (Dierickx, 2020; Gutierrez Lopez et al., 2022). Effective AFC tools require efficient technology. Nevertheless, no technology can be used sufficiently on its own, despite various technical challenges. Therefore, (re)connecting communities means responding to both mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 11 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC technological and social issues that arise upstream and downstream of the automated process. Our observations provide useful reflections for further research in AFC, whether working on improving data quality, models for standardizing annotations, the explainability of processes and results, or improving human-computer interaction and the overall user experience to better support professional practices. Drawing links between the adjacent communities surrounding the BO also implies that is connects research fields because AFC involves different but complementary scientific disciplines. References Abdelnabi, S., Hasan, R., & Fritz, M. (2021). Open-domain, content-based, multi-modal fact-checking of out-of- context images via online resources. Retrieved from http://arxiv.org/abs/2112.00061 Adair, B., Li, C., Yang, J., & Yu, C. (2017, October 13–14). Progress toward “the Holy Grail”: The continued quest to automate fact-checking. Paper presented at the Computation + Journalism Symposium, Evanston, IL. Retrieved from https://ranger.uta.edu/~cli/pubs/2017/factcheckingcj17-adair.pdf Adler, B., & Boscaini-Gilroy, G. (2019). Real-time claim detection from news articles and retrieval of semantically-similar factchecks. Retrieved from http://arxiv.org/abs/1907.02030 Akkerman, S. F., & Bakker, A. (2011). Boundary crossing and boundary objects. Review of Educational Research, 81(2), 132–169. doi:10.3102/0034654311404435 Alhindi, T., Alabdulkarim, A., Alshehri, A., Abdul-Mageed, M., & Nakov, P. (2021). AraStance: A multicountry and multi-domain dataset of Arabic stance detection for fact-checking. In Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda (pp. 171–177). Stroudsburg, PA: Association for Computational Linguistics. doi:10.18653/v1/2021.nlp4if-1.9 Azevedo, L. (2018). Truth or lie: Automatically fact checking news. In Companion Proceedings of the Web Conference 2018 (pp. 807–811). Geneva, Switzerland: International World Wide Web Conferences Steering Committee. doi:10.1145/3184558.3186567 Bañon Castellón, L. (2021). Audiovisual verification in the evolution of television newsrooms: Al Jazeera and the transition from satellite to the cloud. Anàlisi, 64(2021), 85–102. doi:10.5565/rev/analisi.3414 Bardoel, J., & d’Haenens, L. (2004). Media responsibility and accountability. New conceptualizations and practices. Communications, 29(1), 5–25. doi:10.1515/comm.2004.007 Bartneck, C., Lutge, C., Wagner, A., & Welsh, S. (2021). An introduction to ethics in robotics and AI an introduction to ethics in robotics and AI. Cham, Switzerland: Springer Nature. mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 12 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Berendt, B., Burger, P., Hautekiet, R., Jagers, J., Pleijter, A., & Van Aelst, P. (2021). FactRank: Developing automated claim detection for Dutch-language fact-checkers. Online Social Networks and Media, 22(100113), 1–12. doi:10.1016/j.osnem.2020.100113 Boididou, C., Papadopoulos, S., Zampoglou, M., Apostolidis, L., Papadopoulou, O., & Kompatsiaris, Y. (2018). Detection and visualization of misleading content on Twitter. International Journal of Multimedia Information Retrieval, 7(1), 71–86. doi:10.1007/s13735-017-0143-x Borges, L., Martins, B., & Calado, P. (2019). Combining similarity features and deep representation learning for stance detection in the context of checking fake news. ACM Journal of Data and Information Quality, 11(3), 1–26. doi:10.1145/3287763 Brand, E., Roitero, K., Soprano, M., & Demartini, G. (2021, October). E-BART: Jointly predicting and explaining truthfulness. In I. Augenstein, P. Papotti, & D. Wright (Eds.), Proceedings of the 2021 Truth and Trust Online Conference (pp. 18–27). Arlington, VA: Hacks Hackers. Retrieved from https://truthandtrustonline.com/wp-content/uploads/2021/10/TTO2021_paper_16-1.pdf Çarik, B., & Yeniterzi, R. (2021). SU-NLP at CheckThat! 2021: Check-worthiness of Turkish tweets. In Proceedings of CLEF 2021—Conference and Labs of the Evaluation Forum (pp. 476–483). Bucharest, Romania: CEUR Workshop Proceedings. Retrieved from http://ceur-ws.org/Vol2936/paper-37.pdf Cavaliere, P. (2020). From journalistic ethics to fact-checking practices: Defining the standards of content governance in the fight against disinformation. Journal of Media Law, 12(2), 133–165. doi:10.1080/17577632.2020.1869486 Cazalens, S., Lamarre, P., Leblay, J., Manolescu, I., & Tannier, X. (2018). A content management perspective on fact-checking. In Proceedings of the Web Conference 2018 (pp. 565–574). New York, NY: ACM Press. Çoğaltay, N., & Karadağ, E. (2015). Introduction to meta-analysis. In E. Karadağ (Ed.), Leadership and organizational outcomes: Meta-analysis of empirical studies (pp. 19–28). Cham, Switzerland: Springer International Publishing. Davis, J., Mengersen, K., Bennett, S., & Mazerolle, L. (2014). Viewing systematic reviews and metaanalysis in social research through different lenses. SpringerPlus, 3(1), 1–9. doi:10.1186/21931801-3-511 de Haan, Y., Van Den Berg, E., Goutier, N., Kruikemeier, S., & Lecheler, S. (2022). Invisible friend or foe? How mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 13 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC journalists use and perceive algorithmic-driven tools in their research process. Digital Journalism, 10(10), 1775–1793. doi:10.1080/21670811.2022.2027798 Demartini, G., Mizzaro, S., & Spina, D. (2020). Human-in-the-loop artificial intelligence for fighting online misinformation: Challenges and opportunities. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 43(3), 65–74. Retrieved from http://sites.computer.org/debull/A20sept/p65.pdf Denaux, R., & Gomez-Perez, J. M. (2020). Linked credibility reviews for explainable misinformation detection. In J. Z. Pan, V. Tamma, C. D’Amato, K. Janowicz, B. Fu, A. Polleres, . . . L. Kagal (Eds.), Lecture notes in computer science (pp. 147–163). Cham, Switzerland: Springer International Publishing. Dhankar, A., Zaïane, O. R., & Bolduc, F. (2022). UofA-Truth at Factify 2022: Transformer and transfer learning based multi-modal fact-checking. ArXiv. doi:10.48550/arXiv.2203.07990 Diakopoulos, N. (2019). Automating the news: How algorithms are rewriting the media. Cambridge, MA: Harvard University Press. Diakopoulos, N. (2020). Computational news discovery: Towards design considerations for editorial orientation algorithms in journalism. Digital Journalism, 8(7), 945–967. doi:10.1080/21670811.2020.1736946 Diakopoulos, N., Trielli, D., & Lee, G. (2021). Towards understanding and supporting journalistic practices using semi-automated news discovery tools. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2), 1–30. doi:10.1145/3479550 Dierickx, L. (2020). The social construction of news automation and the user experience. Brazilian Journalism Research, 16(3), 432–457. doi:10.25200/BJR.v16n3.2021.1289 Dietterich, T. (1995). Overfitting and undercomputing in machine learning. ACM Computing Surveys, 27(3), 326– 327. doi:10.1145/212094.212114 Du, Y., Bosselut, A., & Manning, C. D. (2022). Synthetic disinformation attacks on automated fact verification systems. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 10581–10589. doi:10.1609/aaai.v36i10.21302 Durier da Silva, F. C., Bicharra Garcia, A. C., & Matsui Siqueira, S. W. (2022). A systematic literature mapping on profile trustworthiness in fake news spread. In 2022 IEEE 25th International Conference on Computer mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 14 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Supported Cooperative Work in Design (CSCWD) (pp. 275–279). Hangzhou, China: IEEE. doi:10.1109/CSCWD54268.2022.9776232 Ebadi, N., Jozani, M., Choo, K.-K. R., & Rad, P. (2022). A memory network information retrieval model for identification of news misinformation. IEEE Transactions on Big Data, 8(5), 1358–1370. doi:10.1109/TBDATA.2020.3048961 Fairbanks, J. P., Fitch, N., Bradfield, F., & Briscoe, E. (2020). Credibility development with knowledge graphs. In C. Grimme, M. Preuss, F. W. Takes, & A. Waldherr (Eds.), Disinformation in open online media (pp. 33–47). Cham, Switzerland: Springer International Publishing. doi:10.1007/978-3-030-39627-5_4 Frost, C. (2015). Journalism ethics and regulation. London, UK: Routledge. doi:10.4324/9781315757810 Fox, N. J. (2011). Boundary objects, social meanings and the success of new technologies. Sociology, 45(1), 70– 85. doi:10.1177/0038038510387196 Gad-Elrab, M. H., Stepanova, D., Urbani, J., & Weikum, G. (2019). ExFaKT: A framework for explaining facts over knowledge graphs and text. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining (pp. 87–95). New York, NY: Association for Computing Machinery. doi:10.1145/3289600.3290996 Gallofré Ocaña, M., & Opdahl, A. L. (2020). Challenges and opportunities for journalistic knowledge platforms. Conference on Information and Knowledge Management (CIKM 2020 Workshops). Retrieved from https://ceur-ws.org/Vol-2699/paper43.pdf Gao, J., Hoffmann, H.-F., Oikonomou, S., Kiskovski, D., & Bandhakavi, A. (2022). Logically at Factify 2022: Multimodal fact verification. Retrieved from http://arxiv.org/abs/2112.09253 Gencheva, P., Koychev, I., Màrquez, L., Barrón-Cedeño, A., & Nakov, P. (2019). A context-aware approach for detecting check-worthy claims in political debates. Retrieved from http://arxiv.org/abs/1912.08084 Graves, L. (2018). Boundaries not drawn: Mapping the institutional roots of the global fact-checking movement. Journalism Studies, 19(5), 613–631. doi:10.1080/1461670x.2016.1196602 Graves, L., & Cherubini, F. (2016). The rise of fact-checking sites in Europe. Oxford, UK: Reuters Institute for the Study of Journalism. mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 15 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Grimmer, J., Roberts, M. E., & Stewart, B. M. (2021). Machine learning for social science: An agnostic approach. Annual Review of Political Science, 24, 395–419. doi:10.1146/annurev-polisci-053119015921 Guess, A. M., Lockett, D., Lyons, B., Montgomery, J. M., Nyhan, B., & Reifler, J. (2020). “Fake news” may have limited effects beyond increasing beliefs in false claims. Harvard Kennedy School Misinformation Review, 1(1). doi:10.37016/mr-2020-004 Guo, Z., Schlichtkrull, M., Thorne, J., Vlachos, A., Christodoulopoulos, C., Cocarascu, O., & Mittal, A. (2022). A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10(2022), 178–206. doi:10.1162/tacl_a_00454 Gupta, N., Patel, H., Afzal, S., Panwar, N., Mittal, R. S., Guttula, S., . . . Saha, D. (2021). Data Quality Toolkit: Automatic assessment of data quality and remediation for machine learning datasets. Retrieved from http://arxiv.org/abs/2108.05935 Gutierrez Lopez, M., Porlezza, C., Cooper, G., Makri, S., MacFarlane, A., & Missaoui, S. (2022). A question of design: Strategies for embedding AI-driven tools into journalistic work routines. Digital Journalism, 11(3), 484–503. doi:10.1080/21670811.2022.2043759 Hanselowski, A., & Gurevych, I. (2017, November). A framework for automated fact-checking for realtime validation of emerging claims on the web. In NIPS 2017 Workshop on Prioritising Online Content (pp. 1– 3). Long Beach, CA: NIPS. Retrieved from https://www.k4all.org/wpcontent/uploads/2017/09/WPOC2017_paper_6.pdf Hassan, N., Zhang, G., Arslan, F., Caraballo, J., Jimenez, D., Gawsane, S., … Tremayne, M. (2017). ClaimBuster: The first-ever end-to-end fact-checking system. Proceedings of the VLDB Endowment International Conference on Very Large Data Bases, 10(12), 1945–1948. doi:10.14778/3137765.3137815 Hoque, M. A., Ferdous, M. S., Khan, M., & Tarkoma, S. (2021). Real, forged or deep fake? Enabling the ground truth on the internet. IEEE Access: Practical Innovations, Open Solutions, 9(2021), 160471–160484. doi:10.1109/access.2021.31315170 Horne, B. D., Dron, W., Khedr, S., & Adali, S. (2018). Assessing the news landscape: A multi-module toolkit for evaluating the credibility of news. In Companion Proceedings of the Web Conference 2018 (pp. 235–238). Lyon, France: International World Wide Web Conferences Steering Committee. doi:10.1145/3184558.3186987 mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 16 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Hui Xian Ng, L., & Carley, K. M. (2021). “The coronavirus is a bioweapon”: Classifying coronavirus stories on fact-checking sites. Computational and Mathematical Organization Theory, 27(2), 179–194. doi:10.1007/s10588-021-09329-w Huvila, I., Anderson, T. D., Jansen, E. H., Mckenzie, P. J., & Worrall, A. (2017). Boundary objects in information science. Journal of the Association for Information Science and Technology, 68(8), 1807–1822. doi:10.1002/asi.23817 Jaroucheh, Z., Alissa, M., Buchanan, W. J., & Liu, X. (2020). TRUSTD: Combat fake content using blockchain and collective signature technologies. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC) (pp. 1235–1240). Madrid, Spain: IEEE. doi:10.1109/COMPSAC48688.2020.00-87 Jiang, S., Baumgartner, S., Ittycheriah, A., & Yu, C. (2020). Factoring fact-checks: Structured information extraction from fact-checking articles. In Proceedings of the Web Conference 2020 (pp. 1592– 1603). New York, NY: ACM. doi:10.1145/3366423.3380231 Kalsnes, B. (2018). Fake news. In Oxford Research Encyclopedia of Communication. Oxford, UK: Oxford University Press. doi:10.1093/acrefore/9780190228613.013.809 Kar, D. (2020). Spotting misinformation to limit the impact of disruption on society by using machine learning. In 2020 IEEE Applied Signal Processing Conference (ASPCON) (pp. 26–30). Kolkata, India: APSCON. doi:10.1109/ASPCON49795.2020.9276723 Katsaounidou, A. N., Gardikiotis, A., Tsipas, N., & Dimoulas, C. A. (2020). News authentication and tampered images: Evaluating the photo-truth impact through image verification algorithms. Heliyon, 6(12), 1–22. doi:10.1016/j.heliyon.2020.e05808 Kläs, M., & Vollmer, A. M. (2018). Uncertainty in machine learning applications: A practice-driven classification of uncertainty. In Developments in language theory (pp. 431–438). Cham, Switzerland: Springer International Publishing. Komendantova, N., Ekenberg, L., Svahn, M., Larsson, A., Shah, S. I. H., Glinos, M., . . . Danielson, M. (2021). A value-driven approach to addressing misinformation in social media. Humanities and Social Sciences Communications, 8(1), 1–12. doi:10.1057/s41599-020-00702-9 mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 17 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Konstantinovskiy, L., Price, O., Babakar, M., & Zubiaga, A. (2021). Toward automated factchecking: Developing an annotation schema and benchmark for consistent automated claim detection. Digital Threats: Research and Practice, 2(2), 1–16. doi:10.1145/3412869 Kotonya, N., & Toni, F. (2020). Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 7740–7754). Stroudsburg, PA: Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.623 Lakshmanan, L. V. S., Simpson, M., & Thirumuruganathan, S. (2019). Combating fake news: A data management and mining perspective. Proceedings of the VLDB Endowment, 12(2), 1990–1993. doi:10.14778/3352063.3352117 Lazarski, E., Al-Khassaweneh, M., & Scotts Howard, C. (2021). Using NLP for fact checking: A survey. Designs, 5(3), 1–22. doi:10.3390/designs5030042 Lease, M. (2011). On quality control and machine learning in crowdsourcing. In Workshops at the TwentyFifth AAAI Conference on Artificial Intelligence (pp. 97–102). San Francisco, CA: AAAI. Lewis, S. C., & Usher, N. (2016). Trading zones, boundary objects, and the pursuit of news innovation: A case study of journalists and programmers. Convergence, 22(5), 543–560. doi:10.1177/1354856515623865 Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., . . . Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Annals of Internal Medicine, 151(4), 65–94. doi:10.7326/0003-4819-151-4-200908180-00136 Majithia, S., Arslan, F., Lubal, S., Jimenez, D., Arora, P., Caraballo, J., & Li, C. (2019). ClaimPortal: Integrated monitoring, searching, checking, and analytics of factual claims on twitter. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (pp. 153–158). Florence, Italy: Association for Computational Linguistics. doi:10.18653/v1/p19-3026 Masood, R., & Aker, A. (2018). The fake news challenge: Stance detection using traditional machine learning approaches. In Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (pp. 128–135), Seville, Spain: KMIS. doi:10.5220/0006898801280135 mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 18 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Mena, P. (2019). Principles and boundaries of fact-checking: Journalists’ perceptions. Journalism Practice, 13(6), 657–672. doi:10.1080/17512786.2018.1547655 Mengist, W., Soromessa, T., & Legese, G. (2020). Method for conducting systematic literature review and meta- analysis for environmental science research. MethodsX, 7(100777), 1–11. doi:10.1016/j.mex.2019.100777 Micallef, N., Armacost, V., Memon, N., & Patil, S. (2022). True or false: Studying the work practices of professional fact-checkers. Proceedings of the ACM on Human–Computer Interaction, 6(CSCW1), 1–44. doi:10.1145/3512974 Miceli, M., Posada, J., & Yang, T. (2021). Studying up machine learning data: Why talk about bias when we mean power? Retrieved from http://arxiv.org/abs/2109.08131 Middleton, S. E., Papadopoulos, S., & Kompatsiaris, Y. (2018). Social computing for verifying social media content in breaking news. IEEE Internet Computing, 22(2), 83–89. doi:10.1109/mic.2018.112102235 Miranda, S., Nogueira, D., Mendes, A., Vlachos, A., Secker, A., Garrett, R., . . . Marinho, Z. (2019). Automated fact checking in the newsroom. In The World Wide Web Conference (pp. 3579–3583). New York, NY: ACM. doi:10.1145/3308558.3314135 Molina, M., & Garip, F. (2019). Machine learning for sociology. Annual Review of Sociology, 45(1), 27–45. doi:10.1146/annurev-soc-073117-041106 Nakov, P., Corney, D. P. A., Hasanain, M., Alam, F., Elsayed, T., Barrón-Cedeño, A., . . . Da San Martino, G. (2021). Automated fact-checking for assisting human fact-checkers. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (pp. 4826–4832). Montreal, Canada: IJCAI. Retrieved from https://arxiv.org/abs/2103.07769 Nguyen, A. T., Kharosekar, A., Krishnan, S., Krishnan, S., Tate, E., Wallace, B. C., & Lease, M. (2018). Believe it or not: Designing a human-AI partnership for mixed-initiative fact-checking. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (pp. 189–199). New York, NY: Association for Computing Machinery. doi:10.1145/3242587.3242666 Northcutt, C. G., Athalye, A., & Mueller, J. (2021). Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv Preprint. doi:10.48550/arXiv.2103.14749 mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 19 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Nygren, T., Guath, M., Axelsson, C.-A. W., & Frau-Meigs, D. (2021). Combatting visual fake news with a professional fact-checking tool in education in France, Romania, Spain, and Sweden. Information, 12(5), 1–25. doi:10.3390/info12050201 Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., . . . Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. International Journal of Surgery, 88(105906), 1–9. doi:10.1016/j.ijsu.2021.105906 Papadopoulou, O., Zampoglou, M., Papadopoulos, S., & Kompatsiaris, I. (2019). A corpus of debunked and verified user-generated videos. Online Information Review, 43(1), 72–88. doi:10.1108/oir03-2018-0101 Pathak, A., & Srihari, R. (2019). BREAKING! Presenting fake news corpus for automated fact checking. In Proceedings of the 57th annual meeting of the association for computational linguistics: Student research workshop (pp. 357–362). Florence, Italy: Association for Computational Linguistics. doi:10.18653/v1/p19-2050 Petticrew, M., & Roberts, H. (2006). Systematic Reviews in the social sciences: A practical guide. Malden, MA: Wiley-Blackwell. Picha Edwardsson, M., Al-Saqaf, W., & Nygren, G. (2021). Verification of digital sources in Swedish newsrooms—A technical issue or a question of newsroom culture. Journalism Practice, 1–18. doi:10.1080/17512786.2021.2004200 Posetti, J., & Matthews, A. (2018). A short guide to the history of “fake news” and disinformation. Washington, DC: International Center for Journalists. Retrieved from https://www.icfj.org/sites/default/files/201807/A%20Short%20Guide%20to%20History%20of%20Fake %20News%20and%20Disinformation_IC FJ%20Final.pdf Qaddoura, R., Faris, H., & Aljarah, I. (2020). An efficient clustering algorithm based on the K-Nearest neighbors with an indexing ratio. International Journal of Machine Learning and Cybernetics, 11(3), 675–714. doi:10.1007/s13042-019-01027-z Ramage, D., Rosen, E., Chuang, J., Manning, C. D., & Mcfarland, D. A. (2009). Topic modeling for the social sciences. NIPS 2009 Workshop on Applications for Topic Models: Text and Beyond, 5(27), 1–4. Retrieved from https://nlp.stanford.edu/pubs/tmt-nips09.pdf mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 20 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Reis, J. C. S., & Benevenuto, F. (2021). Supervised learning for misinformation detection in WhatsApp. In Proceedings of the Brazilian Symposium on Multimedia and the Web (pp. 245–252). New York, NY: Association for Computing Machinery. doi:10.1145/3470482.3479641 Reis, J. C. S., Correia, A., Murai, F., Veloso, A., & Benevenuto, F. (2019). Supervised learning for fake news detection. IEEE Intelligent Systems, 34(2), 76–81. doi:10.1109/mis.2019.2899143 Rony, M. M. U., Hoque, E., & Hassan, N. (2020). ClaimViz: Visual analytics for identifying and verifying factual claims. In 2020 IEEE Visualization Conference (VIS) (pp. 246–250). New York, NY: IEEE. doi:10.1109/VIS47514.2020.00056 Saeed, M., & Papotti, P. (2021). Fact-checking statistical claims with Tables. IEEE Data Engineering Bulletin, 44(3), 27–38. Retrieved from http://sites.computer.org/debull/A21sept/p27.pdf Sepúlveda-Torres, R., Bonet-Jover, A., & Saquete, E. (2021). “Here are the rules: Ignore all rules”: Automatic contradiction detection in Spanish. Applied Sciences, 11(7), 1–15. doi:10.3390/app11073060 Shelby, L. B., & Vaske, J. J. (2008). Understanding meta-analysis: A review of the methodological literature. Leisure Sciences, 30(2), 96–110. doi:10.1080/01490400701881366 Shi, L., Bhattacharya, N., Das, A., Lease, M., & Gwizdka, J. (2022). The effects of interactive AI design on user behavior: An eye-tracking study of fact-checking COVID-19 claims. In ACM SIGIR Conference on Human Information Interaction and Retrieval (pp. 315–320). New York, NY: ACM. doi:10.1145/3498366.3505786 Silva, R. M., Santos, R. L. S., Almeida, T. A., & Pardo, T. A. S. (2020). Towards automatically filtering fake news in Portuguese. Expert Systems with Applications, 146(113199), 1–14. doi:10.1016/j.eswa.2020.113199 Singer, J. B. (2018). Fact-checkers as entrepreneurs: Scalability and sustainability for a new form of watchdog journalism. Journalism Practice, 12(8), 1070–1080. doi:10.1080/17512786.2018.1493946 Singer, J. B. (2021). Border patrol: The rise and role of fact-checkers and their challenge to journalists’ normative boundaries. Journalism, 22(8), 1929–1946. doi:10.1177/1464884920933137 Singh, P., Das, A., Li, J. J., & Lease, M. (2021). The case for claim difficulty assessment in automatic fact checking. Retrieved from http://arxiv.org/abs/2109.09689 mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 21 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Sirén-Heikel, S., Kjellman, M., & Lindén, C.-G. (2022). At the crossroads of logics: Automating newswork with artificial intelligence—(Re)defining journalistic logics from the perspective of technologists. Journal of the Association for Information Science and Technology, 74(3), 354–366. doi:10.1002/asi.24656 Školkay, A., & Filin, J. (2019). A comparison of fake news detecting and fact-checking AI based solutions. Studia Medioznawcze, 20(4), 365–383. doi:10.33077/uw.24511617.ms.2019.4.187 Snyder, H. (2019). Literature review as a research methodology: An overview and guidelines. Journal of Business Research, 104(2019), 333–339. doi:10.1016/j.jbusres.2019.07.039 Stray, J. (2019). Making artificial intelligence work for investigative journalism. Digital Journalism, 7(8), 1076– 1097. doi:10.1080/21670811.2019.1630289 Svahn, M., & Perfumi, S. C. (2021). A conceptual model for approaching the design of anti-disinformation tools. In International Conference on Electronic Participation (pp. 66–76). Cham, Switzerland: Springer. doi:10.1007/978-3-030-82824-0_6 Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8(45), 1–10. doi:10.1186/1471-2288-8-45 Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018). FEVER: A large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol 1: Long Papers, pp. 809– 819). Stroudsburg, PA: Association for Computational Linguistics. doi:10.18653/v1/n18-1074 Trompette, P., & Vinck, D. (2009). Revisiting the notion of boundary object. Revue d’Anthropologie des Connaissances, 3(1), a–v. doi:10.3917/rac.006.0003 Umarova, K., & Mustafaraj, E. (2019). How partisanship and perceived political bias affect Wikipedia entries of news sources. In Companion Proceedings of the 2019 World Wide Web Conference (pp. 1248–1253). New York, NY: ACM. doi:10.1145/3308560.3316760 Vayansky, I., & Kumar, S. A. P. (2020). A review of topic modeling methods. Information Systems, 94(2020). doi:10.1016/j.is.2020.101582 mailto:topacademicjournals@gmail.com American Journal of Art and Communication Vol.7, Issue 2; March-April - 2022; 1252 Columbia Rd NW, Washington DC, United States https://topjournals.org/index.php/AJAC; mail: topacademicjournals@gmail.com 22 | American Journal of Art and Communication | https://topjournals.org/index.php/AJAC Vizoso, Á., Vaz-Álvarez, M., & López-García, X. (2021). Fighting deepfakes: Media and internet giants’ converging and diverging strategies against hi-tech misinformation. Media and Communication, 9(1), 291–300. doi:10.17645/mac.v9i1.3494 Vlachos, A., & Riedel, S. (2014). Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science (pp. 18–22). Stroudsburg, PA: Association for Computational Linguistics. doi:10.3115/v1/w14–2508 Vryzas, N., Katsaounidou, A., Vrysis, L., Kotsakis, R., & Dimoulas, C. (2022). A prototype web application to support human-centered audiovisual content authentication and crowdsourcing. Future Internet, 14(3), 1– 17. doi:10.3390/fi14030075 Wang, W. Y. (2017). “Liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol. 2: Short Papers, pp. 422–426). Vancouver, Canada: Association for Computational Linguistics. doi:10.18653/v1/p17-2067 Welbers, K., Van Atteveldt, W., & Benoit, K. (2017). Text analysis in R. Communication Methods and Measures, 11(4), 245–265. doi:10.1080/19312458.2017.1387238 Yang, K.-C., Varol, O., Davis, C. A., Ferrara, E., Flammini, A., & Menczer, F. (2019). Arming the public with artificial intelligence to counter social bots. Human Behavior and Emerging Technologies, 1(1), 48–61. doi:10.1002/hbe2.115 Yang, Y., Zheng, L., Zhang, J., Cui, Q., Li, Z., & Yu, P. S. (2018). TI–CNN: Convolutional neural networks for fake news detection. Retrieved from http://arxiv.org/abs/1806.00749 Zeng, X., Abumansour, A. S., & Zubiaga, A. (2021). Automated fact checking: A survey. Language and Linguistics Compass, 15(10), 1–21. doi:10.1111/lnc3.12438 Zhou, J., Hu, H., Li, Z., Yu, K., & Chen, F. (2019). Physiological indicators for user trust in machine learning with influence enhanced fact-checking. In A. Holzinger, P. Kieseberg, A. Tjoa, & E. Weippl (Eds.), Machine learning and knowledge extraction (pp. 73–88). Cham, Switzerland: Springer. doi:10.1007/978- 3-030-29726-8_7 Zlatkova, D., Nakov, P., & Koychev, I. (2019). Fact-checking meets fauxtography: Verifying claims about images. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (pp. 2099–2108). Stroudsburg, PA: Association for Computational Linguistics. doi:10.18653/v1/d19-1216 mailto:topacademicjournals@gmail.com