AoIr 2024-final_SPIR_submission


Selected Papers of #AoIR2024:
The 25th Annual Conference of the

Association of Internet Researchers
Sheffield, UK / 30 Oct - 2 Nov 2024

HOW FACT-CHECKERS ARE BECOMING MACHINE LEARNERS: A
CASE OF META’S THIRD PARTY PROGRAMME

Yarden Skop
CRC Media of Cooperation, University of Siegen, Germany

Anna Schjøtt Hansen1

University of Amsterdam, Netherlands

Introduction

Political fact-checking, emerging in the US at the beginning 2000s as a particular form
of journalism, has expanded to an international movement with hundreds of
organisations across the globe (Duke Reporters lab, 2023). One of the recent
developments in the field is what some scholars call the “debunking turn” (Graves et al.
2023), in which fact-checking organisations move from fact-checking expressions of
politicians and public figures to checking claims made on social media.

A main driver of this change is the proliferation of a paid program initiated by Meta,
where fact-checkers check and label claims on the platform in exchange for monetary
remuneration (Meta, 2021). In this process, items (e.g., Facebook posts) are pushed to
fact-checkers through an internal interface created for them by Facebook, which
includes ML components to identify potential false claims (Ananny, 2020).

This paper draws on interviews with and fieldwork amongst fact-checkers who are or
have been part of the Meta partnership. Based on the empirical insights we argue that
the human-machine assemblage in fact-checking is (1) enabling a move beyond the
‘debunking turn’ by turning journalists into ‘machine learners’ (Mackenzie, 2017) and (2)
cements a ‘politics of demarcation’ (Marres, 2018) in which public contestation over
public facts is diminished and moved into networked infrastructures (Annany, 2020).

1 The authors contributed equally to research and writing

Suggested Citation (APA): Skop Y and Schjøtt Hansen A. (2024, October). How Fact-Checkers Are
Becoming Machine Learners: A Case of Meta’s Third Party Programme. Paper presented at AoIR2024:
The 25th Annual Conference of the Association of Internet Researchers. Sheffield, UK: AoIR. Retrieved
from http://spir.aoir.org.

http://spir.aoir.org


With this argument, the paper highlights an additional aspect of the platformisation of
journalism, as the labelling and claim-checking work of journalists now also enables
large tech platforms to expand technical infrastructures that commodify journalistic work
by turning it into training data aimed at improving their ML systems and algorithms. This
enables platforms to move further beyond their current market role (Nieborg and Poell,
2018), as they also participate in the further industrialisation and standardisation of
fact-checking.

As large tech companies become industry leaders in the provision of ML systems for,
among other things, fact-checking, the need to understand what politics they produce
equally increases, as they become integral in the production of democratic ideals of
citizens and public debate (Marres, 2018; Annany, 2020).

The Fact-Checking Assemblage

In this paper, we approach fact-checking as a sociotechnical phenomenon, where
human fact-checkers and machine learning systems work in collaboration to select,
produce and present ‘facts’ on Facebook. Concretely, we draw on existing work that
engaged with machine learning and its interfaces as sociotechnical assemblages
(Bucher, 2013; Schjøtt Hansen and Hartley, 2021; Rieder and Skop, 2021, see also
Deleuze and Guattari, 1987) that highlight how agency emerges through the assembling
of heterogeneous actors (human and non-human). As Bucher (2013: 481) writes: “(...)
Facebook, by organizing heterogeneous relations in a specific way, constitutes a
productive force: it makes new relations possible”.

In this paper, it is the ‘productiveness’ of this assemblage that is at the centre of the
analysis, as we explore how it produces and limits certain agencies for fact-checkers. In
analysing what we refer to as the ‘fact-checking assemblage’, we, therefore, pay careful
attention to how the machine learning system by Meta, its interface and affordances
produce certain understandings of fact-checkers, facts, and the political role of
fact-checking in society. In turn, we also show examples of ways in which fact-checkers
negotiate with the system or simply circumvent it (see also Schjøtt and Bengtsson,
2024).

Methodology

This contribution is based on two bodies of empirical work done separately. The first is a
series of semi-structured interviews with fact checkers who are members of the
International Fact Checkers’ Network (IFCN) and partake in Meta’s third-party
fact-checking program (3PFC). The group is quite heterogeneous in terms of
geographic spread and includes 18 fact-checkers from eight countries and three
continents. Some of them worked directly with Facebook as part of the partnership and
some were related to the periphery of the project in different ways. Interviews were
conducted in March 2021-November 2023, each interview lasting 45 to 90 minutes.
Background and contextual data were also gathered from Facebook’s public website,
which discussed the program, and attendance at two fact-checking global summits,

https://www.zotero.org/google-docs/?kI7HbK
https://www.zotero.org/google-docs/?kI7HbK
https://www.zotero.org/google-docs/?f3I2G2


“Global Fact”, on June 2022 in Oslo and June 2023 in Seoul.

The second body of data is comprised of observations collected during an ethnographic
enquiry at Tjekdet.dk, the Danish fact-checking organisation that acts as the third-party
fact-checker for Meta in Denmark. The fieldwork was conducted in the spring of 2021
and over four months with weekly full-day visits to their offices in Copenhagen.

The interviews were analysed using Abductive Analysis (Tavori and Timmermans,
2014). Similarly, the observational data was iteratively coded using a ‘bottom-up’
strategy to develop thematic codes (Gibbs, 2012). In this analysis, overlapping codes
regarding the fact-checkers attempts to ‘train’ the system are combined, together with
codes relating to the underlying assumptions tied to the democratic role of fact-checking
and the way these were negotiated through the affordances provided by Meta.

Preliminary Analysis

The core of the work fact-checkers do for Meta is labelling claims made by users on the
platform, based on a list of labels provided by Meta. These are False, Altered, Partly
False, Missing Context, True, and Satire (Meta, 2021). Fact-checkers also write an
article on each fact check, which they get paid for. However, the interface visible to
users on Meta platforms only promotes the label and fact-checking organisation name,
and reading the article, where additional context is provided, demands clicking to move
to another link. On their organisational public websites, some fact-checking
organisations use other labels or do not use short labels at all. In the analysis, we show
how the labelling practice has a structuring and disciplinary effect, cementing certain
understandings of public truths or falsehoods, and forcing fact-checkers to comply with
these.

We furthermore illustrate how the fact-checkers were either directed to ‘train’ the system
by adding the ‘true’ label to content in the initial phase of their participation in the
program or took it upon themselves to label false positives because they assumed it
would help enhance the workings of the system. Thereby, the fact-checkers became
part of the system as active ‘machine learners’ (Mackenzie, 2017), who via global
labelling schemes contribute to targeted training of the ML system in different
languages. However, this work reduces the richness of fact-checking by further
standardising how it can be carried out. Something that was also noted by the
fact-checkers who would find the scheme limiting and either leave the programme or
circumvent it by publishing other fact-checks, not identified via Meta’s claim check
interface, which was deemed of more societal value by the fact-checkers.

Based on the analysis we draw attention to how the flow of labour and professional
freedom of fact-checkers is being transformed and industrialised, as it is boiled down to
content labelling that is used on the platform and for deeper enhancement of ML
systems. In this way, platformised infrastructures offered to them by Meta, not only
serve the platform in real-time but also allow for a long-term commodification of fact



checkers’ work and expertise, which might later render them redundant. We also
illustrate how the Meta fact-checking program, via the structuring of fact-checking work,
is participating in the construction of what ‘facts’ are and how they can and should be
publicly dealt with. Thereby, cementing both the liberal ideals of the citizen and the
value of ‘debunking’, as opposed to public deliberation over facts.

References

Ananny M (2020) Making up Political People: How Social Media Create the Ideals,
Definitions, and Probabilities of Political Speech. preprint, 29 May. Open Science
Framework. Available at: https://osf.io/7pd62 (accessed 1 September 2021).

Bucher T (2013) The Friendship Assemblage: Investigating Programmed Sociality on
Facebook. Television & New Media 14(6). SAGE Publications: 479–493.

Deleuze G and Guattari F (1987) A Thousand Plateaus: Capitalism and Schizophrenia
(tran. B Massumi). 2nd edition. Minneapolis: University of Minnesota Press.

Duke Reporters Lab (2023) Misinformation Spreads, but Fact-Checking has Leveled
Off. Available at: https://reporterslab.org/tag/fact-checking-census/ (Accessed: 29
February, 2024).

Gibbs GR (2012) Different approaches to coding. Sociological Methodology, 42(1),
82-84.

Graves L, Bélair-Gagnon V and Larsen R (2023) From Public Reason to Public Health:
Professional Implications of the “Debunking Turn” in the Global Fact-Checking Field.
Digital Journalism: 1–20.

Mackenzie A (2017) Machine Learners: Archaeology of a Data Practice. Cambridge,
MA: The MIT Press.

Marres N (2018) Why We Can’t Have Our Facts Back. Engaging Science, Technology,
and Society 4: 423–443.

Nieborg DB and Poell T (2018) The platformization of cultural production: Theorizing the
contingent cultural commodity. New Media & Society 20(11): 4275–4292.

Meta (2021) How Meta’s third-party fact-checking program works. Available at:
https://www.facebook.com/formedia/blog/third-party-fact-checking-how-it-works
(Accessed: 29 February, 2024).

Rieder B and Skop Y (2021) The fabrics of machine moderation: Studying the technical,
normative, and organizational structure of Perspective API. Big Data & Society 8(2):
205395172110461.

https://reporterslab.org/tag/fact-checking-census/
https://www.facebook.com/formedia/blog/third-party-fact-checking-how-it-works


Schjøtt, A., & Bengtsson, M. (2024). De- and recoding algorithmic systems: The case of
fact checkers and fact checked users. Convergence, 0(0).
https://doi.org/10.1177/13548565241289233

Schjøtt Hansen A and Hartley JM (2021) Designing What’s News: An Ethnography of a
Personalization Algorithm and the Data-Driven (Re)Assembling of the News. Digital
Journalism 0(0). Routledge: 1–19.

Tavory I and Timmermans S (2014) Abductive Analysis: Theorizing Qualitative
Research. Chicago: The University of Chicago Press.

https://doi.org/10.1177/13548565241289233