Selected Papers of #AoIR2024: The 25th Annual Conference of the Association of Internet Researchers Sheffield, UK / 30 Oct - 2 Nov 2024 Zhang, B. Z., Yang, T., Haimson, O. L., & Thomas, M. (2024, October). Conceptualizing Precision Labor in Artificial Intelligence Training. Paper presented at AoIR2024: The 25th Annual Conference of the Association of Internet Researchers. Sheffield, UK: AoIR. Retrieved from http://spir.aoir.org. CONCEPTUALIZING PRECISION LABOR IN ARTIFICIAL INTELLIGENCE TRAINING Ben Zefeng Zhang University of Michigan, Tianling Yang Technische Universität Berlin & Weizenbaum Institut, Oliver L. Haimson University of Michigan, Michaelanne Thomas University of Michigan, Introduction Accuracy and precision are often closely associated with the objectivity (Grill, 2022; MacKenzie, 1993; Suchman, 2020; Wise, 1997), scalability (Tsing, 2012), and intelligence (Crawford, 2021), standing in stark contrast to “ambiguity, uncertainty, messiness, and unreliability” (Wise, 1997). What does it take to achieve a high level of technical accuracy? What are the harms resulting from technology companies’ obsession with technical accuracy and precision, and who incurs the greatest burdens? Using AI training as a context, this work documents AI workers’ everyday working practices, challenges, and harms under the guise of achieving extreme levels of technical precision demanded by clients and ML practitioners. In this work, we offer an ethnographic account of AI trainers, a relatively understudied but emergent category of laborers in China. They often take tasks from data annotation to tasks that complement, augment, and sometimes substitute for autonomous AI systems when they fall short (Miceli & Posada, 2022; Wu, 2023). Previous literature has examined the hidden labor behind AI (Ekbia & Nardi, 2014; Gray & Suri, 2019); specifically on the human labor to produce ML datasets. Existing work has investigated involved work practices, workflows, and social and organizational contexts (Miceli et al., 2020; Muller et al., 2019; Zhang et al., 2020). Informed by feminist scholarship on work and labor (Baym, 2015; Hochschild, 2022), we synthesize and extend our findings by introducing the concept of precision labor. By precision labor, we refer to the hidden work involved in erasing the messy, ambiguous, and uncertain aspects of technology production, all in the pursuit of presenting technology as objective, truthful, and productive, despite such pursuit can be unnecessary, arbitrary, and harmful to laborers. Specifically, precision labor contributes to a new way of understanding the disproportionate impact of unnecessary and unrecognized labor as well as emergent harms such as financial precarity and machine subordination on digital labor communities within AI production. Methods We relied on multi-sited ethnographic fieldwork conducted between January 2023 to January 2024 in China, within the context of AI training. Ethnography is particularly apt for studying work and labor; it highlights the “interpretative flexibility of technological artifacts” (Pinch & Bijker, 1984), allowing us to understand the intricacies, significance, and politics surrounding AI training. To explore Chinese AI trainers' motivations, aspirations, and everyday professional experiences, the first author worked as an AI trainer in data annotation centers, which involves observation and hands-on participation in activities ranging from work projects, team meetings, lunch meet-ups with colleagues, AI training certificate courses and examinations, to major industry events. Much of the empirical data presented in this paper is based on 9 weeks of ethnographic research conducted by the first author when they worked as a data work intern in an AI data annotation center. The data for this paper is grounded in 16 formal interviews with AI trainers and over 150 pages of field notes and online archives. We anonymized all data in this paper and used pseudonyms for participants, companies, and locations to ensure data privacy and security. On the ground, the first author relied on aspects of grounded theory to guide their research (Charmaz, 2006). In the iterative process of data coding, accuracy and precision emerged as one of the dominant themes. Findings Our findings are structured around how technical accuracy is perceived, established, and managed and its associated negative consequences. Perceiving and establishing accuracy. Many workers at data centers believe that the AI training work they do is vital and either directly or indirectly contributes to Al system refinement; yet they understand little about the mechanisms behind AI. Narratives about doing AI training work correctly and accurately were often emphasized even before the employees started working as AI trainers. In the context of AI training, accuracy can have multiple meanings, often depending on the specific context, as illustrated in Figure 1. For instance, tasks and projects involving voice recognition, computer vision, and generative AI technologies could have distinct measurements and requirements. Nevertheless, across various projects, there tends to be a common understanding of accuracy based on a statistical measure that standardizes accuracy. This is “unifying accuracy,” referring to the overall accuracy rate of projects, which is typically required to be over 95%. Despite the multiple meanings and the contextual nature of accuracy, for many workers, accuracy is more of a homogeneous and arbitrary production goal, manifested through unifying accuracy set by the clients and reinforced by different actors. Zimo, who has worked as both an annotator and reviewer, revealed: “Accuracy means meeting clients' requirements. some are set at 98%, others at 99%. The client will approve your project if you achieve their accuracy expectations.” Figure 1. The establishment of accuracy is understood in different scenarios and dimensions. Managing and Making Performative Accuracy. In order to meet clients' specified accuracy standards, workers often undergo training that can, at times, become excessive. In addition, a standard workflow is often established, involving various actors: clients, intermediaries, and service providers. The clients relay data to intermediaries, who then delegate tasks to the service providers. Within the service provider's framework are annotators, reviewers, quality inspectors, and managers. Figure 2, adapted from the company figure and used widely across the country, exemplifies a flowchart designed to ensure perceived high accuracy. Once annotators complete their data annotations, they submit their work to the reviewers. If the reviewers identify multiple errors or believe the accuracy rate is below the threshold, they return the data to the annotators for revision. Once the task passes the reviewers' standards, quality inspectors conduct additional accuracy checks. Successful clearance from both reviewers and quality inspectors means the annotated data could be submitted to the inspection staff appointed by the clients, who sometimes employ algorithms to test the data to ensure that the model performance meets high accuracy expectations. If any stage of this verification process is unsuccessful, the data is either returned to the service provider or the project is deemed a failure. Figure 2. An emblematic example of workflow in AI training projects to ensure high accuracy. Throughout the fieldwork across different contexts, we also observed a hybrid form of labor control and management style. Workers are being controlled and managed by traditional and factory-like settings as well as gamification in an on-demand economy, but not enjoying the benefits of either, specifically, the social benefits and protection in the former or the flexibility in the latter. Moreover, quantification and gamification techniques, as well as several punitive measures such as payment deduction, suspensions, and mandatory unpaid retraining, were used not only to foster productivity but also to uphold accuracy standards, but they often prove unnecessary, harmful, and counterproductive for workers. Our findings also illuminate that the pursuit of performative accuracy could also lead to workers becoming subordinate to machines. While striving for optimal accuracy, workers' decision-making often becomes influenced by, and sometimes even subservient to, machine outputs, which are often regarded as potential arbiters of the ground truth. Importantly, during the pursuit of high technical accuracy, they have to adapt their thought processes, internalize their frustrations, and navigate financial instability, often at the cost of their physical and mental well-being. Conclusion The concept of precision labor is particularly relevant in today's technology landscape, where technology production is increasingly algorithmic and data-driven, and laborers are increasingly feeling compelled to comply with the demand for machine intelligence. Our findings have shown precision labor is often materialized through and amplified by the obsession with the pursuit of performative accuracy and precision. We propose precision labor as a lens to understand the often unnecessary and unrecognized labor, along with the various harms stemming from the relentless quest for technical accuracy. It invites further investigations to scrutinize the legitimacy of technical accuracy, question its reasonableness and sustainability, and call for enhanced reflexivity and timely intervention. References Baym, N. K. (2015). Connect With Your Audience! The Relational Labor of Connection. The Communication Review, 18(1), 14–22. https://doi.org/10.1080/10714421.2015.996401 Charmaz, K. (2006). Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. SAGE. Crawford, K. (2021). The Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press. Daniels, A. K. (1987). Invisible Work. Social Problems, 34(5), 403–415. https://doi.org/10.2307/800538 Ekbia, H., & Nardi, B. (2014). Heteromation and its (dis)contents: The invisible division of labor between humans and machines. First Monday. https://doi.org/10.5210/fm.v19i6.5331 Gray, M. L., & Suri, S. (2019). Ghost work: How to stop Silicon Valley from building a new global underclass. Houghton Mifflin Harcourt. Grill, G. (2022). Constructing Certainty in Machine Learning: On the performativity of testing and its hold on the future. https://doi.org/10.31219/osf.io/zekqv Hochschild, A. R. (2022). The Managed Heart. In Working in America (5th ed.). Routledge. MacKenzie, D. (1993). Inventing Accuracy: A Historical Sociology of Nuclear Missile Guidance. MIT Press. Miceli, M., & Posada, J. (2022). The Data-Production Dispositif (arXiv:2205.11963). arXiv. http://arxiv.org/abs/2205.11963 Miceli, M., Schuessler, M., & Yang, T. (2020). Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2), 115:1-115:25. https://doi.org/10.1145/3415186 Muller, M., Lange, I., Wang, D., Piorkowski, D., Tsay, J., Liao, Q. V., Dugan, C., & Erickson, T. (2019). How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–15. https://doi.org/10.1145/3290605.3300356 Pinch, T. J., & Bijker, W. E. (1984). The Social Construction of Facts and Artefacts: Or How the Sociology of Science and the Sociology of Technology might Benefit Each Other. Social Studies of Science, 14(3), 399–441. https://doi.org/10.1177/030631284014003004 Suchman, L. (2020). Algorithmic warfare and the reinvention of accuracy. Critical Studies on Security, 8(2), 175–187. https://doi.org/10.1080/21624887.2020.1760587 Tsing, A. L. (2012). On Nonscalability: The Living World Is Not Amenable to Precision- Nested Scales. Common Knowledge, 18(3), 505–524. https://doi.org/10.1215/0961754X-1630424 Wise, M. N. (1997). The Values of Precision. Princeton University Press. Wu, D. (2023). Good for tech: Disability expertise and labor in China’s artificial intelligence sector. First Monday. https://doi.org/10.5210/fm.v28i1.12887 Zhang, A. X., Muller, M., & Wang, D. (2020). How do Data Science Workers Collaborate? Roles, Workflows, and Tools. Proceedings of the ACM on Human- Computer Interaction, 4(CSCW1), 22:1-22:23. https://doi.org/10.1145/3392826