Weinberg, Lindsay (2020). Feminist Research Ethics and Student Privacy in the Age of AI. Catalyst: Feminism, Theory, Technoscience, 6(2), 1–10. http://www.catalystjournal.org | ISSN: 2380-3312 © Lindsay Weinberg, 2020 | Licensed to the Catalyst Project under a Creative Commons Attribution Non-Commercial No Derivatives license Feminist Research Ethics and Student Privacy in the Age of AI Lindsay Weinberg Purdue University Lweinber@purdue.edu Abstract This article examines university researchers’ capture of student images on US college campuses for training facial recognition technology, and situates this project within universities’ broader historical alignment with militarism and racial injustice. It argues that feminist STS ethics provides a framework for not only challenging the ways that university research inquiry actively contributes to oppressive power structures, but also for reimagining university research ethics for a greater engagement with questions of justice. The article identifies the limitations of dominant institutional ethics and privacy rights discourses for centering justice considerations, and instead outlines an intersectional feminist approach to university research ethics that reimagines the relationship between research processes, power, and social impacts. Between 2012 and 2014, researchers at Duke University, Stanford University, and the University of Colorado at Colorado Springs (UCCS) captured images of students traveling between classes and visiting on-campus cafés. These images were taken to help improve artificial intelligence’s (AI) capacity to recognize people in motion within a “natural” environment (Fussell, 2019). At UCCS, in a study approved by the Institutional Review Board (IRB), long-range surveillance images of over 1,700 people were collected from February 2012 to September 2013 without students’ knowledge or consent, which helped to produce a dataset Critical Perspectives | Catalyst: Feminism, Theory, Technoscience Issue 6 (Vol 2) Lindsay Weinberg, 2020 2 with desired real-world conditions—poor lighting and partially obstructed views— for defense and intelligence agencies (Harvey & LaPlace, 2019b). In the case of Duke University, the lead researcher made significant changes to the approved 2014 project that went unreported to the IRB, including expanding the project’s scope from the indoors to the outdoors, and giving data access not only to researchers upon request, but to anyone as a publicly available dataset. The Duke dataset has gone on to be used for over one hundred publicly available research papers since its initial publication in 2016 (Harvey & LaPlace, 2019a). Artist-researchers Adam Harvey and Jules LaPlace (2019a) have tracked these papers’ connections to the Chinese military and to companies that help Chinese authorities monitor and oppress Uighur Muslims, as well as to the United States Department of Homeland Security, the Intelligence Advanced Research Projects Activity, IBM, and Microsoft, who has provided surveillance to Immigration and Customs Enforcement. Furthermore, the Duke University study was funded in part by the United States Army Research Laboratory, and the UCCS study received funding from the US Navy. The large-scale government funding of these projects is in stark contrast to the work of feminist researchers, whose projects are often underfunded, undervalued, and poorly understood (Cooky et al., 2018). Yet feminist research holds some of the greatest promise for critically evaluating technology’s development and impacts on society. As a research methodology, feminist research investigates how power shapes research inquiry and discovery, including the production and application of scientific knowledge. How did power shape inquiry and discovery in the context of the UCCS and Duke University facial recognition studies? Furthermore, how can feminist STS help us move university research ethics beyond questions of liability and IRB legitimization, and towards a greater engagement with questions of justice? A close reading of these studies through an intersectional feminist lens shows how dominant relations of power deeply informed the research inquiry process. In particular, it illustrates how research inquiry for improving facial recognition appropriates critiques of non-inclusive technology to justify the expansion of surveillance. Several of the lead researchers on these projects have defended their work by stating that these datasets will ultimately result in more accurate AI. Existing forms of AI have been shown time and again to have higher error rates for women of color (Buolamwini & Gebru, 2018) and transgender people (Keyes, 2018). However, this rationale ultimately justifies the widening of surveillance’s net, and by proxy, the reach of systems of oppression that deploy forms of Critical Perspectives | Catalyst: Feminism, Theory, Technoscience Issue 6 (Vol 2) Lindsay Weinberg, 2020 3 automated recognition. Simone Browne’s (2015) work has demonstrated the long historical arc of the relationship between surveillance and racial oppression, from the transatlantic slave trade to contemporary forms of racial profiling. More accurate facial recognition will not rectify a criminal justice system that is inherently unfair, given the ways histories of slavery and segregation, as well as ongoing race- and class-based oppressions, shape processes of criminalization, resulting in disparate treatment for communities of color and cycles of poverty and incarceration. This is why Browne argues for “fugitive acts of escape, resistance, and productive disruptions” (2015, p. 163), rather than demanding more accurate facial recognition technology for Black people. Feminist surveillance studies scholars have illustrated how practices of surveillance are always already bound up with the monitoring and disciplining of boundaries that sustain racism, patriarchy, and capitalist exploitation (Dubrofsky & Magnet, 2015). Efforts to make vulnerable groups more legible to these systems thus speaks to the ways ideas of inclusion, equity, and identity have been co- opted for the purposes of improving structures of control (Blas & Gaboury, 2016). This is why artists such as Zach Blas have argued for a queer politics of refusal, anonymity, and opacity, meaning a “tactical evasion of the gaze of digital machines…[that] accounts for the specificities of subjects” (2016, p. 150). For instance, in his project Facial Weaponization Suite (2011–2014), Blas created a series of neon-colored masks designed to thwart facial recognition technology, grounded in a critique of the technology's discriminatory and violent impacts on racial, sexual, and religious minorities (Lingel, 2020). Furthermore, a politics of refusal against facial recognition technology has seen legislative success, due to the efforts of diverse grassroots coalition-based movements (Dubal, 2019). New municipal ordinances have banned city agencies from using facial recognition technology in San Francisco and Oakland, California, as well as in Somerville, Massachusetts. In addition to challenging embedded power dynamics within research inquiry, feminist scholars have pointed out the limitations of the IRB for addressing the ways that intersecting power relations condition who gets to conduct research and how research impacts the rights and experiences of vulnerable groups. Yet these facial recognition studies were defended on the grounds of having received IRB approval, which federal regulations require for research involving human subjects. For instance, UCCS university spokesperson Jared Verner said that the IRB “assures the protection of the rights and welfare of human subjects in research” and that “no personal information was collected or distributed in this specific study” (Dent, 2019). The inadequacy of IRBs for meaningfully addressing Critical Perspectives | Catalyst: Feminism, Theory, Technoscience Issue 6 (Vol 2) Lindsay Weinberg, 2020 4 privacy rights in the context of machine learning research is particularly acute, given that the IRB focuses on whether individuals are identified, and if so, whether being identified puts them at risk of financial or medical harm. As Jasmine R. Lindabary and Danielle J. Corple have noted, “in the context of academic research, power is…nestled in institutions and structures (e.g., IRBs) that define what constitutes privacy and dictate whose vulnerabilities count” (2019, pp. 1458– 1459). Regardless of whether individuals in these studies are personally identified or not, their images are being used to train technologies that will have disparate impacts on vulnerable groups when put in the service of institutions that seek to monitor, criminalize, and punish, including policing and border security. Feminist critiques of research ethics have also emphasized how, over the last thirty years in the United States, institutional research ethics discourses have largely shifted from questions of what morals or rules should guide research conduct to questions of what will help mitigate concerns about university liability (Cooky et al., 2018). In the case of these studies, one liability concern would be whether students have a reasonable expectation of privacy in the areas of campus researchers intended on photographing. Yet a range of feminist scholars, social theorists, and critical legal scholars have demonstrated that traditional understandings of privacy typically do not account for the ways privacy intersects with relations of power and inequality (Gilliom, 2001). This traditional notion of a reasonable expectation of privacy originally comes from the Supreme Court decision in Katz v. United States (1967), which laid the foundations for what would become the “reasonable expectation” test, based on whether the individual exhibits an actual (subjective) expectation of privacy, and whether this expectation is “objectively” reasonable. In the lead researcher’s defense of the UCCS study, he noted that students have no reasonable expectation to privacy when walking around campus, and that there is nothing illegal about taking photos of people in public (Stanley, 2019). However, while one can easily imagine being seen, overheard, or even photographed by others in the vicinity, in the UCCS study, students were photographed from over 492 feet away without their knowledge, accelerating the advancement of multi-target multi-camera tracking and other forms of computer vision.1 The UCCS study thus demands a consideration of how dominant understandings of privacy norms, which the academy institutionalizes and reproduces, fail to address how advances in digital technology intensify conditions of vulnerability to structures of domination. Power plays a role in what norms become institutionalized and accepted within a given context, as feminist theorists have long argued (Rule, 2019). While some privacy scholars have suggested that we should turn to emerging social norms to Critical Perspectives | Catalyst: Feminism, Theory, Technoscience Issue 6 (Vol 2) Lindsay Weinberg, 2020 5 determine appropriate political responses to innovations in information technology (Nissenbaum, 2010), this framework hinges on the idea that norms are a reliable indicator of how much information, and what type of information collection, should be permissible. In the case of the Duke University study, lead researchers pointed out that, on the heels of the controversy, no students had come forward requesting their image to be removed from the dataset (Satisky, 2019). While this might be read as suggesting that students did not experience the research study as a privacy invasion, it is worth considering how dominant understandings of the work universities do in the world might inform students’ reception of this study. These dominant understandings confer benevolence upon university research, on the one hand, and conceal its longstanding complicity in social injustice, on the other. Within dominant discourse, there is a pervasive assumption that the university’s “drive to know the world is virtuous” (Boggs & Mitchell, 2018, p. 435). Abigail Boggs and Nick Mitchell (2018) have shown that even within critical university studies scholarship, there is an assumption that the university is, at its core, progressive and a good in itself. Boggs and Mitchell point to the work of scholars like Roderick A. Ferguson (2012), Craig Steven Wilder (2013), Stefano Harney and Fred Moten (2013), Piya Chatterjee and Sunaina Maira (2014), Jennifer Doyle (2015), and Tressie McMillan Cottom (2017), who show how universities are deeply entangled in histories of racial injustice, US imperialism, the suppression of leftist social movements, and class inequality. These histories suffer frequent erasure, including within pedagogical approaches to research ethics. Students are often taught to understand research ethics through sensationalist lenses that exceptionalize cases like the Tuskegee experiment, and which, as Sheeva Sabati explains, “obscure the forms of pervasive, iterative violences that have been sustained through projects of knowledge production” (2019, p. 1058). Given that these facial recognition studies were funded, in part, through military agency spending, it would be helpful to situate the use of student images to train facial recognition software using a critical feminist perspective that accounts for the historical relationship between university research and the military-industrial complex. Dominant scientific and technical epistemologies in university research are deeply tied to histories of militarism, capitalism, colonialism, and male supremacy (Haraway, 1988). With the US’s rise as a global superpower during World War II, President Roosevelt issued an executive order to create the Office of Scientific Research and Development, which decisively aligned university research efforts with militaristic national policy priorities. Universities would go on to produce Critical Perspectives | Catalyst: Feminism, Theory, Technoscience Issue 6 (Vol 2) Lindsay Weinberg, 2020 6 research for missile technology, gun sights, bombsights, radars, chemical and biological weapons, and the atom bomb for the war effort (Abrams, 1989). Under McCarthyism, public universities’ state funding shifted from land-grant agricultural resources to the defense establishment, which further laid the groundwork for “deep strata of connection and complicity between imperial statecraft and the knowledge complex of the U.S. academy” (Chatterjee & Maira, 2014, p. 13). By 1980 the Department of Defense as well as the State Department were likely to insist on limiting access to state-funded university research, arguing that transparency about new technology could jeopardize the nation’s economic standing, and by proxy, national security (Abrams, 1989). Today, the Department of Defense is playing an increasingly large role in funding research that was previously supported by the National Science Foundation, the budget of which has been significantly curtailed under the Trump administration. Furthermore, several contemporary partnerships between the academy, the US government, and private business have resulted in the proliferation of whitewashed technology ethics discourses that are strategically aligned with both the priorities of the US military and Silicon Valley (Ochigame, 2019). Thus, a feminist perspective that takes seriously the relationship between power, research inquiry, and discovery urges us to understand these controversial facial recognition studies not in isolation, but as forms of knowledge production that are symptomatic of the university’s broader historical alignment with the defense establishment, on the one hand, and with contemporary austerity and corporatization measures, on the other. Duke University and UCCS have since removed access to the datasets derived from these projects. As media attention wanes, the collection of student data within the university requires ongoing critical feminist inquiry, as do university partnerships with both the US government and for-profit companies. These partnerships stand to become increasingly popular with higher education administrators as state funding dwindles, and are mostly led by male researchers, thus further amplifying male privilege within the academy (Smith-Doerr & Croissant, 2011). These research efforts can directly and indirectly support US projects of criminalization domestically and imperialism abroad. However, simply adding women to these partnerships will not necessarily upend the reproduction of systems of oppression (Smith-Doerr & Croissant, 2011); rather, insurgent, feminist, justice-oriented approaches to research inquiry and discovery are needed. Existing IRB policy and privacy law, even when appropriately followed, are not up to the task of ensuring ethical research practices concerning student data, nor just research outcomes. This is in part due to institutionalized research ethics’ emphasis on liability and immediate risks, which silo questions of long- Critical Perspectives | Catalyst: Feminism, Theory, Technoscience Issue 6 (Vol 2) Lindsay Weinberg, 2020 7 term effects. While some might argue that demanding university researchers address potential long-term impacts would slow down scientific progress, this argument presupposes progress itself to be an inherent social good. Instead, and as Ruja Benjamin argues, if we are “committed to building a more just society, it is vital that we demand a slower and more socially conscious innovation” (2019, p. 183). An intersectional feminist approach to university research ethics would demand that researchers who engage in technological development do so in a more reflective way, taking as a starting point the voices and experiences of those who are most vulnerable. This approach would require accounting for the social relations of power embedded within research inquiry and discovery, as well as the potential social impacts of a given technology’s development, in order to center justice considerations (Costanza-Chock, 2018). It would be a research ethics that goes beyond questions of liability and IRB legitimization, and instead foregrounds the ways that new knowledge-making practices might contribute to the epistemological and material violences that sustain systems of oppression, and thus reproduce the university’s longstanding imbrication in racial-colonial, patriarchal, and military histories. As Catherine D’Ignazio and Lauren F. Klein explain, “tracing these links to historical and ongoing forces of oppression can help us answer the ethical question, Should this system exist?” (2020, p. 13). A push for a feminist research ethics in the academy could help counter the underfunding and institutional marginalization of critical feminist approaches to technology (Shilton et al., 2017), and create more opportunities for work in solidarity with movements outside the academy for progressive technological reform.2 If these demands sound impossible to enact within the existing structures of higher education, then this speaks to the urgency with which the academy needs to be transformed. Notes 1 Students are certainly not the only group whose images have been used to test facial recognition technology without their knowledge or consent. This list includes immigrants, abused children, people who have had mug shots taken, and deceased people (Keyes et al., 2019). 2See, for example, the Movement for Black Lives and the #TechWontBuildIt campaign. Critical Perspectives | Catalyst: Feminism, Theory, Technoscience Issue 6 (Vol 2) Lindsay Weinberg, 2020 8 References Abrams, R. M. (1989). The US military and higher education: A brief history. The Annals of the American Academy, 502(1), 15–28. Benjamin, R. (2019). Race after technology: Abolitionist tools for a new Jim code. Polity Press. Boggs, A., & Mitchell, N. (2018). Critical university studies and the crisis consensus. Feminist Studies, 44(2), 432–463. Blas, Z. (2016). Opacities: An introduction. Camera Obscura, 31(2 92), 149–153. Blas, Z., & Gaboury, J. (2016). Biometrics and opacity: A conversation. Camera Obscura, 31(2 92), 155–165. Browne, S. (2015). Dark matters: On the surveillance of blackness. Duke University Press. Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 1– 15. Chatterjee, P., & Maira, S. (2014). The imperial university: Race, war, and the nation- state. In P. Chatterjee & S. Maira (Eds.), The imperial university: Academic repression and scholarly dissent (pp. 1–50). University of Minnesota Press. Cooky, C., Lindabary, J. R., & Corple, D. J. (2018). Navigating big data dilemmas: Feminist holistic reflexivity in social media research. Big Data & Society (July- December), 1–12. https://doi.org/10.1177/2053951718807731 Costanza-Chock, S. (2018). Design justice: Towards an intersectional feminist framework for design theory and practice. Proceedings of the Design Research Society 2018, 1–14. http://doi.10.21606/dma.2017.679 Cottom, T. M. (2017). Lower ed: The troubling rise of for-profit colleges in the new economy. The New Press. Dent, S. (2019). Colorado college students were secretly used to train facial recognition. Endgadget. https://www.engadget.com/2019/05/28/uccs-facial- recognition-study-students/?guccounter=1 D’Ignazio, C., & Klien, L. F. (2020). Data feminism. MIT Press. Doyle, J. (2015). Campus sex, campus security. Semiotext(e). Dubal, V. (2019, May 30). San Francisco was right to ban facial recognition. The Guardian. https://www.theguardian.com/commentisfree/2019/may/30/san-francisco- ban-facial-recognition-surveillance https://doi.org/10.1177%2F2053951718807731 https://www.engadget.com/2019/05/28/uccs-facial-recognition-study-students/?guccounter=1 https://www.engadget.com/2019/05/28/uccs-facial-recognition-study-students/?guccounter=1 https://www.theguardian.com/commentisfree/2019/may/30/san-francisco-ban-facial-recognition-surveillance https://www.theguardian.com/commentisfree/2019/may/30/san-francisco-ban-facial-recognition-surveillance Critical Perspectives | Catalyst: Feminism, Theory, Technoscience Issue 6 (Vol 2) Lindsay Weinberg, 2020 9 Dubrofsky R., & Magnet, S. A. (2015). Introduction. In R. Dubrofsky & S. A. Magnet (Eds.), Feminist surveillance studies (pp. 1–17). Duke University Press. Ferguson, R. A. (2012). The reorder of things: The university and its pedagogies of minority difference. University of Minnesota Press. Fussell, S. (2019, June). You no longer own your face: Students were recorded for research—and then became part of a data set that lives forever online, potentially accessible to anyone. The Atlantic. https://www.theatlantic.com/technology/archive/2019/06/universities-record- students-campuses-research/592537/ Gilliom, J. (2001). Overseers of the poor: Surveillance, resistance, and the limits of privacy. University of Chicago Press. Haraway, D. (1988). Situated knowledge: The science question in feminism and the privilege of partial perspective. Feminist Studies, 14(3), 575–599. Harney, S., & Moten, F. (2013). The undercommons: Fugitive planning and black study. Autonomedia. Harvey, A., & LaPlace, J. (2019a). Duke MTMC is a dataset of surveillance camera footage of students on Duke university campus. https://megapixels.cc/datasets/duke_mtmc/ Harvey, A., & LaPlace, J. (2019b). MegaPixels: Origins, ethics, and privacy implications of publicly available faces. https://megapixels.cc/ Keyes, O. (2018). The misgendering machines: Trans/hci implications of automatic gender recognition. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), article 88. Keyes, O., Stevens, N., & Wernimont, J. (2019, March 17). The government is using the most vulnerable people to test facial recognition software. Slate. https://slate.com/technology/2019/03/facial-recognition-nist-verification-testing- data-sets-children-immigrants-consent.html Lindabary, J. R., & Corple, D. J. (2019). Privacy for whom?: A feminist intervention in online research practice. Information, Communication & Society, 22(10), 1447–1463. Lingel, J. (2020). Dazzle camouflage as queer counter conduct. European Journal of Cultural Studies, 1–18. https://doi.org/10.1177/1367549420902805 Nissenbaum, H. (2010). Privacy in context: Technology, policy and the integrity of social life. Stanford Law Books. Ochigame, R. (2019, December 20). The invention of ethical AI: How big tech manipulates academia to avoid regulation. The Intercept. https://theintercept.com/2019/12/20/mit-ethical-ai-artificial- https://www.theatlantic.com/technology/archive/2019/06/universities-record-students-campuses-research/592537/ https://www.theatlantic.com/technology/archive/2019/06/universities-record-students-campuses-research/592537/ https://megapixels.cc/datasets/duke_mtmc/ https://megapixels.cc/ https://slate.com/technology/2019/03/facial-recognition-nist-verification-testing-data-sets-children-immigrants-consent.html https://slate.com/technology/2019/03/facial-recognition-nist-verification-testing-data-sets-children-immigrants-consent.html https://doi.org/10.1177%2F1367549420902805 https://theintercept.com/2019/12/20/mit-ethical-ai-artificial-intelligence/?comments=1 Critical Perspectives | Catalyst: Feminism, Theory, Technoscience Issue 6 (Vol 2) Lindsay Weinberg, 2020 10 intelligence/?comments=1 Rule, J. B. (2019). Contextual integrity and its discontents: A critique of Helen Nissenbaum’s normative arguments. Policy & Internet, 11(3), 260–279. Sabati, S. (2019). Upholding “colonial unknowing” through the IRB: Reframing institutional research ethics. Qualitative Inquiry, 25(9–10), 1056–1064. Satisky, J. (2019, June 12). A Duke study recorded thousands of students’ faces. Now they’re being used all over the world. The Chronicle. https://www.dukechronicle.com/article/2019/06/duke-university-facial-recognition- data-set-study-surveillance-video-students-china-uyghur Shilton, K., Zimmer, M., Fiesler, C., Narayanan, A., & Metcalf, J. (2017, November 15). We’re awake—but we’re not at the wheel. Medium. https://medium.com/pervade- team/were-awake-but-we-re-not-at-the-wheel-7f0a7193e9d5 Smith-Doerr, L., & Croissant, J. (2011). A feminist approach to university-industry relations: Integrating theories of gender, knowledge, and capital. Journal of Women and Minorities in Science and Engineering, 17(3), 251–269. Stanley, J. A. (2019, May 22). UCCS secretly photographed students to advance facial recognition technology. Colorado Springs Independent. https://www.csindy.com/coloradosprings/uccs-secretly-photographed-students-to- advance-facial-recognition-technology/Content?oid=19664437 Wilder, C. S. (2013). Ebony and ivy: Race, slavery, and the troubled history of America’s universities. Bloomsbury Press. Author Bio Dr. Lindsay Weinberg is a clinical assistant professor in the Honors College at Purdue University. Her research and teaching are at the intersection of science and technology studies, media studies, and feminist studies, with an emphasis on the social and ethical impacts of digital media technology. She received her PhD from the History of Consciousness Department at the University of California, Santa Cruz. https://theintercept.com/2019/12/20/mit-ethical-ai-artificial-intelligence/?comments=1 https://www.dukechronicle.com/article/2019/06/duke-university-facial-recognition-data-set-study-surveillance-video-students-china-uyghur https://www.dukechronicle.com/article/2019/06/duke-university-facial-recognition-data-set-study-surveillance-video-students-china-uyghur https://medium.com/pervade-team/were-awake-but-we-re-not-at-the-wheel-7f0a7193e9d5 https://medium.com/pervade-team/were-awake-but-we-re-not-at-the-wheel-7f0a7193e9d5 https://www.csindy.com/coloradosprings/uccs-secretly-photographed-students-to-advance-facial-recognition-technology/Content?oid=19664437 https://www.csindy.com/coloradosprings/uccs-secretly-photographed-students-to-advance-facial-recognition-technology/Content?oid=19664437 Feminist Research Ethics and Student Privacy in the Age of AI Abstract Notes References Author Bio