The Journal of Community Informatics ISSN: 1721-4441 44 Research Article Surfacing Human Service Organizations’ Data Use Practices: Toward a Critical Performance Measurement Framework Alexander Fink, School of Social Work, University of Minnesota, finkx082@umn.edu Ross VeLure Roholt, School of Social Work, University of Minnesota, rossvr@umn.edu The Journal of Community Informatics ISSN: 1721-4441 45 Surfacing Human Service Organizations’ Data Use Practices: Toward a Critical Performance Measurement Framework Abstract Community-level data systems, often called collective impact, increasingly define the landscape of human service data creation. Collective impact strategies develop shared performance measurement metrics across numerous human service organizations (HSOs) in a geographic region to move the needle on specific social problems. Such systems encourage funders to support the development of client tracking and data sharing infrastructure, meaning more HSOs have more information about any given client. However, while many HSOs are using more data than ever, questions remain: how is this data being read, understood, and utilized in HSOs? What differences can we discern in organizational operation and service provision? This study builds on three years of participant observation as program evaluators in youth-serving organizations (a subtype of HSOs) around the world. It also included a national study of youth- serving organizations with a strong focus on data use. Finally, it includes interviews with program staff in youth-serving organizations and focus group data with young people. Situating this data between the literature on performance measurement in HSOs and critical data studies, we surface emerging tensions in the ways youth-serving organizations are creating and using data, drawing to the fore salient questions for those invested in supporting the just use of data and technology for our communities. Keywords: critical data studies, performance measurement, human services, youth work, data use Data in Youth-Serving Organizations Data collection and use within youth-serving organizations and their counterparts in social services has a unique history that has arguably evolved more from political movements than evolutions in technology. This history is seen by those in the field of surveillance studies as overlapping with efforts to surveil and control the poor in significant ways (Gilliom, 2001; Henman & Marston, 2008). Responding to trends in Human Service Organizations (HSOs) more broadly over the last 30 years (influenced by government and funder policies and practices), performance measurement has become one of the primary ways Youth-Serving Organizations (YSOs) understand and communicate their work and impact on social problems (Lynch-Cerullo & Cooney, 2011). Additionally, though many YSOs have long tracked young people to better shape interventions to their needs, rapid improvements in the accessibility of robust databases to track and share data about young people inter-organizationally has led to YSOs creating, sharing, and using more data about young people than ever before. Data and its use are rapidly becoming a ubiquitous part of practice in youth programs, often without thorough consideration of its intended uses and consequences (Fink, 2018; Donovan, 2020). Further, the unintended uses and The Journal of Community Informatics ISSN: 1721-4441 46 consequences resulting from the labor to create and use this data are under-documented and rarely discussed by those pushing for more data use. This study aims to understand an emerging shifts and tensions in the operation of YSOs based on the increasing demand to create (some say collect, but we say create in recognition that no data is neutral), share, and utilize data – to become “data-driven.” It addresses a gap in the existing literature between performance measurement in HSOs and the critical lenses that fields like critical data studies and community informatics bring to the examination of the effects of data practices on young people and the organizations that serve them. Performance measurement in HSOs Performance measurement in Human Service Organizations has a long history in the form of program evaluation, a process used to understand a program’s achievements (outcomes) and its impacts (broader change resulting from these achievements). Over the last 30 years, in part due to a “what gets measured gets done” attitude, driven by the Government Performance and Results Act (1992), the focus of many organizations has been on developing internal data tracking and reporting systems. Organizations are responsible for proving to donors and funders that they are achieving desired outcomes and most non-profits (91% according to one survey) collect outcome data (Benjamin, Voida, & Bopp, 2018). The study of performance measurement strategies as they rapidly grow due to lower barriers to technology access is relatively new, but already frameworks are emerging that detail successes, challenges, and limitations of these strategies (Carnochan et al., 2014). These include challenges defining outcomes (including tensions between funder-mandated measures and service worker ideas about client progress), problems in the design and use of data systems, and issues adapting organizational processes (Carnochan et al., 2014). One of the major (though still understudied) impacts of a drive toward performance measurement is what Bopp et al. (2017) refer to as a cycle of disempowerment, wherein front- line workers, some managerial staff, and clients are left out of decision-making processes around data. The absence of these voices leads to creating/using data without a feedback mechanism about the ways these data are impacting clients/staff interactions or services provided, leading to further decisions using collected data that may further disempower clients and staff. This phenomenon is one of the few in this area that recognize the problematic foundations and impacts of performance measurement strategies in a way that demonstrates awareness of systematic oppression and injustice in data use. Critical perspectives on data use The use of the term “data” in YSOs (at least in youth-serving organizations) has traveled far beyond its dictionary definition: “Individual facts, statistics, or items of information.” (Dictionary.com, 2019). Indeed, spend a few days in nearly any youth-serving organization and you are bound to hear staff talking about “the data” in one form or another (almost always in reference to something quantifiable). However, this is always in reference to the data that is The Journal of Community Informatics ISSN: 1721-4441 47 recorded. The dictionary, researchers, evaluators, and many youth workers are quite aware of the other kinds of data that exist that are not “the data.” As we observe a young person, this includes: body language, energy, language they use, what they say and do not say, do and do not do, who they do these things with (and who they do not), what they carry and wear (and do not). A seasoned youth worker might walk into a building housing a youth program and notice a great deal of data: what is (and isn’t) on the walls, the level and type of noise, who is present (and absent), the ethos and energy of the space, and so forth. These same people also know that what becomes “the data” is less objectively determined than it is a product of organizational processes and decisions. Take attendance at the start and end of a youth program. A great youth program might have better attendance at the end than at the beginning. And the opposite for a terrible program. Take attendance 20 minutes into a youth program and the numbers might be quite different than taking it at the moment the program begins. Discussing school with a young person you know well might yield very different answers than a youth you’ve only just met. “The data,” in other words, are a creation of a specific person, at a specific time and place, with a specific viewpoint, recording in specific categories. This is not neutral - far from it, it is almost entirely shaped by relationship, context, question, and time(ing)! Therefore, we refer to “creating,” rather than the traditional “collecting,” of data. Despite this complexity (and the possibilities it opens), “the data” has come to have a very specific meaning in YSOs. It must be collected to convince funders outcomes are being met. And to be convincing, it must be quantifiable, preferably using a well-tested instrument. With these basic assumptions in place, getting data has become trendy enough that we want it, even if we don’t understand what it is we want or why. Scholars of data call this phenomenon “data fetishism.” As Sharon and Zandbergen write, ‘“data fetishists,” [are] enamored by the authority of numerical data and motivated by a desire to control and optimize the overwhelming complexity and uncertainty of life’ (2017). This authority, and the temptation to reduce our incredibly complex relationships and work with youth into understandable numbers and metrics, is leading the field to move increasingly in the direction of collecting more data about as much as possible. Perhaps the result of all this data is “datafication,” or the conversion of a whole, individual person into a constellation of available data points (Van Dijck, 2014). In other words, the person and the data that represents them becomes synonymous to us, and to treat the data is to treat the person (Van Dijck, 2014). These critical inquiries into data collection and use recognize the always-biased nature of data and the ways that data are weaponized to surveil and control (Gitelman, 2013; Eubanks, 2017). They recognize that this biased nature might be caused by, and cause, discrimination (Gangadharan, Eubanks, & Barocas, 2014). Further, they recognize the ways that presented data may cause us to interact differently with a client, forcing us to see and respond to this client in particular ways. They recognize that “performance measurement” is not only a strategy for accountability, but significantly alters the everyday practice of working with young people, in ways that may be harmful and helpful (Fink, 2018; Fink & Brito, 2020; Gillborn, Warmington, & Demack, 2018). This includes the ways a focus on labels like “at-risk youth” distract from environmental, community-level, and systemic problems, like gentrification, racism, and socioeconomic differences (Cahill, 2006; Yosso, 2005). The Journal of Community Informatics ISSN: 1721-4441 48 This study looks at the field of performance measurement in youth-serving organizations (as a sub-type of HSOs) through a critical lens, attempting to surface the ways that increasing demands for performance measurement are causing shifts in service provision that bring to the fore significant ethical questions about how a community engages with, responds to, and tries to control, the young people who live in it. Methodology Starting in 2014, we began a community based participatory research (CBPR) collaboration with an intermediary organization and five community program sites (Wallerstein & Duran, 2006) in the midwestern United States. In 2016, we negotiated requirements and agreements around data creation, analysis, publication, and ownership over the first few months of the study. Typical practices of research, especially in marginalized communities and when researching youth programs, are well-documented and include: researchers designing studies without community input and then gathering ‘data’ (Cochran et al, 2008), study participants being treated as objects and data sources without considering community needs or potential harm of research process (Walters et al, 2009), and the production of damage-centered studies (Tuck, 2009). We were not interested in these practices and instead followed a research process we call a ‘learning partnership.’ In this model, we seek to collaborate with programs and sites and situate our research in their expressed interests, concerns, and desires. In addition to our commitments to CBPR, we submitted and were approved by the Institutional Review Ethics Board at our university. Using a multiple-method, grounded theory approach, we collected data from a variety of sources in YSOs from 2016-2019, both our partner organizations and those outside of the partnership who also agreed to participate in the study. Our goal was to surface a holistic picture of the practices at play in organizations surrounding data. In the process, we conducted participant observations in three organizations. We worked in this role as collaborators on youth programming, and we both had conversations directly about and observed unfolding practices surrounding data as part of these efforts. We had two major questions in mind: • How is the demand for data re-shaping the practices of youth programs? • How are these changes impacting young people and those who serve them? As participant observers and collaborators, we also contributed to these conversations. We worked alongside organizations as they tried, explicitly and unconsciously, to make decisions about data collection and use. This collaboration informed both the emergent themes and suggestions that follow. The irony of creating data about creating data was not lost on us (or the organizations we worked with) and we suspect these interactions brought more conscious attention both to our process of data gathering and the organization’s use of data than would otherwise be present. The result of this is that the youth and staff we worked with, as well as ourselves, have done more thinking about data use than we otherwise might have. The Journal of Community Informatics ISSN: 1721-4441 49 In addition to these participant observations, we conducted interviews with staff at these organizations and several others. These staff ranged from direct service youth workers to city- level program managers to nonprofit executive directors. The organizations that participated ranged in annual budget from $70,000 to multi-millions. Some organizations were based locally, and others were participants in national youth programs. A few of these organizations created almost no data at all, several were extremely data-oriented, with staff dedicated to data creation and analysis and even holding semi-annual organization-wide “data summits.” Our interviews were 60 to 90 minutes and semi-structured (Rubin & Babbie, 2011). They asked participants to reflect openly about the data their organization collected and how they used that data. Interviews used the term “data” loosely and allowed it to conjure up whatever meanings it prompted for participants. If participants got caught on particular understandings, which happened often around quantitative measurement, we would follow-up with questions that asked about other meanings of data. We also asked how staff and youth were involved in the organization’s data processes, their motivations for data use, and about data they did not explicitly collect or collect at all but wished they did. These interviews almost all concluded with interviewees expressing that this conversation was important, was absent in their organization or professional development opportunities, and was a conversation essential to the future of youth services. Finally, we spoke with young people in a focus group setting to understand some of their experiences with data. Though we had many similar conversations during our participant observations, we wanted to sit down with a group of youth and explicitly learn about their experiences around data use in the organizations they took part in. The Midwestern United States carries a particular sociopolitical context that constrains the results of this study. Though some government funding drives nonprofit work, many nonprofits are supported partially or entirely by funds from private philanthropic organizations. Due to multiple funding streams, many nonprofits (including those in our study) report to multiple funders, each with their own unique reporting structure and data gathering needs. Further, unlike colleagues in the European Union, there is a minimalist regulatory regime regarding data collection, even about minors. Laws are in place to protect minors’ data in schools (FERPA) and in health care (HIPAA), but there is very little comprehensive regulation of data collection and use. Data was interpreted using a thematic approach, adapted from Van Manen (1990). Data sources were first read individually, with attention focused on the themes within individual data source. These themes were then compiled together and analyzed again looking for themes from across the entire data set. As themes emerged, we brought this back to both the individual data source and to the entire set of data sources to seek out confirmation and challenges to the emergent themes we developed. The themes that met these tests are included in our findings. Findings Five primary themes emerged from the thematic analysis of our data. These themes are presented in the forms of tensions, as this is the frame through which they were most often named for us in our research. These tensions emerged most frequently between perceptions of The Journal of Community Informatics ISSN: 1721-4441 50 reality and a desired ideal. These tensions were also intersectional, which will be addressed in the Discussion. Data, use-full and use-less Two contrasting understandings of “data” were presented to us: data that was used in meaningful ways, and data that felt like “hording junk.” In this case, data was discussed in its popular use, with numbers stored in a database to be aggregated or to track an individual. While staff and youth alike could see the value of having aggregated numbers (how many attended, how many graduated, etc.) to talk to funders or even to support young people, they also saw ways that this kind of data collection was problematic – both wasteful to collect and not particularly helpful to use. For example, one staff person told us: [Funder] requires us to ask like stupid questions like satisfaction surveys, like: Were you happy with this program? Okay, not at all kind of neutral, or useful, super useful. The most ridiculous survey questions ever. Further (and this was echoed by other participants), this individual went on to say that they already collected this data in other formats and in ways that felt more relevant to improving their program, but due to funding requirements, were required to utilize this survey as well. Many see the present focus on tracking as too much data. For some, the data is not useful and is therefore occupying precious organizational resources. Similar to what [Funder] requires, but also like, you know, specifics about what they did every day and kind of, you know, interesting information. But we don't have a full time evaluation person. So there's no point in asking information and just holding it… The first thing we did when I became the associate director was revise all of our goals and outcomes for the organization, because each program has their own, even though each program had a leadership goal, and enjoying being in a diverse community goal, whatever, right? They all ask them differently, and then evaluated them differently. So then you could talk about the organization as a whole. And then you people would just add more on, they'd be like, Oh, well, I want to know this. And then all sudden, you're asking the young person 100 questions at the end of the year, when you could have asked them 25. This kind of data was not always disparaged by these organizations. At least two used data to make initial decisions about program focus. For example, one organization used these data to ensure they were delivering services to the right youth and that they could try to reach out to young people who might otherwise be ignored by the system: As well as just to try to, like, watch for trends and see if we have, you know, like yeah we got a lot of girls in fifth grade, I wonder what that means, you know, I mean, I don't know, it's, sometimes it means nothing, a lot of times, it means nothing, but sometimes it helps us as we're trying to make sure that we have, you know, fair and equitable access to our programs. And, you know, we want to make sure that, for example, that we're like, The Journal of Community Informatics ISSN: 1721-4441 51 engaging enough, like homeless, highly mobile kids, right, who are hard to enroll and keep in after school, and so will track that kind of stuff. And watch, their enrollment and attendance to try to make sure that we're removing barriers as needed. Staff at another program reiterated this point in their use of data to identify holes in programmatic delivery: You know, so it's showing us, when you get when you're able to just aggregate it like that, it shows you where you have those pockets of kids that are not getting the services they need or the support they need? Nearly everyone in our study saw data in these sorts of nuanced ways – that it had the potential to be very useful if it was engaged with meaningfully. However, they saw a large part of it as useless, necessary to collect to satisfy supervisors but wasteful in the sense that it consumed limited time, distracted from meaningful work with clients, and was often duplicative. Examples of data being useful were much rarer, and in most cases came up mainly when talking about hopes and dreams, rather than reality. Data contextualized in relationship In contrast to “datafication” (Van Dijck, 2014), which turns real people into abstractions, study participants named a kind of data that comes into existence in the context of a relationship with a particular person and is used within that context. The young people we interviewed told us unequivocally that this was the only way they felt comfortable with and respected by data collection about them - when it existed in the context of a relationship they trusted. Many staff we spoke with agreed, for example: Like you need, you need to have relationships, to make sure that things are going well, like computers are great. Technology is great, but it has a beginning and an end point of its usefulness. And it can't replace human beings. And in developing relationships and understanding that things are going well. Because a young person, I mean, like from a very simple standpoint, a young person could just lie. And the supervisor could just lie on there, right? Because we know that young people try to please, right, so they answer more favorably than they may be their real expectation or their real experience. Several organizations collected data almost exclusively within the context of relationships, and this data typically remained in these relationships or a connected network of relationships. One organization collected attendance data only to provide to funders. Otherwise, they relied on extensive conversations with staff to understand whether the program was accomplishing its desired outcomes successfully. Another (relatively small) organization accepted only private donations with no strings attached and therefore did not require any data at all to provide to stakeholders. In the case of both these organizations, however, staff developed strong and ongoing relationships with youth participants. Staff usually had many ways to contact youth and went to tremendous efforts to help young people through difficult situations. Though both could collect significant data on young people, and likely this data could be used to develop new The Journal of Community Informatics ISSN: 1721-4441 52 revenue streams and expand the program, they were unwilling to compromise the program’s ethos to collect this kind of data. Data use that involved young people While using data to track outcomes was common practice, involving young people in data collection and use was rarer. However, many organizations saw benefit to involving youth in these practices and some were involving youth in significant ways. The most basic way young people were involved in data use were in conversations with program staff. This often sounded something like, Staff: Your math scores are relatively low given your other grades, so we need to switch you into math tutoring. Young person: I get that, but I am only keeping my grades up in reading because I’m working really hard in that class and I can’t afford to miss it. Staff: Okay, we’ll figure out another option. Though this appears quite basic, even this interaction only occurred in the context of a program or relationship with a staff member who appeared to the young person to care and be open to feedback. Otherwise, this type of decision often went unchallenged and without the input of youth. Many staff we spoke with involved young people more significantly in the organization’s data use. This most frequently occurred in the practice of data analysis - whether as part of a youth leadership council or advisory board, as youth program evaluators, as program participants, or as part of a data summit, young people were asked to evaluate data the organization had collected and offer their opinions about it. For example, as one participant stated, “... the young people who are helping us then evaluate data would like then look at and go, ‘Oh, how come … we don't serve any Latina girls?’” Much rarer was the invitation to participate in the entire process of data use - from choosing organizational outcomes to measurement instruments or other data collection and use strategies. Most staff we spoke with had not considered this possibility. However, a few organizations worked to involve young people in the management of the organization, either through a youth advisory/leadership board, the board of directors, or on the organization’s staff. In these cases, young people participated in shaping the program, deciding on the program’s goals and desired outcomes, and communicating success (or failure) toward those outcomes to funders. Young peoples’ involvement worked to increase reflection on what data was collected and how it was used. The Journal of Community Informatics ISSN: 1721-4441 53 Inefficiencies in data collection and tracking A consistent theme across many organizations was present inefficiencies in data collection and use. Staff at all organizations indicated that they are now required to collect more data than ever before - by funders, administrators, and other stakeholders. They also indicated that this occurred in ad hoc ways. Most organizations maintained more than one database and more than one measurement instrument. In fact, some organizations had several databases, all recording the same information. Several had staff whose role in part or full is simply to copy data from one database to another (clearly a use of staffing that did little to improve the work or increase program outcomes). Others administer multiple surveys to participants to meet requirements. This provides a constant source of headaches for staff and to most, this consistent requirement for data that had no use did not look likely to be resolved in the near future. Yeah, so there's a whole ‘nother database, I kind of forgot about that. We don't, that the [Funder] runs, called [Database Name]. Hopefully, we like to use the one that we have and has more data in it, we have to re enter young people into the [Funder’s] database. Okay, so we have a clerical trainee. That's 30% of her job is doing all the data entry into the [Funder’s] database. For several, these additional databases were additionally frustrating because of the ways they duplicated, seemed to provide so little value, and its use seemed to be of minimal value even to the funder: It doesn't come back to us like we can look things up in the [Funder’s], the [Funder’s] database, the [Database Name], whatnot, and we could look stuff up if we wanted to. But we don't cuz we already have it? I don't know exactly what they do with it. For some, the inefficiencies of these databases (and the stakeholders that required their use) felt like they compromised their ability to execute best practices and/or to have Data in Relationships. As one participant told us: And then I think some of the most frustrating data that we have to use is for government contracts, so for instance, I oversee [a grant], which is federal funding that goes from the feds to [...] the state to the [city], to us, so guess what, there's three layers of bureaucracy to deal with. And obviously, sometimes even the different layers will get into conflict around how to translate the policy because the policy isn't written for practitioners to actually do anything with. So then we'll be like, no, we know this is the best practice, we're going to do this. And then the city will be like we think it may be and then they'll have the state and the state said "no", and like, because of this, or this or this, and you're just like, "Oh my gosh", and then the things that we're wasting so much time tracking, and like: 30 day follow up, 90 day follow up, when like, that's not how you have relationships with youth. The Journal of Community Informatics ISSN: 1721-4441 54 Others also felt that they compromised the quality of youth programs to ensure compliance with data collection systems for stakeholders. This was especially frustrating when the data collection seemed pointless (the stakeholder did not seem to use it or care about it later) or redundant (using multiple surveys to collect the same data because different stakeholders use different systems). Though all participants acknowledged that it was sometimes reasonable for funders to ask for outcomes data, most thought that many current practices impeded everyday work without being otherwise useful. This connected to a concept we came to refer to as data hoarding, in which programs collected and saved as much data as possible, without any plan for its use. Some participants felt this was necessary: it was difficult to tell when data would be needed to justify their work, and it was better to have that data historically than to need to gather it in the future and have no history. Others felt this was a practice that violated both the privacy of participants, required youth and program staff to fill out too many forms and data entry terminals, and created a bunch of “junk” that needed to be maintained and potentially parsed. Data quality and biases Many staff expressed concerns that the kind of data collected, the instruments used to collect data, or the ways data were being used were either of poor quality or contained significant biases. In some cases, this was due to a dearth of measures on particularly complex or tricky outcomes to conceptualize. As one interviewee told us: yeah, but we don't have any we don't have any measures yet on financial literacy, in particular. And that one might be yeah, I think that one's a little bit harder. So I think that'll probably just end up being like, this many young people participated in financial literacy classes. Though measures exist for financial literacy, these measures felt like they failed to capture the efforts of the program - the focus was wrong. In other cases, though measurement instruments had gone through significant testing and analysis, staff and young people experienced them as biased and even racist. In these cases, staff were forced to decide whether and how to implement these problematic tools given funding requirements and other stakeholder demands. As one staff told us about a commonly used measurement instrument for youth outcomes: Well and our staff just started with the well, it's culturally biased from the beginning, right. So you're asking all of these questions that have nothing to do that are just totally irrelevant for the young person living at [subsidized housing] townhomes, right, in your public assistance housing, who are immigrants or children of immigrants, right? … So let's say they do fill it out. And now you're telling me that they need to work on their resiliency. And so as a youth worker, I now have a list of my 15 kids. Yeah, and what thing they need to work on, and our staff are like, this is terrible. And I think they gave a lot of feedback, The Journal of Community Informatics ISSN: 1721-4441 55 like just on the cultural relevance or irrelevance of the of the stuff that they were asking, but just the idea that a kid could take a quiz, or that you could fill out a form for a young person, and then it tells you what SEL [(Social Emotional Learning)] skill they need to build for it. Right. This was a common experience for those we spoke with. The expectation that these instruments could provide solutions, or even areas of focus, for participating youth, when the instruments themselves seemed irrelevant to the lives of those youth, was very frustrating for staff. Staff were not against these instruments wholesale, but felt they needed to be significantly modified to be applicable to the youth in their programs. As one participant told us, “they need a lot more folks of color and indigenous folks at the table for that to give feedback on how we do those surveys.” Other staff were concerned that these types of instruments are fundamentally flawed in that they seek to transform all children into particular types of children - namely, middle-class white children. In other words, that the focus of the instruments is on making all children culturally and behaviorally similar to white, middle class kids: “We're trying to I mean, because of reuse that framework, we're trying to create little middle class white kids everywhere. Erase all culture.” Though many were required to use instruments and database systems they believed failed to capture important data or were biased, many were aware of and tried to mitigate these flaws. Where possible, they tried to create change in these instruments. If this was not possible, some would administer the surveys but also talk with youth about the problems. Discussion The five themes that emerged from our analysis form a response to our research questions. The demand for data is re-shaping the youth programs in our study. Without much input from clients (young people) or front-line workers, programs are expected to use more data gathering instruments and databases. These instruments, and methods for using them within programs, seem to orient programs toward achieving specific outputs and outcomes as defined by a funder, organizational leadership, or even the company creating the databases in use. All participants indicated that data was collected about them or that they collected data about others, despite significant concerns on the part of young people and front-line workers (and even some program leaders) about: the ethics of collecting this data, the impacts of the data collection process on the programs, the exclusion of positive views of youth, and the consent of participants to collect this data in the ways it is currently collected. To collect this data, they were forced to change the ways they worked, and often were pushed to even change the focus of their work. Finally, the added effort of all this work distracted from program operations, especially the essential work of building relationships with young people. Those that participated spoke to these many problems with the data work happening in their organizations. They also spoke more hopefully about what might be possible. If instruments could be designed to work against bias and oppression, perhaps they would help the organization The Journal of Community Informatics ISSN: 1721-4441 56 achieve its goals of working toward anti-racist or anti-oppressive goals. If data collection strategies were designed with young people and front-line workers, perhaps they’d be more useful in supporting youth and providing data about the real outcomes of the work. If tracking data looked at young people’s strengths, dreams, aspirations, hobbies, and desires, rather than only their “failings”, youth workers and young people would find it not only more useful, but more ethical. The themes are also not cleanly distinguishable – there is some intersection. For example, data that was “use-ful” most often came in the context of relationships. It was also done in ways that were less biased, and more efficient. Indeed, looking across each of these themes, it felt as though participants had given us a framework for a more critical, just, and equitable engagement with data in YSOs. Based in our analysis of their responses, we propose this framework as a series of questions: 1. How can we co-create data collection systems that begin with the needs of young people and front-line workers and primarily aggregate data that views young people from the perspective of their strengths, aspirations, talents, hobbies, and desires? 2. How can data collection instruments be revised or co-created with the involvement of young people, front-line workers and the specific communities that will be surveyed by them? 3. How can funders, who have greater resources and ability to manage technological change, adapt to the data collection systems of organizations, rather than vice versa (as most often happens now, creating duplicative systems and significant data hoarding)? 4. How can data collection processes center first around developing and maintaining real relationships of trust and caring between young people and youth workers? How can it occur only through consent of all involved? 5. How might data collection focus on data about systems, institutions, and environments that affect young people’s lives, rather than young people themselves? Although the themes that emerged do not speak directly to this, there was some fear amongst participants that inter-organizational data sharing was leading, or would soon lead to, the misuse of data for harmful ends. For example, if we share data about attendance in a youth program, will that be used as evidence against a young person who is delinquent from school or has been arrested? We therefore add a final potential question, which we believe echoes some of these concerns: 6. How can data sharing processes ensure that young people do not come to additional harm from systems and institutions that surveil and police them? This actionable framework of questions, derived from the themes in our data, provides an approach that could be used to revise or create data collection systems in YSOs. It still offers the The Journal of Community Informatics ISSN: 1721-4441 57 opportunity to fulfill performance measurement requirements given by funders, governments, and non-profit review systems. It also considers many of the challenges involved in performance measurement, as outlined in the existing literature (Carnochan et al., 2014). For example, it responds to tensions between funder-mandated measures and staff/client views on progress, addresses the divide between quantitative and case-specific data, and involves staff and youth in design and organizational process change (Carnochan et al., 2014). However, it also integrates perspectives from critical (data) studies, which are concerned with the biases and prejudices of data collection, the use of data for surveillance, the ways data processes disempower clients and front-line workers, the lack of consent in most data work (Wilbanks & Friend, 2016) and the deficit focus of most data (Yosso, 2005; Tuck, 2009). It addresses concerns about the datafication of young people by involving them in the process and focusing on data that exists within the context of interpersonal relationships with front-line workers (van Dijck, 2014; Fink & Brito, 2020). It potentially upends traditional models of data collection in YSOs, which involve tracking young people (Eubanks, 2017), replacing it with environmental-level or social analysis that study the worlds young people struggle through (Couldry & Powell, 2014). It echoes a data feminism framework, that addresses the problematic ways data is framed and inverts these frames by re- inserting emotion, embodiment, and pluralistic interpretations of data (D’Ignazio & Klein, 2020). Though such concerns have been raised in the fields of health and education (ex: the social determinants of health, Geronimus et al, 2006), they have not yet altered data collection for the participants in our study. Of course, the questions in this framework all fall into the “easier said than done” category. To have these conversations, funders, leaders, workers, and young people all require some education about the problems of current data collection models (both practical and systemic) and about potential alternatives. They need to make space to have robust conversations about organizational processes and data collection instruments, including space for authentic critique. They need to have the ability to speak back to funders, who may not be open to hearing these kinds of critique and might aim to take funding elsewhere. Some data collection instruments critiqued by study participants have existed for decades and are backed by significant psychometric research, thus making it difficult to argue for replacement. Perhaps most importantly, organizations and funders need to see data work as part and parcel of how their organization functions, not as an “add-on” that must be suffered. The ways an organization does data work is a direct and exact reflection of the way an organization does everything else. Conclusion This study attempts to open conversation, research, and practice around data work in youth- serving organizations, and more broadly, in community informatics. For those in performance management and measurement, the problematics of data work are limited primarily to their efficient implementation and management, along with their productive use. However, for those in critical data studies, this sea change in data collection and use represent significant dangers to young people, youth programs, education, community work, and so on. This field surfaces serious problems and demands that others start to pay attention. Unfortunately, these two fields are The Journal of Community Informatics ISSN: 1721-4441 58 rarely engaged in serious dialogue. This study recognized that this dialogue is already taking place in the field – with front-line workers and young people themselves. They are concerned and are critically engaged, but often go unheard and feel they need to just “make due” with what they are given when it comes to data systems. They also recognize a need for some of the data tracking and performance measurement tools and procedures pressed upon them. Emerging from their concerns and critiques, in conversation with the existing literature from both fields, we believe we have presented an actionable outline that could be used to design and revise data systems and procedures within organizations. Future research should test this framework, ideally by using it (or something like it) to guide the creation of a data system for a YSO or other community organization. Does it lead to the creation of a system that performance measurement scholars might agree fits their needs? Does it also address the legitimate and significant concerns of critical data scholars? Can we create a data system that serves to promote justice and helps an organization communicate about its accomplishments? In truth, we are not sure. However, we believe that engaging this dialectic is of significant importance in the space of community informatics – both through study and practice. References Benjamin, L. M., & Campbell, D. C. (2015). Nonprofit Performance: Accounting for the Agency of Clients. Nonprofit and Voluntary Sector Quarterly, 44(5), 988–1006. https://doi.org/10.1177/0899764014551987 Benjamin, L. M., Voida, A., & Bopp, C. (2018). Policy fields, data systems, and the performance of nonprofit human service organizations. Human Service Organizations: Management, Leadership & Governance, 42(2), 185–204. https://doi.org/10.1080/23303131.2017.1422072 Bopp, C., Benjamin, L. M., & Voida, A. (2019). The Coerciveness of the Primary Key: Infrastructure Problems in Human Services Work. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1–26. https://doi.org/10.1145/3359153 Cahill, C. (2006). At risk"? The fed up honeys re-present the gentrification of the lower east side. Women's Studies Quarterly, 334-363. Carnochan, S., Samples, M., Myers, M., & Austin, M. J. (2014). Performance Measurement Challenges in Nonprofit Human Service Organizations. Nonprofit and Voluntary Sector Quarterly, 43(6), 1014–1032. https://doi.org/10.1177/0899764013508009 Couldry, N., & Powell, A. (2014). Big Data from the bottom up. Big Data & Society, 1(2), 1–5. https://doi.org/10.1177/2053951714539277 Donovan, G. T. (2020). Canaries in the data mine: Understanding the proprietary design of youth environments. http://public.eblib.com/choice/PublicFullRecord.aspx?p=6381050 D’Ignazio, C. & Klein, L. F. (2020). Data feminism. Boston, MA: The MIT Press. https://doi.org/10.1177/0899764014551987 https://doi.org/10.1080/23303131.2017.1422072 https://doi.org/10.1145/3359153 https://doi.org/10.1177/0899764013508009 https://doi.org/10.1177/2053951714539277 http://public.eblib.com/choice/PublicFullRecord.aspx?p=6381050 The Journal of Community Informatics ISSN: 1721-4441 59 Eubanks, V. (2017). Automating inequality: How high-tech tools profile, police, and punish the poor. New York: St. Martin’s Press. Fink, A. (2018). Bigger data, less wisdom: The need for more inclusive collective intelligence in social service provision. AI & Society, 33(1), 61–70. https://doi.org/10.1007/s00146-017- 0719-2 Fink, A., & Brito, M. (2020). Real Big Data: How We Know Who We Know in Youth Work. Child & Youth Services, 1–29. https://doi.org/10.1080/0145935X.2020.1832888 Gangadharan, S. P., Eubanks, V., & Barocas, S. (2014). Data and Discrimination: Collected Essays. Open Technology Institute. Geronimus, A. T., Hicken, M., Keene, D., & Bound, J. (2006). Weathering and age patterns of allostatic load scores among blacks and whites in the United States. American Journal of Public Health, 96(5), 826–833. Gillborn, D., Warmington, P., & Demack, S. (2018). QuantCrit: Education, policy, ‘Big Data’ and principles for a critical race theory of statistics. Race Ethnicity and Education, 21(2), 158– 179. https://doi.org/10.1080/13613324.2017.1377417 Gilliom, J. (2001). Overseers of the poor: Surveillance, resistance, and the limits of privacy. University of Chicago Press. Gitelman, L. (Ed.). (2013). Raw data is an oxymoron. Cambridge, MA: MIT Press. Henman, P., & Marston, G. (2008). The social division of welfare surveillance. Journal of Social Policy, 37(02), 187–205. Kramer, M., Parkhurst, M., & Vaidyanathan, L. (2009). Breakthroughs in shared measurement and social impact. FSG Social Impact Advisors. Lynch-Cerullo, K., & Cooney, K. (2011). Moving from Outputs to Outcomes: A Review of the Evolution of Performance Measurement in the Human Service Nonprofit Sector. Administration in Social Work, 35(4), 364–388. https://doi.org/10.1080/03643107.2011.599305 Sharon, T., & Zandbergen, D. (2017). From data fetishism to quantifying selves: Self-tracking practices and the other values of data. New Media & Society, 19(11), 1695–1709. https://doi.org/10.1177/1461444816636090 Tuck, E. (2009). Suspending Damage: A Letter to Communities. Harvard Educational Review, 79(3), 409–427. van Dijck, J. (2014). Datafication, dataism and dataveillance: Big Data between scientific paradigm and ideology. Surveillance & Society, 12(2), 197–208. Van Manen, M. (2016). Researching lived experience: Human science for an action sensitive pedagogy. Routledge. https://doi.org/10.1007/s00146-017-0719-2 https://doi.org/10.1007/s00146-017-0719-2 https://doi.org/10.1080/0145935X.2020.1832888 https://doi.org/10.1080/13613324.2017.1377417 https://doi.org/10.1080/03643107.2011.599305 https://doi.org/10.1177/1461444816636090 The Journal of Community Informatics ISSN: 1721-4441 60 Wilbanks, J., & Friend, S. H. (2016). First, design for data sharing. Nature Biotechnology, 34(4), 377-9. Yosso, T. J. (2005). Whose culture has capital? A critical race theory discussion of community cultural wealth. Race Ethnicity and Education, 8(1), 69–91. https://doi.org/10.1080/1361332052000341006 https://doi.org/10.1080/1361332052000341006