Impacting Education: Journal on Transforming Professional Practice New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by Pitt Open Library Publishing. New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol.9 No.3 (2024) DOI 10.5195/ie.2024.446 30 Qualitative Data in the Driving Seat: Applying Qualitative Methods in High School Siobhan Reilley University of California, Davis sireilley@ucdavis.edu ABSTRACT The purpose of this essay is to discuss the impact of the EdD experience on one teacher’s understanding of data and research. From a first-person narrative, the author shares how learning to collect and analyze qualitative data has the potential to change the way teachers can engage with “data-driven decision making” in a high school setting. By exploring one scholar practitioner’s personal evolution demonstrates the potential of qualitative research as a tool for both teachers and site administrators tackling stubborn challenges on our campuses. Includes examples of qualitative data sets collected on site and how data informed and inspired further work. KEYWORDS EdD, teacher as researcher, teacher-leader, qualitative research methods In .39 seconds, you can get 277,000 results for the Google search “data driven instruction.” They include TED talks on YouTube, symposium papers from universities across the country, and listicle infographics from Pinterest. Likely every reader of these very words has participated in, if not led, numerous meeting that starts with a look at The Data. Self-describing as a “data nerd” has become both ubiquitous and socially accepted as a humble brag among ambitious educators. Data is (or are, if you’re a true connoisseur) sexy in that it is highly sought after yet difficult to articulate just what makes it so irresistible. Data holds the promise of solving all that ails our schools and teaching. Data holds the answers if only we take the time to find them. In this era of “data-driven instruction” the data doing the driving has been almost exclusively quantitative. When it is time to “take a deep dive into the data,” here come the test scores and grade distribution charts and discipline statistics. It is generally from the year prior and shows aggregate rather than individual performance. Schools are measured in numbers and percentages and the job is to see what those numbers can tell teachers and administrators about how to change those numbers. These numbers should drive the decisions of the adults on the campus for the benefit of students. The proof that it works is that good things like tests scores and percentages of students passing go up and disruptions go down. All measured in numbers. Numbers, while straightforward and empirical, cannot tell the whole story. Many a deep dive into data has been derailed by teachers deriding the numbers for lacking context. Rather than pointing towards answers, the statistics raise more questions. Far from driving instructional shifts or changes in school policy, the ride comes to a standstill. Often teachers want to discuss conditions or external forces that give the numbers meaning before they analyze and those asking them to analyze believe the numbers should speak for themselves. Thus emerges a common and long-standing research conflict between “number-crunchers” and “storytellers” (Smith ,1983). In my experience as a teacher, the reliance on numerical and statistical information contributes to teachers’ resistance and lack of trust in leaders who emphasize “data” to the exclusion of other factors. Existing literature reviews suggest that while quantitative data is prioritized and valued by schools, teachers generally rely on their own experiences to shape the ways in which they interpret and apply that data (Ho, 2022). Qualitative data is simply missing and possible sources such as empathy interviews, student surveys and focus groups have either been overlooked, dismissed as “anecdata” or considered unreliable. These kind of data, when gathered correctly, can provide the context lacking in assessment scores in a systematized, meaningful way. My objective for writing this essay is to share how learning to collect and analyze qualitative data has changed the way I engage with “data-driven decision making” as a high school teacher. The purpose of exploring my personal evolution is to demonstrate the potential of qualitative research as a tool for both teachers and site administrators tackling stubborn challenges on our campuses. Almost immediately upon beginning my work towards an EdD, the way I think about data changed profoundly. Some of the first assigned readings explored the role of lived experiences in any kind of equity focused work (Martinez, 2016; Torres 1998). The concept that discovering how community members perceive their own educational experiences as a necessary part of research was such a departure from the type of data analysis I had engaged in as a teacher. Additionally, I learned about the importance of interrogating my own experiences and came to understand the meaning of positionality. That was the beginning of me unlearning and relearning what it means to be “data-driven.” With that in mind, not only did my understanding grow of https://library.pitt.edu/e-journals http://creativecommons.org/licenses/by-nc-nd/3.0/us/ http://cpedinitiative.org/ https://orcid.org/0009-0000-3009-4624 Qualitative Methods in High School Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 9 No. 3 (2024) DOI 10.5195/ie.2024.446 31 positionality as a concept, I began (and continue) to redefine my own roles within my own school and as a doctoral researcher. I was a well-informed teacher who sought out research-based sources about education, public schools and teaching before I began the CANDEL program. As a well-educated white woman, I believed in the power of education/school to improve the lives of individuals. I also believed that traditional education, and traditionally educated people, have the power to change society for the better. My teaching philosophy has always been predicated on the belief that every student can be successful academically. I worked to remove barriers to access for students who were historically marginalized and locked out of the benefits of education. However, as CANDEL pushed me to examine the ways public schools and teachers (including me) define success continues to center whiteness and uphold the inequities of the status quo. I explored the ways that I played my part as one of the gatekeepers who maintain the system, but I also began to look for new ways to resist, dismantle and ultimately rebuild those systems. I recognized that my EdD journey, particularly research, was a necessary part of becoming a true leader for equity. As a doctorate student my reading diet expanded dramatically, and I learned to read academic research studies. This is where I started to understand how qualitative research methods can produce data that compliment and exceed purely numerical findings. The findings of studies that used interviews and focus groups and observations gave me more to consider. Years of looking at quantitative data exclusively had always left me frustrated, particularly when a single set of numerical outcomes (such as test scores) were used to compare widely different students and schools. Research findings based on qualitative methods finally helped to fill in much of the context that is missing from statistical outcomes alone. In an almost literal sense, reading qualitative studies give voice to the numbers. As I became a consumer of qualitative research, it made me ask better questions about what kind of data we use to inform our decisions at school and opened my eyes to everything we are missing by ignoring qualitative methods and data sources. Looking at statistics like test scores can lay bare disparity in outcomes and remove any doubt of inequity. But being repeatedly presented with quantitative data that reinforces these outcomes threatens to normalize them. I think many teachers, even those who consider themselves committed to equity, stop interrogating why all these numbers show the same thing. The numbers themselves become the outcome, rather than the students or their experiences. Many times, I have felt like data-driven discussions with colleagues are a loop and perhaps adding in qualitative data could give us a way to interrupt that loop and move forwards towards closing achievement and opportunity gaps. Like so many things in education, this is easier said than done. If the changes we seek to make are to be found in the data, changing the data we use sounds almost dishonest. Critics may accuse by saying something like, “we are tired of looking at these poor outcomes so let’s just stop and look at something that makes us feel better”. If “the numbers don’t lie” has been one of the mantras of data-driven work, suggesting “the numbers don’t tell the whole story” can sound like a dodge. Wise and well-intended educational leaders can doubt the reliability and validity of qualitative data, particularly when the methods are not well-constructed or consistently applied. It is difficult to ensure that perspectives gathered from interviews or focus groups are not over generalized. However, the more I read and the more I contemplated how to conduct my own research for my dissertation, I wondered what it would take to bring qualitative research to professional discussions about student outcomes at the high school level. Inequity among student groups within my own campus has felt so intractable for so long, different data could help us find different solutions. Before going any further in my own path to becoming a novice qualitative research-practitioner, the broader context of what else was happening in the world was needed. This was 2020 and 2021. COVID hit like an earthquake and rendered teaching and learning and school unrecognizable in the immediate aftermath. Discussions about access to learning were front of mind as teachers and school leaders had no choice but to address inequality across our student population. The very purpose of school came into question and as educators we had to confront the conflicting ways diverse families felt about how we should do our jobs. At the same time, the George Floyd murder and subsequent demands for a radical reconsideration of race and power forced educators to further grapple with personal and systemic shortcomings. Teachers, students, parents and school leaders struggled to find footing in this tumultuous time and received little comfort from political leaders or institutions. The 2020 election only furthered a sense of chaos and instability. There were loud calls for “reimagining education” as parents, teachers, students and leaders all proclaimed that the traditional ways of doing things simply were not working. Here in 2024, the more things change, the more things stay the same. Systematically, very little of the alleged reimagining has turned into practice. Impact from disruptions in schooling from COVID are evident in students and teachers. From declining student performance to record numbers of educator vacancies, schools have not recovered from the traumas of the last four years. Nearly every story in the non-academic press or media includes interviews with teachers or families explaining the challenges they still face. Qualitative data is essential to suggesting causes for the very problems that need immediate policy action. And yet, despite all the soul searching and existential questioning, it is generally business as usual in terms of the metrics and practices we continue to rely on for data. No one is denying that students and schools have changed and yet there is no longer an active push to fundamentally change the way we do things, from my perspective and experience. The quantitative data continues to show us the failings of our current actions but has yet to suggest new solutions. My own professional path, however, has altered somewhat. I am no longer in the classroom as a teacher but rather in a teacher- leader position as a training specialist focused on 9th grade success. It is in this position that I have continued to reexamine what data means in terms of informing decisions and practices to increase equitable access and achievement in my school. One thing that I have noticed is missing from almost all district or site data are the voices behind the statistics. During COVID and distance learning, education leaders at all levels encouraged prioritizing social- emotional support for students and providing opportunities for them to share their experiences and needs. Now that we are supposedly back to normal, it seems equally valuable to keep listening to what students say. Student voice is, essentially, an overlooked source of qualitative data that will flesh out a richer narrative behind and around traditional quantitative data such as test scores, passing rates and statewide assessments. In many cases, teacher experiences are often left out as well. Teachers are the ones in the classrooms every day yet frequently told what they should be doing differently rather than asked about what they have noticed has Reilley Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 9 No. 3 (2024) DOI 10.5195/ie.2024.446 32 changed with students. These are the two largest groups most directly involved in the numbers we look at in quantitative data and should not be overlooked as rich sources of insight. One reason why this kind of data is missing is that there are unique challenges in collecting and analyzing it. Teachers may draw meaning from informal qualitative data that is likely gathered in unsystematic ways (Ho, 2019), and they may not all have access to the same information. This means all stakeholders are not acting on the same information. My CANDEL coursework and my own research study gave me the skills to conduct the kind of qualitative work that would enhance discussion on student data. In my current position, I have the opportunity to conduct that kind of work and, thanks to CANDEL, the confidence to advocate for greater use of these data points. While I am not an administrator, I am able to be a leader for equity because I have knowledge and experience in how to gather valid and reliable qualitative data that contributes to the collective body of data that informs our efforts at improvement. Utilizing what I learned in CANDEL while becoming a “scholar- practitioner,” I am currently collecting, analyzing and sharing this type of data with colleagues and administration to address pressing issues on campus. As the coordinator of an effort to support 9th grade success, I have prioritized including student voice in our team meetings as we collaborate around instruction. We conducted a survey at the beginning of the year to gather information about how 9th graders were feeling about school and their participation in fall activities. The final survey question was open-ended and asked them to complete the statement, “One thing I wish all teachers knew about students is…” We received 272 responses. Using the same methods I employed for my dissertation, I coded the responses using key words and phrases and then created themes to group similar responses. From that, I was able to summarize some of the repeating ideas that students shared about what they wanted teachers to know. Within our 9th grade team, I shared not only my findings but my methods and provided time for teachers to do their own analysis. The eventual discussion we had about instruction and classroom practices was richer and more productive as a result. Collaboratively, we came up with guidelines and agreements about regularly implementing integrated social-emotional learning activities in response to the needs students expressed. At meetings throughout the rest of the school year, we have committed time to sharing examples of how social-emotional skills can be integrated into content instruction. One of our teachers modeled a web-based app to provide feedback on student discussion and other teachers have adopted it as a result. Without the data, our conversation would not have been grounded in authentic student needs, as expressed by the students themselves. Because of the careful attention to survey design and strong methods of analysis, what was once considered anecdotal became data. 9th grade teachers within the team were not only more engaged in practicing social-emotional learning skills, but we have developed trust in the validity and reliability of the outcomes. We recently launched a mid-year survey that repeated some of the same questions regarding students’ feelings about school in order to look for change over time. We added multiple choice questions about students’ understanding of graduation requirements to identify the need for reteaching. This time, there were two open- ended questions. First, we asked, what has been the most successful in helping you adjust to high school? Rather than repeating the same prompt as the first survey, we asked, “One thing I wish I had known about high school before I started is…” We also provided an optional space if there is anything else about their first semester that they’d like adults to know. We will follow a similar process of analysis as the first survey to make decisions about second semester interventions as well as planning for next year’s 9th graders. I argue that our efforts will be more effective and responsive as a result of the student input we solicitated and analyzed. Another very specific challenge we have with 9th graders is math. We have high failure rates in Math 1 and have conducted focus-group empathy interviews to gather student perspectives about their needs and the course. Again, applying methods I learned and practiced about how to select participants, we focused on students who passed all their classes the first quarter except for Math 1. I invited these students to talk about what additional supports they needed and what instructional practices were and were not working for them. After four rounds of focus group empathy interviews, I shared both direct quotes from students as well themes with both the math department and administration to shape interventions for struggling students and professional learning for teachers. This is the first time that student perspectives have been systematically collected and used in conjunction with assessment data and grades. The number-crunchers and storytellers are able to cooperate to address an enduring problem. Old ways of thinking and teaching are hard to change. We are still working to shift some mindsets about student needs, and continued collection of student perspectives is needed. While we do not have interventions in place yet, this additional information may suggest new ways of constructing those solutions. It is not always an easy sell, despite the support of administration and many teacher colleagues. We, as teachers, have grown accustomed to believing that quantitative data is objective and somehow qualitative data is just opinions. However, by practicing strong research methods and constructing reliable tools for gathering data, on my campus we are learning new things about students and teachers. It is not yet clear how to operationalize much of the information gleaned. The greater use of qualitative data requires some shifts in school cultures and mindsets. Listening to student voices and rewriting narratives around how we teach and do school overall represents a major change that isn’t universally welcomed. Ultimately, it forces us to be much more clear about how we define inclusion and equity. These conversations can be uncomfortable and often raise more questions than they answer, yet they are essential for making real change. I agree with scholar Duncan-Andrade (2007) when he wrote, “the measurement of an equitable education would require significantly greater attention to qualitative assessment of schools and classrooms to determine the specific needs of the community and how those are being met, or not.” (p. 618). Engaging in this kind of work is a direct result of my EdD and it illustrates the value in the scholar-practitioner model. Without my experience in CANDEL, I would not have these skills in my repertoire and would not have the knowledge nor confidence to suggest new ways of thinking about data. If one of the ultimate goals of education is empowerment and agency, earning a doctorate in educational leadership has done both for me. It is now my duty and honor to try to pay that forward. REFERENCES Duncan-Andrade, J. (2007). Gangstas, wankstas, and ridas: Defining, developing, and supporting effective teachers in urban schools. International Journal of Qualitative Studies in Education, 20(6), 617–638. Qualitative Methods in High School Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 9 No. 3 (2024) DOI 10.5195/ie.2024.446 33 Ho, J. E. (2022). What counts? The critical role of qualitative data in teachers’ decision making. Evaluation and Program Planning. 91. 1–11. Martinez, D. C. (2016) “This ain’t the projects”: A researcher’s reflections on the local appropriateness of our research tools. Anthropology & Education Quarterly 47 (1), 59–77. Smith, J. K. (1983). Quantitative versus qualitative research: an attempt to clarify the issue. Educational Researcher, 12(3), 6–13. Torres, C. A. (1997). Education, power, and personal biography. Routledge.