Microsoft Word - Vasudeva VF.docx This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol. 2 (2017) DOI 10.5195/ie.2017.47 Articles  in  this  journal  are  licensed  under  a  Creative   Commons  Attribution  4.0  International  License.   This  journal  is  published  by  the  University  Library  System  of  the   University  of  Pittsburgh  as  part  of  its  D-­‐Scribe  Digital  Publishing   Program  and  is  cosponsored  by  the  University  of  Pittsburgh  Press.   1 Adapted from the 2017 CPED Convening Presentation Reflections and New Directions on CPED’s 10th Anniversary Ash Vasudeva Carnegie Foundation for the Advancement of Teaching Vasudeva@carnegiefoundation.org Inspired by the theme for the Summer 2017 CPED convening – reclaiming to innovating – I would like to offer some reflections on the organization’s work to date and suggest possible new directions for the future. This year marks the 10th year anniversary of the organiza- tion. While it would be presumptuous for me to talk about its rich history and accomplishment – luminaries such as Lee Shulman and David Imig are far more capable than me in this regard –I do want to reflect on the profound changes that have taken place over the last ten years – and ponder with you the opportunities that await this group as you consider the decade ahead. After all, a lot does happen in 10 years. For example, ten years ago this November, my wife Pamela and I welcomed our son Keshav into the world. We had no idea what life would be like with an infant – after all it’s the utter cluelessness of parents-to-be which perpetuates the human race – and we had even less of an idea of who he was going to become over time. Fast forward 10 years and we have a much better sense of Keshav’s temperament, his personality, his strengths and his challenges. For now, his biggest challenge is his little brother, Ishaan – who transforms from best friend to fiercest rival seemingly 10-20 times per day. Everyday. But more importantly, as parents, Pam and I feel more prepared to survive – and even thrive – during the next 10 years because of what we’ve learned during the first 10. And I have no doubt that CPED is in a similar and probably slightly better place. After all, unlike our family, the CPED family is still growing – I’d like give a big shout-out to the 22 new partners who joined this past year – testament to the ambitious and vibrant work of your network. When Pam and I got the itch to grow the family last year, we thought about it long and hard - and settled for a dog. Let the scoreboard read: CPED 110 – Vasudeva family 4. Five if you include Waffles. Of course, scoreboards tell you only part of the story. Watching our family grow over the last decade raises a question that I believe is a relevant for CPED organizationally as it is for me personally. The question that I want to pose to you all today is: What is the family story you want to tell 10 years from now? When you look back at what CPED has accomplished over the past decade – what is it you want to be known for in the coming decade? What obstacles will you have surmounted? Which challenges will you make progress on? What legacy will you leave for your successors, and inevitably, what dilemmas will you leave behind? Just what is the story you want to tell? The story of CPED’s next decade – the one that begins this summer and that you and your teams will be writing about through the work you do and the actions you take – does not begin with a blank slate. Rather, it builds on a verse crafted during CPED’s first 10 years. Jill Perry and David Imig (2016), writing in the CPED jour- nal, Impacting Education, reveal a critical theme over this time: While a plethora of programs have come in and out of schools of educa- tion, many without leaving a trace, CPED has successfully enabled positive change in an infamously resistant sector. What enables CPED to impact the designs of member’s educa- tion doctorate programs, sustain itself over time, and continue to attract new Institutions of Higher Education into the fold? Perry and Imig (2016) cite a federally-sponsored study to suggest at least three reasons: • First, CPED itself became a source of interest among other programs – You can imagine Deans of non- CPED institutions huddling together like rival cliques of middle schoolers, asking themselves “What exactly is CPED up to and are we missing out?” • Second, CPED members not only talked about change during their meetings, they also facilitated and supported change back on their campuses; and • Third, the network – the relations, provocations and inspirations that connect your teams together -- proved to be important to the work of individual member campuses. And yet, while CPED’s future story builds on this strong founda- tion it is also bedeviled by equally vexing challenges. Perry and Imig (2016) remind us of the critiques of Graduate Schools of Education raised by Arthur Levine in 2007 and earlier by Harry Judge in 1982 that continue to resonate and reverberate and in policy discourse inside and outside of higher education. Of the two, Levine is more blunt: “The EdD should be eliminated”, he writes, “as it is “unnecessary for any job in school administration and cre- ates a meaningless and burdensome obstacle to people who want to enter senior levels of school leadership” (p. 68). He adds this opti- mistic note: [In education] “Credentials have come to overshadow competence” (Levine, 2005, p. 68). Compared to Levine, Judge (who hails from England) offers a critique couched in classic British reserve. He notes that U.S. Grad- uate Schools of Education emulate the scholarly activities -- and the scholarship -- of the social sciences and that they appear to abhor having anything to do with the practical needs of schools and school- ing (Judge, 1982). Now, it’s important to acknowledge that it is precisely these kinds of critiques that inspired the formation of CPED and that this organization is particularly well-equipped to address them. But it is also important to recognize that broad skepticism about the use, utility and value of Graduate Schools of Education are alive and well Vasudeva Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.47 2 today, and if anything, the critiques of them are sharper and more widely held than when Judge and Levine published their commen- taries. Which leads me to a question that I hope we will ask ourselves today, which is: Are CPED’s existing strategies, approaches and structures sufficient to achieve the goals you have for the organiza- tion and your individual campuses over the next decade? Or put differently: Is what GOT you here today enough to GET you where we want to go tomorrow? To help answer that question, I’d like to put aside Levine’s critique for a moment – which to my 2017-ear sounds more like an early-morning tweet from a dyspeptic ex-business ex- ecutive – “EdD Programs. Shut ‘em down…SAD.” Instead, I’d like focus on the two issued raised by Judge (1982): 1. Our tendency to emulate traditional forms of academic research and scholarship; and 2. Our tendency to less-than adequately address the pressing needs of practitioners in schools and school systems – particularly those related to equitable oppor- tunities and outcomes. The two issues are related and speak to longstanding challeng- es of productively integrating research and practice. They also represent entry points into the discussion about how the methods and approaches of improvement science enacted through Networked Improvement Communities (NICs), might build-on and extend CPED’s work over the next decade. Specifically, how might improvement science and NICs be part of re-thinking and re-tooling the preparation of education leaders in ways that are grounded in and consistent with the goals, values and history of CPED? How might these approaches provide new oppor- tunities to advance our field’s problem-solving capacity and make progress toward our equity goals? What I’d like to suggest to you today is that an improvement lens can help sharpen and focus the work of researchers on the practical problems of practitioners, and when done well, advances equity in ways that either group would struggle to accomplish alone. I’ll illustrate this with three dilemmas where the best intentions and contributions of research can go slightly askew when played out in the landscape of schools and school systems. I refer to the three dilemmas as: • The Effect Size Fallacy • High Infidelity • The Individual vs. The Collective THE EFFECT SIZE FALLACY Think of three prefixes that have become part of the education lexicon over the last 15 years; scientifically-based, research-based, and evidence-based. The terms are widely used to describe pro- grams and policies designed and tested to deliver school and systems improvement. And overall, this has been a positive trend – after all who of us doesn’t believe that evidence is a critical compan- ion to intuition – one that keeps us honest in the face of our all-too- human biases, assumptions and tendency to generalize on the basis of personal experience. But, as Anthony Bryk, President of the Carnegie Foundation, mentions in a keynote presentation at the 2017 Carnegie Summit, our attention to treatment effects often focuses on the overall aver- age effect instead of the variability of effects across different contexts. He cites the Reading Recovery literacy program as an example. The average effect size of Reading Recovery from its most recent I3 evaluation is .7 – an impressively large figure for an educa- tional intervention. But this part of the story – the average effect - typically overshadows an equally important part of the story: the variability of effects found across schools (Bryk, 2017). It’s this varia- bility – how and why something works in specific settings and what we can learn from that experience to pursue and promote effective- ness at scale – that tends to get lost in the ways we educate leaders about what it means to be evidence-based. One result of this is a generation of educators who are savvy enough to do some compari- son shopping as they think about programs they may wish to adopt – which one as bigger effect size? – but perhaps not equally well equipped to understand the leadership strategies and instructional supports it takes to produce positive outcomes in their particular contexts. At the Bill and Melinda Gates Foundation, where I spent nearly seven years prior to coming to Carnegie – I witnessed this problem play out at the highest levels. As a member of the Foundation’s Re- search and Data Team, I helped develop and design a series of quasi-experimental studies of literacy and math reforms being sup- ported by the foundation. Over the course of multiple years, the studies produced a raft of findings about contexts, conditions, and variables that impacted high quality implementation. But in the end, what folks inside and outside the foundation tended to fixate on was - you guessed it – the effect size. Tongue-in-cheek I say that I was lucky that the effect sizes were positive or else my time there may have been a little bit shorter! The integration of improvement science into the education doc- torate curriculum could help educational leaders deeply examine – and do something about - the variability of performance across set- tings. This would support systems improvement and build new knowledge about how leaders can work to support quality outcomes reliably at scale. This I think could be one important theme for CPED’s story over the next decade. HIGH INFIDELITY The focus on variability of performance is a useful contrast to the conventional way that variation gets addressed in many educa- tional research studies. It’s not that our field doesn’t attend to variation now, it’s that we do it in ways that can be a little self- serving. Let me explain. Another well-established phrase that has accompanied the shift to research-based efforts in education is “fidelity of implementation.” I don’t need to tell this audience about the ubiquity and utility of the term. Fidelity of implementation is used to give practitioners helpful guidance about the steps needed to enact a particular reform or intervention as the designers intended. While the concept of fidelity of implementation is useful – it’s al- so insufficient. Fidelity measures in of themselves do little to help practitioners achieve success in their schools and school systems, and in fact, may provide a false sense of precision around the com- plex pathways and processes that support improved outcomes. One problem with the term is that it can be used to render judgment on practitioners while absolving designers and researchers of shared responsibility for disparate results. Get good outcomes? Credit research-based design. Results not up to snuff? Not our prob- lem – it was simply insufficient fidelity on the part of, guess who – that’s right, the practitioners. Not only does this concept reify the asymmetrical power rela- tions between researchers and practitioners – WE figure it out Adapted from the 2017 CPED Convening Presentation Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.47 3 solutions and THEY keep messing things up – it can also be self- serving fiction. Consider the study of Reading Recovery that I referenced earli- er. It turns out the fidelity of implementation measures used in the study were only loosely coupled to the outcome measures. In many cases Reading Recovery works well and some places it doesn’t but in both instances we’re still struggling to understand why. And it’s not education alone that stumbles in its treatment of implementation. In his book “Good Strategy / Bad Strategy,” UCLA Business School professor, Richard Rumelt (2011) cites implementation – he uses the slightly more morbid term “execution” – as the most common lament corporate executives have about their organizations. He cites one executive who said: “We have a sophisticated strategy process, but there is a huge problem of execution. We almost always fall short of the goals we set for ourselves” (Rumelt, 2011, p. 6). Rumelt’s droll response to the executive gets right to the point. If your strategy results in failure most of the time – just how good was it to begin with? Here’s where I think improvement science can help. Improve- ment science can help extend the educational research base into the mechanisms and processes that reside within systems. Drawing from the fidelity of implementation concept, improvement science attends to adaptive integration – the way that work, people, and resources are organized, sequenced and operate together. Under- standing when combinations and pathways come together to produce desired outcomes – and just as important, when they don’t – is fundamental knowledge for educational leaders seeking to build high-reliability organizations. When done well, the outcome of this work is more than an im- plementation checklist. It is a way of developing the skills, mindsets, and orientations of people – and the capacity within systems -- to address three fundamental questions about the work they are doing to make schools and districts better. The three improvement ques- tions are: 1. What is the specific problem you’re trying to solve? 2. What’s your theory and how does it reflect an under- standing of the system you’re working in? 3. How do you know when a change is an improve- ment? Building the capacity to address these improvement questions – within our own institutions and more importantly in the field – is a promising path and a potential new direction in solving enduring and endemic problems of practice in education. And if, by working shoul- der to shoulder with practitioners in the systems you serve, you all develop the next generation of education leaders capable of routinely asking and answering these questions, then I think CPED’s next chapter – the one you’re writing over the next ten years – has the potential to be a blockbuster. Thus far, I’ve talked about how improvement science can ad- dress two common dilemmas at the interface of research and practice. Building on our use of effect sizes and fidelity of implemen- tation measures, improvement science orients us to work more closely with practitioners inside of systems to better understand vari- ability and apply this knowledge to adaptively integrate research- based efforts within their own settings. I believe this approach can extend the reach, expand the utility and increase the impact of edu- cation schools, like those in CPED, that are committed to solving persistent problems of practice. The third dilemma that I’d like to focus on today involves the relative contributions of individual actors versus groups of people working together. Networked Improvement Communities, or NICs, are designed to magnify the strengths of improvement science through the power of collective action. In this way, NICs reflect re- cent shifts in scientific discovery and advancement that could benefit and accelerate the work of educational researchers and practition- ers. This third dilemma is: The Individual versus the Collective. Albert Einstein is probably the most iconic figure of science – and arguably the single greatest contributor to our understanding of the universe. Other approaches to discovery engage far more people than either a solitary scholar or multiple scholars in a single laborato- ry. Putting Einstein aside, for he really is an outlier, I ask you: Which approach is more likely to help us solve the enormous equity chal- lenges that confront our schools and our nation? Let’s start by examining how astronomers are tackling the big- gest challenges in their field – like the structure and functioning of black holes. Over the last three years astronomers have detected at least three black-hole mash-ups that have helped confirm Einstein’s theories of the universe while expanding the scientific knowledge base. But unlike scientific discovery in the 20th century, science in the 21st century is a team sport. Figure 1 demonstrates just the first page of authors from the paper describing the astrophysical implications of the black-hole merger. It continues, and continues, and continues, painting a picture of how scientists are collaborating to advance understanding their field. Vasudeva Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.47 4 Figure 1. Sample of authorship on a scientific study Can we organize ourselves to do the same? I believe we can and I believe that Networked Improvement Communities can help us get there. NICs marry the social relations of networks with the scien- tific discipline of improvement science to accelerate not just individual learning but the field’s collective capacity to learn and improve. Most importantly, NICs can be powerful tools for taking on the field’s biggest equity challenges. In 2008, when the Carnegie Foundation was exploring the use of improvement science in education, the national problem that we couldn’t turn away from was the failure rates in developmental math- ematics courses offered in community colleges. Typically, just 5% of students enrolled in developmental mathematics earn a math credit within one year, and 80% of these students don’t succeed after three years. The 400,000 students whose education pathway is cut short by developmental mathematics are more than the entire K-12 en- rollment in at least 18 states. I don’t think it’s an overstatement to say that developmental mathematics is where the aspirations of Ameri- ca’s most diverse students by race, class, age, and ethnicity – often go to die. Carnegie launched the developmental mathematics NIC with over 30 institutions committed to solving a common problem – get- ting more students to succeed in developmental mathematics and credit-bearing math courses. Rather than starting with solutions – a new program, a new curriculum – the process focused on how stu- dents experienced the system, where failure was most acute, and where the biggest drop-offs in enrollment occurred. This approach – developed hand-in-glove with community college faculty and stu- dents – created change packages that were tested and refined through Plan Do Study Act (PDSA) cycles. The most promising changes were retained and others were not. This work focused less on rolling out a single big new reform idea than identifying key levers for improvement and quickly testing how best to move them. The results were impressive. The Mathematics Pathways work – which now includes over 60 community colleges – has improved the rate for which students receive mathematics credit from 5% to over 50%. It has been an equity-driver in one of America’s most important edu- cational vehicles for opportunity and upward mobility. How might Graduate Schools of Education adapt the NIC con- cept to the work of developing leaders for the field? The equity challenges facing K-12 school systems are no less daunting than in community colleges – and as persistent and pervasive. Might there be ways of creating Networked Improvement Communities that work together to solve, or at least make progress toward solving – one or Adapted from the 2017 CPED Convening Presentation Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.47 5 more of these equity challenges? How might this approach help build capacity in the field AND develop a generation of leaders with a fundamental approach and set of skills that helps them solve the inevitable challenges that await them in the field? This direction I believe, is not only consistent with CPED’s ac- complishments over the first decade, it’s consistent with organization’s guiding principles that: • Focus on questions of equity • Bring about solutions to complex problems of practice; and • Develop a professional knowledge base that integrates both practical and research knowledge. Of course, only you can judge whether the opportunity to inte- grate improvement science and Networked Improvement Communities into the work of CPED makes sense as you author the organization’s next chapter. And as you consider the family-story you want to tell over the next decade, I hope you consider whether and how an inter-stellar mash-up of Improvement Science, NICs and EdD programs could help reclaim, proclaim, and exclaim CPED as the field’s cutting-edge innovator in leadership preparation. Ten years from now when we look back on the new chapter of work that you are just now begin- ning to write, I’m confident that CPED’s family story will be one that the entire field will find worth reading – and emulating. Good luck in the work ahead and thank you for your time today. ABOUT ASH VASUDEVA, PHD Ash Vasudeva is vice president of strategic initiatives, where he oversees the Carnegie Foundation’s policy and communications efforts to build the field’s capacity for improvement research and networked improvement communities. Prior to joining Carnegie, Vasudeva was a senior program officer at the Bill & Melinda Gates Foundation, where he focused on supporting school systems to im- plement college and career ready standards and strengthen educa- tor effectiveness systems. Previously, Vasudeva was co-executive director of Stanford University’s School Redesign Network, where he developed the LEADS network (Leadership for Equity and Account- ability in Districts and Schools), which enabled superintendents and their cabinets to collaborate on systems-reforms with faculty from Stanford’s School of Education, School of Business, and School of Design (d. school). Vasudeva taught science at Pasadena High School and entered the field through Teach for America. He received his Bachelor of Science degree from Carnegie Mellon University and his doctorate from the University of California, Los Angeles. REFERENCES Bryk, A. (2017). Redressing Inequities: An aspiration in search of a method. Keynote address at the Carnegie Summit. San Francisco 2017. Retrieved: https://www.carnegiefoundation.org/wp- content/uploads/2017/04/Carnegie_Bryk_Summit_2017_Keynote.pdf Carnegie Foundation (2017). How a Networked Improvement Community improved success rates for struggling college math students. Stanford: Carnegie Foundation for the Advancement of Teaching. Re- trieved:https://www.carnegiefoundation.org/resources/publications/how- a-networked-improvement-community-improved-success-rates-for- struggling-college-math-students/ Judge H. (1982). American graduate schools of education: A view from abroad. Ford Foundation: New York, NY. Levine, A. (2005). Educating school leaders. The Education Schools Project: New York, NY. Perry, J.A., & Imig, D.G. (2016). What do we mean by impact? Impacting Education: Journal on Transforming Education. 1(1). Rumelt, R. (2011). Good strategy/Bad strategy: The difference and why it matters. Random House Publishing: New York, NY.