Microsoft Word - Kochhar-Bryant VF.doc 6 This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol.2 (2017) DOI 10.5195/ie.2017.25 Articles  in  this  journal  are  licensed  under  a  Creative   Commons  Attribution  4.0  International  License.   This  journal  is  published  by  the  University  Library  System  of  the   University  of  Pittsburgh  as  part  of  its  D-­‐Scribe  Digital  Publishing   Program  and  is  cosponsored  by  the  University  of  Pittsburgh  Press.   Symbiotic Space Exploring the Nexus of Rigor, Problems of Practice and Implementation Carol A. Kochhar-Bryant The George Washington University kochhar@gwu.edu ABSTRACT Across the United States, doctoral programs in education are in deep reflection about their purpose, content, and expected outcomes for graduates. Many are in the throes of redesign and testing to better differentiate be- tween two pathways to the doctorate—one for scholars preparing for research and academic roles and one for scholar-practitioners who wish to focus on problems of practice and implementation. This paper explores a ma- jor source of challenge among faculty as they engineer both research-oriented and practitioner-oriented degrees, and that is the interconnections among the constructs of rigor, problems of practice, and implementa- tion. The paper examines the tension between the “gold standard” of random assignment studies and causal inferences, and the more recent frames of inquiry in which research is translated to practice through shared pathways that engage multiple stakeholders. Several questions are explored: Are “rigor” and “relevance” oppos- ing constructs? What is rigor and should there be a new standard for scholars navigating within a symbiotic space – the arena in which they seek to impact highly complex social problems within authentic field-based set- tings? Is rigor subjective, or can it be measured? These questions are examined within the context of implementation science and through the lens of the lived experience of Education faculty who find themselves in a creative crucible as they navigate the redesign process. KEY WORDS: rigor, relevance, symbiotic space, problem of practice, implementation science, redesign INTRODUCTION The so-called dichotomy between research and practice is a common theme in the field of education. Doctoral programs in edu- cation in the U.S. and around the world are in deep reflection about their purpose, content, and expected outcomes for graduates. Many are in active redesign and testing to better differentiate between two pathways to the doctorate—one for scholars preparing for research and academic roles and one for scholar-practitioners who wish to focus on problems of practice and implementation (Reardon & Shakeshaft, 2013; Walker et al., 2009; Zambo et al., 2014; Zusman, 2013). Programs, enrollments, and degrees in new professional- practice doctoral fields have bourgeoned over the past decade, add- ing over 500 programs with more than 10,000 degrees awarded in 2011-12 and 35,000-40,000 students enrolled (6% of all U.S. doctor- al degrees) (Zusman, 2013). Our experience navigating the dual implementation of both a Professional Practice Doctorate and a new PhD program within the environment of traditional EdD degrees presents an opportunity for close examination of the construct of “rigor” within each program structure. It may be instructive to others in similar situations. How faculty communicate, debate, and cooperate to resolve the collateral tensions represents a primary challenge as practitioner-oriented doctoral programs expand. THE CARNEGIE PROJECT AND OUR PARTICIPATION The Carnegie Project on the Education Doctorate (CPED) is an organization dedicated to ensuring high-quality EdD programs. For 10 years it has explored the purpose and nature of the doctorate in education and encouraged alternatives to the traditional dissertation (e.g., the “dissertation in practice”). Doctoral programs concerned about relevance and impact are based on a set of principles aimed at graduating scholars who can integrate theory and practice as well as address critical problems of practice. Six working principles were established by CPED member institutions to ensure excellence in educational doctorate preparation programs. According to the princi- ples, the professional doctorate in education: (1) is framed around questions of equity, ethics, and social justice to bring about solutions to complex problems of practice; (2) prepares leaders who can con- struct and apply knowledge to make a positive difference in the lives of individuals, families, organizations, and communities; (3) provides opportunities for candidates to develop and demonstrate collabora- tion and communication skills to work with diverse communities and to build partnerships; (4) provides field-based opportunities to ana- lyze problems of practice and use multiple frames to develop meaningful solutions; (5) is grounded in and develops a professional Symbiotic Space Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.25 7 knowledge base that integrates both practical and research knowledge and links theory with systemic and systematic inquiry; and (6) emphasizes the generation, transformation, and use of pro- fessional knowledge and practice (Carnegie Project on the Education Doctorate, 2009). Over the past three years, in the Graduate School of Education and Human Development, our faculty participation with the Carnegie Project on the Education Doctorate has stimulated an appraisal of our EdD Programs—the intellectual content, rigor, and relevance to our consumers as they prepare to face the dominant challenges in education today in the U.S. and internationally. Outcomes of Our Participation in the Carnegie Project A faculty learning community was formed in 2013 to identify core features of effective doctoral programs and to work toward pur- poseful transformation as we rethink the foundations of our programs. Such transformation has not meant reinvention for the sake of change, but rather faculty recognized the need to nurture a new generation of courageous transformative scholars who could translate research and impact practice. They realized that new standards of rigor needed to be defined for those working within a symbiotic (mutually beneficial) space between the University and the field—“transformative” scholars who are change agents seeking to impact highly complex social problems within authentic field-based settings. We currently offer two doctoral degrees, the Doctorate in Edu- cation (EdD) and the Doctor of Philosophy (PhD). EdD degrees can be obtained in several subfields, including Educational Leadership, Education Policy, Higher Education Administration, Human and Or- ganizational Learning, Curriculum and Pedagogy, and Special Education. PhD degrees are offered in Education and in Counseling. A research-focused PhD program in Education has recently been approved and a faculty-launch committee formed to implement the program. Features of the PhD program include a modified admis- sion process for applicants, a research course sequence that differs between the EdD and PhD and research apprenticeships that begin in the first semester of the program. Frameworks for a third degree—the Professional Practice Doc- torate (PPD)—are under development in Education Policy, Educational Leadership, Special Education, and Higher Education Administration. These programs are characterized by a shorter yet intense program and field-based apprenticeships in which scholar- practitioners draw on their coursework to define and examine signifi- cant problems of practice. Capstone projects are designed to be of direct benefit in impacting a school system, organization, govern- mental agency, or community organization. Dimensions of Culture Change As faculty navigate the redesign, they have begun to experi- ence a wholesale culture change that includes aspects of faculty work, behavior, values, and attitudes. First, faculty expressed con- cern about whether resources would be shifted away from the EdD degrees (or new PPDs) to the PhD program, as the latter would ad- mit only full-time students who rely on full fellowships. The PhD program was long fought for over a three-year period, and therefore stakes are high for its success. It is important that faculty and the finance director of the department or unit closely observe the pat- terns of change in resource distribution among the programs in the subsequent years, with attention to issues of equity and the sustain- ability of both types of programs. Second, the redesign of academic programs in a school of edu- cation challenges well-established ideology and traditions of many faculty. For example, for faculty who have deep-seated beliefs that the PhD is appropriate only for those aiming for careers in research and academia, advanced research courses and extensive research apprenticeships are viewed as a central part of the program and the program is expected to take 5 years or longer. Other faculty are interested in advancing the development of the professional-practice doctorate designed for applied research, field-based apprenticeships that prepare students for leadership roles, and can be completed in 3-1/2 years. It was vital to the success of the PPD to support those faculty members who were excited by alternative models and willing to work together to create new programs. They came to represent a new “force” within the School and were encouraged by the Dean and academic leadership. Third, balance in teaching, research, and advising loads are a paramount concern in an environment with both PhD and practition- er-based EdDs or PPD. PhD programs tend to have smaller cohorts than the PPD programs. However the PPD is equally as intensive, as faculty load is increased by time spent mentoring students as they create projects of impact, guiding them as they negotiate with partner schools or organizations, and closely supporting their work with stakeholders in the field. A major source of cultural dissonance, however, occurs as faculty works to ensure rigor in the curriculum, the field experiences, and the dissertation within the two programs. THE CONSTRUCTS OF RIGOR AND THE PROBLEM OF PRACTICE Rigor vs. Relevance Relevance for practice and for implementation of research have also become an important criteria for rigor. Regardless of how tech- nically sound research projects are, if they do not address issues or questions of concern to practitioners and policymakers, it is unlikely that the research will be used (Bradford, 2015). However, despite considerable focus on the research-to-practice gap and on identi- fying evidence-based practices as a means to bridge it, there is little evidence that suggests that the gap has been meaningfully narrowed (Cook & Odom, 2013). The implementation of research, therefore, becomes an important focus for understanding the problem. Fixsen et al. (2005) defined implementation broadly as‘‘activities designed to put into practice an activity or program“ (p. 5). In the inaugural issue of Implementation Science, Eccles and Mittman (2006) defined im- plementation science as “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice“ (p. 1). Evidence-based practices were required to be supported by multiple, high-quality, experimental or quasi-experimental studies demonstrating that the practice has a meaningful impact on subjects (e.g., student outcomes) (Odom, 2005). However, implementation and translation of research findings to practice—“wicked problems,” according to Fixsen, Blase, Duda, Naoom, and Van Dyke (2009)—represent the most perplexing as- pects of evidence-based practices. Implementation conditions are always in flux and engage multiple layers of infrastructure of a sys- tem that has been referred to as “contested spaces,” in which key stakeholders can either support or obstruct transformation (Gutiérrez & Vossoughi, 2010; Penuel & Spillane, 2014). Gutiérrez and Penuel (2014), therefore, argue for a reconceptualization of rigor that requi- res sustained, direct, and systematic documentation of what takes place inside programs in order to document how systems being stu- Kochhar-Bryant Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.25 8 died change and adapt interventions in their interactions with each other in relation to their dynamic local contexts. Rigor, Problems of Practice, and Implementation Science The aim of the Educational Sciences Reform Act in 2002 was to call for scientifically based research that would “apply rigorous, sys- tematic, and objective methodology to obtain reliable and valid knowledge relevant to education activities and programs” (Pub. L. No. 107-279, p. 116). However, the field continued to express con- cern about the narrow set of criteria used to define “rigor,” and the establishment of a “scientific culture” that relied primarily on the “gold standard” random assignment studies of program effects as the remedy for the failures of education research to offer credible guid- ance for policy and practice (Gutiérrez & Penuel, 2014). At the time, Erickson and Gutiérrez (2002) argued that rigor in studies that aimed to draw causal inferences about policies, programs, and practices required in-depth qualitative research—an understanding of the Why, How, and Under what conditions programs and policies work, includ- ing the following: • Sustained, direct, and systematic documentation of what happens inside programs • How individuals under study (e.g., students and teachers) change and adapt to interventions in inter- action with each other in relation to their local contexts • A focus on persistent problems of practice examined within the context of their development • Attention to ecological resources and constraints (Gutiérrez & Penuel, 2014) As a result, the Institute for Education Sciences (IES) began re- search programs that focused on problems of practice and for which relevance for practice and policy was an explicit criterion for judging the quality of research proposals (Easton, 2013). Examples of these programs include Continuous Improvement Research in Education and the Researcher-Practitioner Partnership in Education Research. These programs were not intended to promote translation of re- search to practice in a unidirectional manner, but rather through a more reciprocal pathway (Easton, 2013). Criteria for judging the “relevance” of research proposals were required to include the fol- lowing: • Documentation that the problem of focus is perceived by multiple stakeholders to be significant, persistent, and wor- thy of investigation • Evidence provided by researchers that they have engaged in a process to negotiate the focus of their joint work and which documents how participation in the process was structured to include diverse education stakeholders • Specific methods for bringing relevant stakeholders to- gether and deliberating about the problems that can and should be addressed through research and development • Engaging the multiple layers of a system’s infrastructure that have accumulated over time (contested spaces) so that they can support rather than obstruct transformation (Gutiérrez & Vossoughi, 2010; Penuel & Spillane, 2014) These criteria include important aspects such as ‘problems wor- thy of investigation,’ ‘multiple stakeholders,’ ‘negotiating the focus,’ ‘participation in the process,’ ‘deliberation about problems,’ and ‘en- gaging layers of infrastructure. Following Gutiérrez and Vossoughi (2010), rigorous and consequential study of efficacy of educational interventions involved sustained first-hand observation, sharing in the action and cognition of practitioners, studying side-by-side in symbiotic space, and being jointly engaged in work to transform systems. These conditions are more likely to produce sensitive and robust measurement and ecologically valid accounts of institutional change. Educational interventions have been viewed as contested spaces that are filled with tension and resistance from a range of stakeholders. Supporting and engaging more diverse stakeholders in defining the focus of research and development requires that re- searchers rethink the nature of educational interventions. Approaches to interventions can be viewed as dialectical systems that are subject to revisions, disruptions, and contradictions (Gutierrez & Vissoughi, 2010). Therefore, research into the “social life of interventions” demands more open-ended, socially embedded studies that involve mutual engagement, and the identification of contradictions within and across the levels of the system under study. The contested spaces, then, become symbiotic spaces. RECOMMENDATIONS RELATED TO STANDARDS OF RIGOR AND IMPACT IN DOCTORAL PROGRAMS Differences of Opinion About Research Preparation The definition of ‘rigor’ as a key standard for innovative PPD programs actually encompasses a set of transformative developmen- tal skills that reach beyond the aims and capacities of most traditional research oriented programs. These transformative skills enable scholars to harness their creativity to solve crucial and highly complex social problems. Transformative programs create trans- formative scholars who can redraw or expand the boundaries of practice and policy because they have developed the commitment and identity of a change agent, are guided by a powerful vision of the future, and can translate that vision into reality regardless of the environment (Nisan & Pekarsky, 2009). Transformative scholars envision, in the field of education, that which has never been, and they do whatever it takes to make it happen. In short, they are pre- pared for high-impact action in the world. Rigor and the PhD In recent years, academics have debated the value of the EdD and considered plans to improve it, and to raise its stature to that of the PhD (Basu, 2012), or to offer both degrees. University program redesigners tend to distinguish the degrees by candidates’ role aspi- rations, but do not attempt to define ‘rigor’ in relation to the two degrees. For example, in 2012, Harvard University eliminated the EdD which it had offered for 90 years and replaced it with a PhD The Academic Dean, Hiro Yoshikawa, explained that the EdD program was already a research-based degree, and so to eliminate the con- fusion between the two degrees they moved to the PhD for those preparing for research roles. The EdD was replaced by a shorter, practice-based program designed for those preparing for leadership in education. The goal was to equate such a degree with other pro- fessional degrees rather than with the PhD. The Rossier School of Education, University of Southern Cali- fornia, distinguishes the PhD program (a 44-month program) as “research-oriented,” whereas the EdD programs (32-33 months) are directed toward educational “practice” and the application of theory and research. The EdD is considered equal in rigor but different in Symbiotic Space Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.25 9 substance from the PhD (USC, 2017). The PhD student is typically a person who (a) anticipates a faculty career or an area of practice that demands research expertise, (b) is excited by theory and conceptual analysis, and (c) intends to participate in research and has the po- tential for primarily advancing the theory in the field rather than implementing its practice. The EdD student is a person who (a) is planning to work in the field as a practitioner, (b) is particularly inte- rested in developing new practical or technological capabilities, and (c) is interested in research that emphasizes development, evaluati- on, or field-based projects (USC, 2017). Concerns about creating and sustaining rigor represented the greatest challenge for the quality and coexistence of the two pro- grams. A review by the current author of the content and requirements of 25 PhD programs yielded the following criteria by which quality of the programs is judged: intensity of coursework; high level of critical thinking; a common domain of knowledge in the disci- pline that all students should master; advanced knowledge of the research methodologies of the discipline; original research contribu- tions that build on current research available in the field of study; and the stature of the journal or other media for publications. Require- ments in these programs for candidates who complete a doctoral degree, either the PhD or the EdD (in the 20 programs that offered both), should demonstrate the following skills: • Broad and advanced knowledge within the discipline • Successful use of a range of methodologies of the discipline • Independent performance of original or applied re- search • Effective communication, written and oral • Performance as a professional in the discipline The Crucible of Implementation: Rigor and Laboratories of Practice Definitions of rigor and program quality require re-imagination in the context of authentic field-based settings. The previous discussion about the meaning of rigor and relevance leads to other questions that are equally as provocative in the context of program design and expectations for candidate performance. Can a practitioner-oriented program achieve the same level of “rigor” as a traditional research doctoral program or should the construct be redefined? Does “rigor” encompass a special set of skills that are linked with problem solving and field impact? Program “rigor” as defined within a practitioner- oriented program affects matters of program status and value, faculty identity, student identity, student performance expectations and ex- periences, program content, and distribution of resources. CPED held a discussion about this topic in its November 2012 UCEA meeting in Pittsburgh, offering the following observation Because the professional practice doctorate in education pre- pares graduates to engage in a type of work fundamentally different from a research-heavy PhD, CPED asserts that the standards for the two degrees should display strikingly different definitions of excel- lence and quality. Reframing the question of what constitutes rigor, based on a criterion unique to the EdD, rather than a standard bor- rowed from the PhD offers an important opportunity for the development and longevity of a true professional practice degree in education. To that end, perhaps we should be asking: What is the IMPACT of our programs? (Perry, 2012) Other discussants suggested that quality be defined as the de- gree of impact our programs have on the profession we serve. The challenge is, however, to reach consensus on how quality should be defined and how impact should be determined and measured in terms of the leadership performance of our graduates. Several di- mensions of quality and rigor can be defined as core elements of “laboratories of practice” and authentic field-based settings (Shul- man, 2005) in which theory and practice inform and enrich each other and allow students to address complex problems of policy and practice. A Vignette: Research Reciprocity and Impact The following vignette illustrates the process of interweaving professional and personal development through the curriculum, ex- periential activities, development experiences, and dissertation. This vignette reflects the development and dissertation work of the current author’s recent doctoral graduate. The choice. Angela H. had examined several doctoral pro- grams that offered PhDs. However, she was concerned about a “traditional” program, in which she may have to fit into existing re- search orthodoxy and faculty agendas rather than forge her unique path. As a school psychologist, she believed that research should be conducted in collaboration with the “researched” and in real-world settings. She was inspired by the idea of embedding her research in the professional community of which she was a part and creating a context for shared inquiry. When she was interviewed for the pro- gram, she was asked, “How do you want to make a difference in the world?” She had never been asked this question and was intrigued by the idea of having the freedom to envision what had never been and then do whatever it took to make it happen. Faculty knew that Angela was self-motivated, inspired, and committed, and could be guided to translate her vision to practice. The preparation. Angela was well prepared for the dissertation stage as a result of her coursework and related development experi- ences. Her program plan combined courses in special education leadership, systemic change and consultation, qualitative and quanti- tative research tools, legal issues and public policy, preparation for the professorship, and coursework that bridged neuroscience with child development and education. She had also participated in two field-based research internships under the direction of faculty lead- ers. Each student was required to examine a topic of interest related to a specific population of individuals with particular neurodevelop- mental disorders requiring educational interventions. The requirements for the student’s project portfolio included information related to prevalence and incidence data, assessment approach, social profile (e.g., a picture of the social conditions for the popula- tion in terms of access to education, technology, economic well- being and employment, postsecondary education, and independent living), an individual case example, overview of research, implica- tions for educational research, and recommendations for further research. For example, one of the projects engaged her, along with other doctoral students, in collaborative exploration of the neurosci- ence of attachment disorders in adolescents. The team provided a history of reactive attachment disorder (RAD), diagnostic criteria, definitions, and behavioral manifestations and symptomatology. They distinguished among different types of RAD, closely examined the DSM-IV classifications, and critically examined underlying theo- ries of RAD, particularly those of Bowlby (1983), Ainsworth (1978), and Blount-Mathews and Hartenstein (2005). A compelling exemplar case study captured not only the behavioral manifestations, but also the impact on the family and on schools attended. They provided available prevalence and incidence data and explained the limits of research literature on the subject. Definitions of RAD by the Ameri- can Psychiatric Association, World Health Organization, Centers for Kochhar-Bryant Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.25 10 Disease Control, and Administration on Children, Youth and Families were presented. The team presented a review of the social conditions for the population, addressing factors such as self-concept and mental health, success in employment and postsecondary education, inde- pendent living, and community participation. Outcome data on the population of children were compared with that of children without the disorder. A framework was provided for understanding RAD, including impacts of early care-giving and brain development; effects of trauma and abuse on self-regulation, communication, and cogni- tion; and psychobiological relationships. The team discussed emerging efforts to understand the role that early negative experi- ences have on the neural system and brain structures, aspects of early brain development, stress responses, and neurobiology and neurotransmitters. They also addressed interacting neurobiological systems (sympathetic and parasympathetic systems, limbic system), genetic predispositions, and temperament. The project closed with the framing of important research ques- tions, with examples of emerging and promising research on the topic. A summary was provided on what is known about the effects of RAD on cognition and learning, the challenges to educating chil- dren and youth with the disorder, and identified interventions in practice with the population. An important part of this kind of project is the practice of “as- sumption hunting” (Brandenburg, 2008), in which students examine the complexity of a problem as well as the beliefs and assumptions at its root. The grounding assumptions of contemporary interven- tions were critically analyzed in the light of new neuroscience knowledge (e.g., limitations in effectiveness of medications) and recommendations were made. Another important element of the project was the hunting of former theories of change associated with previous interventions into the problem. While the team concluded that the disorder is “entirely preventable,” their course professor queried whether enough is known to date to be certain that attach- ment disruptions would not occur even in healthy caregiver relationships. During this team project, Angela was challenged to think deeply about the kind of impact she wanted to prepare for and create. Mentoring for impact. Angela’s program included mentoring and guided processes in which students clarified their personal and professional identity, examined their motivations for their commit- ment to change, and reflected on the importance of culture and diversity in their decisions and actions. Angela spoke about her con- cern with the rising number of older children with math disabilities and attentional problems, and wanted to design an intervention that she could test. Doctoral programs that design for impact require intensive fac- ulty time devoted to advising the student, guiding the study design, and coaching for the negotiation of the field-based project implemen- tation. Angela viewed the dissertation as an opportunity to design and measure an intervention with a population of children for whom she and school professionals were concerned. She also hoped that a successful intervention could be replicated in the future with a larg- er population in her school district. She wanted to achieve a goal greater than that of individual personal and professional develop- ment—to in fact transform her commitment to relevance and impact in her work. She and her peer cohort had become aware that they had stepped outside the narrow band of advancing their self- interests as scholars and had embraced the transformative scholars’ doctrine: impact the world and work deeply in the community. In summary, Angela was oriented early to a critical interrogation of her beliefs as well as of current practices. She strengthened her ability to express her values and talents and thus to actualize her identity as a skilled professional. She reflected on the personal and professional transformations that contributed to her identity as a change agent and on the value-commitments that guided her ac- tions. Finally, she learned what it meant to construct a theory of change that was grounded in an understanding of the history of pre- vious interventions into the problem of interest. The dissertation: Rigor and relevance. Angela’s dissertation is titled the Effectiveness of Computerized Working Memory Training on Math Achievement and Other Transfer Effects in Children with ADHD and Math Difficulties (Heishman, 2015). As a school psy- chologist in rural Pennsylvania, she was aware that children with learning disabilities and attention deficit hyperactivity disorder (ADHD) struggle daily and are at risk for poor long-term outcomes. She was interested in emerging evidence suggesting that working memory (WM) may improve by adaptive computerized working memory training. She enlisted 23 school-aged children (11 females), from the 5th grade to the 12th grade, who had both math difficulties and ADHD to participate in a quasi-experimental, repeated- measures study in a school to investigate transfer effects of working memory training (Cogmed RM) on math achievement, fluid reason- ing, and memory and learning tasks. As part of a pilot, the Cogmed Progress Indicator (CPI) was used to measure transfer effects on working memory, following directions, and math challenge through- out the training. Cogmed Working Memory Training is a software- based program reported to improve working memory and designed for children and adolescents. It combines cognitive neuroscience with computer game design and professional support for teachers. Standardized instruments were administered at baseline and at 4 weeks and 4 months post-interventions. Teachers and students completed the Conners-3 to assess ADHD and teachers completed the Behavior Rating Inventory of Executive Function (BRIEF) to measure executive functioning. The Cogmed Coach discussed with parents and students the concept of working memory, how it impacts academics and functioning in daily life, and how Cogmed working training works. Statistically significant improvement on the CPI was found on the following-directions tasks, and on indices measuring verbal memory, visual memory, verbal working memory, symbolic working memory, attention/concentration, working memory, general memory, and fluid reasoning 4 weeks post-intervention. Statistically significant differences were also found at the 4-month follow-up period in math fluency, applied problems, and math calculation. Working memory, inhibition, organization, and the Behavior Rating Index scales of the BRIEF were found to be statistically significant at the 4-month post- test, and ADHD showed significant improvement. Angela concluded that the results were very promising, though additional research was needed to address the limitations of the study. Angela knew from the beginning that her research needed to be relevant to those in the research environment. In the initial stage of her research she identified shared interests with the school commu- nity and engaged multiple stakeholders in her research design and its implementation, including her participants and their families. She worked with teachers and school administrators to identify what they needed and expected from her research and learned of their interest in using the results of her study to improve instruction with students and to integrate the practices into the curriculum. She also asked parents and teachers what they believed needed to be explored and how they might contribute to the study. Symbiotic Space Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.25 11 Lessons about relevance and stakeholder perspectives. Acting on the ethic of relevance and reciprocity, Angela considered how the different groups in the study would use the work. She ex- plored cultural issues that might shape the perspectives on the problems of children with ADHD and math difficulties. She asked the parents, teachers, and administrators what they were interested in seeing as a result of their participation, and she provided a thorough report of processes and results. This report resulted in the develop- ment of additional professional development materials for teachers, and recommendations for wider use of the Cogmed instructional strategies. Angela worked continuously to adjust her study to ensure that she was exploring questions that were relevant and important to the community she was studying. In the process, she became an advocate for the student participants and their families. Collaboration and reciprocity meant viewing research as a relational endeavor— asking questions that mattered to the community and engaging them in interpreting the meaning and the results. Lessons about reciprocity in research. The notion of reci- procity regarding the dissertation elicited provocative questions from Angela about who owns a research project. Her advisor reframed her more general questions as follows: if the research is contextual, engages multiple stakeholders, is dependent upon these relation- ships, and the interpretation is context-sensitive, then who ultimately owns the work (Kochhar-Bryant, 2016)? She concluded from the question that it is not a dichotomy between the researcher and the researched, but it is both-and. She also learned that the dissertation in practice that embraces reciprocity leads education researchers to a broader conceptualization of evidence, one that expands the trans- formative potential of collective work (Trainor & Bouchard, 2013). Reflecting on barriers and threats in the process. Angela re- flected on many challenges and barriers in the process of preparing for and negotiating the research project in an authentic setting. Echoing Schein (2004), she learned that for a culture to effectively change within an organization, the staff not only had to learn some- thing new, but also had to “unlearn” something. Angela also realized that for transformative change to occur there needed to be “discon- firming data” that suggested the inadequacy of former strategies. It was important that she gathered and presented that data. Third, Angela learned that the school culture was perpetuated through a process of socialization that involved the transmission of values and information. It was, therefore, important to ensure that the senior staff received continued training to be able to communicate and reinforce these concepts effectively with new staff (Van Maanen & Schein, 1979). Angela came to understand that the perfect reform initiative does not exist and that the focus on reform distracts educa- tional leaders from focusing on the school culture and the elements of that culture that must be understood and nurtured for the reform to succeed (Ross, 2010). Fourth, Angela noted several factors that may have affected the outcomes of the behavioral checklists. One factor in particular that may have affected teachers’ observations of students’ behaviors may have been the excessive demands placed upon them that es- sentially led to a negative school climate. Teachers reported feeling overwhelmed during the school year in which the research study was conducted, particularly with changes in the teacher evaluation sys- tem whereby their students’ performance on standardized assessments could affect their ratings. Additionally, teachers ex- pressed concerns about pending changes taking place in the curriculum due to requirements to meet Common Core standards. Adding to these new demands, teachers were faced with changes in the assessment of their students, and they expressed some concern about students participating in computerized interventions during the resource period in the day instead of receiving additional homework support. These stressors on teachers may have produced negative attitudes affecting the ratings. In other words, bias against the partic- ipants’ involvement in the computerized working-memory training program needed to be considered, particularly as the teachers were well aware of the training program. It is plausible to think they were resentful as they were further frustrated by a system that gave them less autonomy in how they structured their lessons and met the needs of their students. A fifth serious threat to the successful implementation of the ini- tiative was the inadequate time for planning, sharing information, and working through implementation problems. Finally, the shrinking pool of financial resources available for staff training and for implementa- tion required creative thinking about sustaining the project through alternative funding. Lessons about bridging knowledge across disciplines. An- gela raised an important concern that many stakeholders in education are unaware of the knowledge from in the field of neuro- science explaining the learning processes involved in reading, writing, and mathematics. They could be at risk of employing instruc- tional practices and using expensive materials that are not appropriately addressing the developmental needs of many of the students in the classroom. This is of particular concern when working with struggling learners who may have specific cognitive deficits that contribute to their inefficiency in learning. By not considering the neurodiversity that exists among learners, educators could clearly miss opportunities to make significant changes in young people’s lives. One of the strengths of this research, as well as others on computerized working-memory training, is that it provides additional translational research through the examination of cognitive under- pinnings linked with learning and tests whether specific targeted interventions may lead to improvements in neuropsychological pro- cesses and academic skills of struggling students. Angela asserted that probably the most important group of stakeholders that needed to be aware of the research connecting the fields of neuroscience and education were teachers. One of the most powerful concepts that teachers should understand is that the brain is malleable, and not fixed as many educators were once trained to believe. Angela had reviewed the research on the plasticity of the brain and the discovery that reducing the negative effects of certain variables, such as poverty and stress, can improve specific cognitive processes that directly impact school performance. Therefore, she surmised, translating this knowledge to teachers may help them to feel more empowered about their crucial role in shaping young chil- dren’s current and future neuropsychological development. SUMMARY AND CONCLUSIONS: COMPARISON OF THE PHD AND PPD IN RIGOR AND RELEVANCE A Quantitative PhD Study on Working Memory The following summary of a quantitative dissertation study by Southard (2014) is presented here as a comparison with the previ- ous study in order to discuss distinctions between the criteria for rigor and the construct of relevance between the PhD and PPD pro- grams. The Southard quantitative study investigated the relationships among working memory, creativity (measured as divergent thinking and creative achievement), and nonverbal intelligence. The study also examined the roles of working memory and intelligence in the creative process. In order to examine this, participants were evaluat- Kochhar-Bryant Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.25 12 ed using a variety of cognitive tasks that included the Alternative Uses Test, the Consequences Task, the Creative Achievement Questionnaire, the Alloway Working Memory Assessment, and the matrix test from the Wechsler Abbreviated Scale of Intelligence. The sample for this study consisted of 166 students enrolled at the University of North Florida. Of the students in our sample 29 were Freshmen, 21 were Sophomores, 65 were Juniors, and 51 were Seniors. A large portion of our sample (42.77%) were between 18 and 20 years old, 30.72% were between 21 and 23 years old, 9.64% were ages 24–26, and 15.66% were 27 or older. The majority, 53.61%, of the participants’ ethnicity was Caucasian; 18.07% were African American, 13.25% were Hispanic, 7.23% were Asian/Pacific Islander, and 7.83% identified their ethnicity as Other. The results of this study indicate that verbal working memory was related to diver- gent thinking over and beyond intelligence and creative achievement (Southard, 2014). Standards for Rigor in the Scientific Process Many academicians equate good (i.e., rigorous) research with a positivist perspective and recommend an emphasis on quantitative skills and methods (Torff, 2011), proposing a hierarchy of data coll- ection and analysis techniques, some more “rigorous” than others. For example, doctoral programs funded by the Institute for Educa- tional Sciences (2008) to increase the supply of researchers in education give preference to rigorous education research prepara- tion for the conduct of efficacy and scale-up trials. IES refers to the “methodological superiority of randomized trials for drawing causal claims in areas in which outcomes are affected by many variables” (pg.10). The National Academies of Science (2002) concludes that nonrandomized studies are weaker in their ability to establish causa- tion than are randomized field trials (National Research Council, 2002). The practice of scientific research—whether quantitative, quali- tative, or mixed method—demands that the researcher’s work be intellectually rigorous, accurate on all points, ethical, and dispassion- ately executed (Bradford, 2015). Rigor in quantitative research is judged by how narrow, concise, and objective the design and analy- sis techniques are and how strictly the rules have been adhered to and applied to all decisions (IES, 2008). The steps involved in the scientific process vary slightly depending upon the field; the following is an outline of the general approach: • Make an observation or observations. • Ask questions about the observations and gather in- formation. • Form a hypothesis—a tentative description of what's been observed and predictions based on that obser- vation. • Test the hypothesis and predictions in an experiment that can be reproduced. • Analyze the data and draw conclusions; accept or re- ject the hypothesis or modify hypothesis as necessary. • Reproduce the experiment until there are no discrep- ancies between observations and theory (Bradford, 2015; Kantrowitz, 2014). However, there are differences of opinion about what the prepa- ration should look like for doctoral students. Rigor and the Problem of Practice Dissertation Kantrowitz (2014) argues that researchers need to be cautious about making claims that some data collection or analysis techni- ques are “more rigorous” than others, and furthermore, that researchers should stop associating standards and rigor only with confirmatory and hypothesis-driven research. She also advises cau- tion when claiming that fixed standards are needed for particular methodological techniques. Methodological techniques share a common set of properties but their power lies in the ability to adapt a specific research technique to different research contexts (Kantro- witz, 2014). Standards of rigor can be established for exploratory and descriptive research. Problems of practice—of implementation and translation of re- search findings to practice—have been referred to as “wicked problems” (Fixsen et al., 2009). They represent complex problems that occur in implementation conditions (“contested spaces”) that are in flux and that engage multiple layers of infrastructure of a system where various stakeholders can either support or obstruct transfor- mation (Gutiérrez & Vossoughi, 2010; Penuel & Spillane, 2014). Such conditions require a set of skills in the researcher that enables him or her to engage in a process with diverse stakeholders to define the problem of focus as being significant, persistent, and worthy of investigation. A reconceptualization of rigor for studies in these con- ditions requires sustained, direct, and systematic documentation of what takes place inside programs to document how systems being studied change and adapt interventions in interaction with each other in relation to their dynamic local contexts. Scientific rigor associated with qualitative research (nonexperi- mental) is defined as being open to the data, scrupulously adhering to a specific philosophical perspective, and thoroughness in collec- ting data (Guba, 1981). It is also judged by the logic of the emerging theory and whether the findings are contributing to what is already known about the phenomenon under study. Criteria for rigor and trustworthiness of the data included the following: • Truth value / credibility of the study: degree to which the researcher has established confidence in the truth of the findings for the subjects and the context in which the study was undertaken • Applicability: degree to which the findings can be ap- plied to other contexts, settings, or groups • Consistency: extent to which the findings would be consistent if inquiry were replicated with the same subjects or similar context • Neutrality: degree to which the findings are a function of the subjects/informats and condition of the rese- arch and not of other biases, motivations, and perspectives (Lincoln & Guba,1986) Ensuring a standard of rigor within a problem of practice disser- tation requires that the program provide candidates with the following: 1. An opportunity to envision solutions to significant prob- lems within the symbiotic space between the university and the stakeholder community 2. Challenging opportunities to express one’s values and talents, and thus to actualize one’s identity as a skilled professional 3. Opportunities to define and reflect on personal and pro- fessional transformations that contribute to one’s identity as a change agent, and commitment to a larger community impact Symbiotic Space Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.25 13 4. An opportunity to construct a theory of change that is grounded in theory and understanding of the history of previous interventions into the problem in the past; 5. An opportunity to develop a sense of mastering a chal- lenging situation and, through this, to feel a sense of responsibility for future outcomes 6. The challenge of exercising leadership for social change through professional and applied research ac- tivities that result in appreciable improvements on processes, policies, or social conditions within the community, particularly for vulnerable populations 7. Opportunities to use data to understand the effects of innovation, and ability to gather, organize, judge, and analyze situations, literature, and data through a critical lens (Shulman, 2005) 8. The shaping of a perspective that views cultural and linguistic diversity as an asset 9. Mentoring that engenders a sense of deep commitment to a research problem that is compelling for the individ- ual and for the community or organization in which it occurs The dissertation, therefore, is viewed as having the aim of achieving a goal that is much greater than the individual’s personal and professional development. Rather, it becomes an instrument for transformative development within the research context, and for the shaping of both a personal and a shared identity for change and a commitment to making a difference or effecting a change. The shared, negotiated process becomes an enriching aspect of the dissertation for the researcher and his or her relationship with the researched. Departing from the model of ‘get the data and run, the researcher develops a reciprocal relationship (Fine, Weis, We- seen, & Wong, 2003; Trainor & Bouchard, 2013; Zigo, 2001), ensuring that the subjects benefit from the study. Closing With several practitioner-oriented doctoral programs in embry- onic stages, along with the concurrent implementation of a new PhD, our faculty are navigating the tensions of past practices and behav- ioral and value sets, and the demands of a new conceptual frame. They are faced with disquieting issues related to the balance of re- sources among programs, equity in the balance of faculty load, challenges to program traditions, threats to faculty and student iden- tities, and dynamic standards of rigor. High-impact projects are rigorous, powerful, inspiring, bold, and often dramatic. Deliberate and sustained efforts within and outside the academic unit to promote understanding and positive communication about PPD contributions and expectations for candidates can add abundant reputational value to the institution. REFERENCES Ainsworth, M. S., Blehar, M. C., Walters, E., & Wall, S. (1978). Patterns of attachment: A psychological study of the strange situation. Hillsdale, NJ: Lawrence Erlbaum Associates. Basu, K. (March 29, 2012). Ending the first Ed.D. Program. Inside Education. Becker, H. S. (1982). Culture: A sociological view. Yale Review, 71, 513–527. Blount-Matthews, K. M., & Hertenstein, M.J. (2005). Attachment. Encyclopedia of Human Development. Sage Publications. Retrieved from http://www.sage- references.com.proxygw.wrlc.org/humandevelopment/Article_n59.html Bowlby, J. (1982). Attachment (Attachment and Loss: Vol. 1) New York: Basic Books. Bradford, A. (March 30, 2015). What is science and the scientific method? Retrieved from http://www.livescience.com/20896-science-scientific- method.html Brandenburg, R. T. (2008). Powerful pedagogy: Self-study of a teacher educa- tor’s practice. New York, NY: Springer Science & Business Media. Bryk, A. S. (2009). Support a science of performance improvement. Phi Delta Kappan, 90(8), 597–600. Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building networked improvement communities in education. In M. Halli- nan (Ed.), Frontiers in sociology of education (pp. 127–162). Dordrecht, Netherlands: Verlag. Caporael, L. R., Griesemer, J. R., & Wimsatt, W. C. (2013). Developing scaf- folds in evolution, culture, and cognition. Cambridge, MA: MIT Press. Carnegie Project on the Education Doctorate (CPED) (October, 2009). Work- ing Principles for the Professional Practice Doctorate in Education, pg 1. Retrieved from http://www.cpedinitiative.org/default.asp?page=HistoryPrinciples Chwe, M. S. (2001). Rational ritual: Culture, coordination, and common knowledge. Princeton, NJ: Princeton University Press. Cook, B. G., & Odom, S. L. (2013). Evidence-based practices and implementa- tion science in special education. Exceptional Children, 79 (2), 135–144. Easton, J. (2013, June). Using measurement as leverage between develop- mental research and education practice [Talk given at the Center for the Advanced Study of Teaching and Learning, Curry School of Education]. University of Virginia, Charlottesville. Eccles, M. P., & Mittman, B. S. (2006). Welcome to Implementation Science. Implementation Science, 1(1), 1–3. Retrieved from http://www.implementation science.com/content/1/1/1 Engeström, Y. (2011). From design experiments to formative interventions. Theory & Psychology, 21(5), 598–628. Erickson, F., & Gutierrez, K. (2002). Comment: Culture, rigor, and science in educational research. Educational Researcher, 31(8), 21–24. Fine, M., Weis, L., Weseen, S., & Wong, L. (2003). For whom? Qualitative research, representations, and social responsibilities. In N.K. Denzin & Y.S. Lincoln, The Sage handbook of qualitative research, 2nd ed. 107– 131. Thousand Oaks, CA: Sage. Fixsen, D. L., Blase, K. A., Duda, M. A., Naoom, S. F., & Van Dyke, M. (2009). Wicked problems: From demonstrations to transformation zones. Re- trieved from http://www.k12.wa.us/RTI/Implementation/pubdocs/ WA_2WickedProblems_NIRN0409_HO.pdf Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publica- tion #231). Retrieved from http://www.fpg.unc.edu/~nirn/resources/publications/ Mono- graph/pdf/Monograph_full.pdf Gallagher Freire, P. (1998). Pedagogy of freedom: Ethics, democracy and civic courage. Lanham, MD: Rowman & Littlefield. Gerson, E. M. (2013). Some problems of analyzing cultural revolution. In L.R. Caporael, J.R. Griesemer, & W.C. Wimsatt (Eds.), Developing Scaffolds in Evolution, Culture, and Cognition. Cambridge, MA: MIT Press. DOI:10.7551/mitpress/9780262019552.003.0013 Guba, E.G.(1981). Criteria for assessing the trustworthiness of naturalistic inquiries. Educational Resources Information Center Annual Review Pa- per, 29, 75-91. Gutierrez, K. D., & Penuel, W. R. (2014). Relevance to practice as a criterion for rigor. Educational Researcher, 43(1), 19–23. DOI: 10.3102/0013189X13520289 Gutiérrez, K. D., & Vossoughi, S. (2010). Lifting off the ground to return anew: Mediated praxis, transformative learning, and social design experiments. Journal of Teacher Education, 61(1/2), 100–117. Heishman, A. (2015). Effectiveness of computerized working memory training on math achievement and other transfer effects in children with ADHD and math difficulties. The George Washington University. 495; 3687559 DAI-A 76/08(E). Abstract retrieved from Dissertation Abstracts Interna- tional. Individuals with Disabilities Education Act, 20 U.S.C. § 1400 (2004). Institute of Education Sciences, U.S. Department of Education. (2008). Rigor and relevance redux: Director’s biennial report to Congress (IES 2009- 6010). Washington, DC. Kantrowitz, B. (July 15, 2014). Scientists bring new rigor to education research. Retrieved from https://www.scientificamerican.com/article/scientists- bring-new-rigor-to-education-research/ Kochhar-Bryant Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 2 (2017) DOI 10.5195/ie.2017.25 14 Kemmis, Wilkenson, Edwards-Groves, Hardy, Grootenboer, & Bristol. (2014). Changing practices, changing education. Springer Education. Kochhar-Bryant, C.A. (2016). Identity, commitment and change agency: Bedrock for bridging theory and practice in doctoral education. In V.A. Storey and K.A. Hesbol, Contemporary Approaches to Dissertation Development and Research Methods, Edition 1, 29-42. Hershey, PA: IGI-Global. DOI: 10.4018/978-1-5225-0445-0 Lincoln, Y. S., & Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. In D.D. Williams (Ed.), Naturalistic evaluation (pp. 73–84). San Francisco: Jossey-Bass. Mandel Scholars Program, Mandel School for Educational Leadership. Home Page, Background. Retrieved from https://institute.mandelfoundation.org.il/english/LeadershipSchool/Pages/ Introduction.aspx National Research Council. (2002). Scientific research in education. Washing- ton, DC: National Academy Press. Nisan, M., & Pekarsky, D. (2009). Educational identity as a major factor in the development of educational leadership. Jerusalem, Israel: Mandel Insti- tute, Hebrew University. O’Brien, M. J., & Shennan, S. J. (Eds.) (2010). Innovation in cultural systems. Cambridge, MA: MIT Press. Odom, S. L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. (2005). Research in special education: Scientific methods and evidence-based practices. Exceptional Children, 2, 137–148.Osterman, K., Furman, G., & Sernak, K. (2013). Action Research in EdD Programs in Educational Leadership. Journal of Research on Educational Leader- ship, 9 (1). http://journals.sagepub.com/stoken/default+domain/gpjGvuqgscXx9Uwpf aBC/full Penuel, W. R., & Spillane, J. P. (2014). Learning sciences and policy design and implementation: Key concepts and tools for collaborative engage- ment. In R.K. Sawyer (Ed.), Cambridge handbook of the learning sciences (2nd ed.). Cambridge, UK: Cambridge University Press. Perry, J. (2012). Discussion extended—Feb 10th: Rigor, consensus or impact? What is the CPED measure of high quality? Retrieved from http://cpedinitiative.org/forum-topic/discussion-extended-feb-10th-rigor- consensus-or-impact-what-cped-measure-high-quality Pickering, A. (2010). The mangle of practice: Time, agency, and science. Chicago, IL: University of Chicago Press. Ravitch, S. M. & Riggan, M. (2012). Reason & rigor: How conceptual frame- works guide research. Thousand Oaks, CA: Sage. Reardon, R., & Shakeshaft, C. (2013). Criterion-inspired, emergent design in doctoral education: A critical friends perspective. In V. Storey, Redesign- ing Professional Education Doctorates: Applications of Critical Friendship Theory to the EdD. (pp. 177–194). New York: Palgrave-Macmillan. Reardon, S. F. (2011). The widening socioeconomic status achievement gap: New evidence and possible explanations. In R. J. Murnane & G. J. Dun- can (Eds.), Whither opportunity? Rising inequality, schools, and children's life chances. New York: Russell Sage Foundation. Ross, C. (2010). Changing your school? First, work on the culture. Lansing, MI: Education Partnerships Inc. Rossier School of Education. (n.d.). Side-by-side comparisons: Doctoral pro- grams. University of Southern California. Retrieved from http://rossier.usc.edu/programs/doctoral-related/ Sands, D., Blunck, R. Davis, A., Fulmer, C., Leech, N., Ruiz-Primo, M., … Zion, S. (2013). Critical friends' perspectives on problems of practice and inquiry in an EdD Program. In V. Storey, Redesigning Professional Edu- cation Doctorates: Applications of Critical Friendship Theory to the EdD. New York: Palgrave-Macmillan. Schein, E.H. (2004). Organizational Culture and Leadership. 3rd Edition. San Francisco: Jossey-Bass. Shulman, L. (2005). Pedagogies of uncertainty. Liberal Education, Spring. Washington, D.C.: Association of American Colleges and Universities. Southard, E. M. (2014). Examining the relationships among working memory, creativity, and intelligence. Doctoral Dissertation. UNF Digital Commons. University of Northern Florida. Torff, B. (2011). Preparing the educational researchers the world needs. Teachers College Record. Retrieved from http://www.tcrecord.org/Content.asp?Contentid=16285 Trainor, A., & Bouchard, K. (2013). Exploring and developing reciprocity in research design. International Journal of Qualitative Studies in Educa- tion, 26 (8), 986-1003. University of Southern California (2017). Side-by-side Comparisons – Doctoral Programs. Rossier School of Education, Los Angeles, California. Re- trieved from http://rossier.usc.edu/programs/doctoral-related/ Van Maanen, J., & Schein, E. H. (1979). Toward of theory of organizational socialization. Research in Organizational Behavior, 1, 209–264. Walker, G., Golde, C., Jones, L., Bueschel, A., & Hutchings, P. (2009). The formation of scholars: Rethinking doctoral education for the twenty-first century. New York: Wiley & Sons. Zambo, R., Zambo, D., Buss, R. R., Perry, J. A., & Williams, T. R. (2014). Seven years after the call: Students’ and graduates’ perceptions of the re-envisioned EdD. Innovative Higher Education (39)2, 123–137. Zigo, D. (2001). Rethinking reciprocity: Collaboration in labor as a path toward equalizing power in classroom research. International Journal of Qualita- tive Studies in Education, 14(3), 351–365. Zusman, A. (2013). Degrees of change: How new kinds of professional doctor- ates are changing higher education institutions. Research and Occasional Paper Series, CSHE 8-13. Center for Studies in Higher Edu- cation. Berkeley, CA: University of California.