Journal website: http://epaa.asu.edu/ojs/ Manuscript received: 10/1/2017 Facebook: /EPAAA Revisions received: 12/29/2018 Twitter: @epaa_aape Accepted: 1/17/2018 SPECIAL ISSUE Redesigning Assessment and Accountability education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 26 Number 10 January 29, 2018 ISSN 1068-2341 “Seeing the Whole Elephant”: Changing Mindsets and Empowering Stakeholders to Meaningfully Manage Accountability and Improvement Susan Bush-Mecenas Julie A. Marsh University of Southern California David Montes de Oca California Office to Reform Education (CORE) & Heather Hough Policy Analysis for California Education (PACE) United States Citation: Bush-Mecenas, S., Marsh, J. A., Montes de Oca, D., & Hough, H. (2018). “Seeing the whole elephant”: Changing mindsets and empowering stakeholders to meaningfully manage accountability and improvement. Education Policy Analysis Archives, 26(10). http://dx.doi.org/10.14507/epaa.26.3440 This article is part of the special issue, Redesigning Systems of Assessment and Accountability for Meaningful Student Learning, guest edited by Soung Bae, Jon Snyder, and Elizabeth Leisy Stosich. http://epaa.asu.edu/ojs/ http://dx.doi.org/10.14507/epaa.26.3440 Education Policy Analysis Archives Vol. 26 No. 10 SPECIAL ISSUE 2 Abstract: School accountability and improvement policy are on the precipice of a paradigm shift. While the multiple-measure dashboard accountability approach holds great promise for promoting more meaningful learning opportunities for all students, our research indicates that this can come with substantial challenges in practice. We reflect upon the lessons learned from our recent research on the CORE districts’ use of multiple - measure data dashboards. Our research indicated that a shift to greater flexibility and locally determined capacity building efforts brings its own set of challenges. Building on this foundation and drawing upon the expertise of a central practitioner-leader, we explore these issues through one promising case: the use of these multiple-measure accountability systems in one CORE district, Oakland Unified School District (OUSD). OUSD’s approach suggests that districts have substantial agency to help modify the mindsets of practitioners and the community, by modeling the values of inquiry, openness, and flexibility. By loosening the reigns, districts can give school stakeholders the space and authority to meaningfully manage their own accountability and improvement. Keywords: accountability; assessment; improvement “Viendo el elefante entero”: Cambiando las mentalidades y empoderando los accionistas de manera significativa para la rendición de cuentas y la mejoría Resumen: Reflexionamos sobre las lecciones de nuestra investigación reciente sobre el uso de los paneles de datos de medidas múltiples por parte de los distritos CORE. Nuestra investigación indicó que cambios hacia más flexibilidad y esfuerzos de desarrollo de capacidades locales trae desafíos. Sobre esta fundación y aprovechando la experiencia de un líder médico central, exploramos estos asuntos a través de un caso prometedor: el uso de estos sistemas de responsabilidad múltiple en un distrito CORE, el Distrito Escolar Unificado de Oakland (OUSD). El enfoque de OUSD sugiere que los distritos tienen agencia sustancial para ayudar a modificar la mentalidad de los profesionales y la comunidad, usando los valores de la investigación, la actitud receptiva y la flexibilidad. Al aflojar las riendas, los distritos pueden dar espacio a las partes interesadas de la escuela y la autoridad para gestionar de manera significativa su propia responsabilidad y mejoría. Palabras clave: responsabilidad; evaluación; mejoría “Ver o elefante inteiro”: Mudar mentalidades e capacitar os acionistas de forma significativa para a responsabilidade e a melhoria Resumo: Refletimos sobre as lições de nossa pesquisa recente sobre o uso de painéis de dados de medidas múltiplas por distritos CORE. Nossa pesquisa indicou que as mudanças para mais flexibilidade e esforços de desenvolvimento de capacidade local trazem desafios. Com base nessa base e com base na experiência de um líder médico central, exploramos essas questões através de um caso promissor: o uso desses sistemas de responsabilidade múltipla em um distrito CORE, o Distrito Escolar Unificado de Oakland (OUSD). A abordagem OUSD sugere que os distritos tenham uma agência substancial para ajudar a modificar a mentalidade dos profissionais e da comunidade, usando valores de pesquisa, atitude receptiva e flexibilidade. Ao afrouxar as rédeas, os distritos podem dar espaço às partes interessadas da escola e à autoridade para gerenciar significativamente suas próprias responsabilidades e melhorias. Palavras-chave: responsabilidade; avaliação; melhoria “Seeing the whole elephant” 3 Seeing the Whole Elephant”: Changing Mindsets and Empowering Stakeholders to Meaningfully Manage Accountability and Improvement School accountability and improvement policy are on the precipice of a paradigm shift. Most importantly, the federal Every Student Succeeds Act (ESSA) requires a more comprehensive assessment of school performance and a less prescriptive, local approach to school support. While the multiple-measure, locally-led accountability approach holds great promise for promoting more meaningful learning opportunities for all students, our research indicates that this can come with substantial challenges in practice. We reflect upon the lessons learned from our recent research on the CORE districts’ use of a multiple-measure accountability system (see Marsh, Bush-Mecenas, & Hough, 2016, 2017). Building on this foundation, we explore a promising case: the use of these multiple measures to drive school improvement in one CORE district, Oakland Unified School District. Lessons from CORE As discussed by Bae (2018), six California districts (Oakland, Fresno, Long Beach, Los Angeles, San Francisco, and Santa Ana) developed an innovative accountability system under a waiver from No Child Left Behind (NCLB) resting on three key tenets: 1) the importance of local control and local solutions to problems, 2) a focus on shared responsibility and support over compliance, and 3) the belief that districts can best build capacity through peer-to-peer collaboration focused on data. At present, the CORE districts (now including Sacramento City and Garden Grove) continue to work together to improve outcomes for the million students they serve, the majority of whom are minority students who are socioeconomically disadvantaged. Together, the CORE districts have created a unique school performance measurement system that includes holistic measures of academic growth, social-emotional learning, and school culture/climate. While no longer implementing their waiver since the passage of ESSA, the districts use this measurement system to inform and deepen California state’s accountability and measurement system. Thus, the CORE districts present an opportunity to learn how to effectively utilize multiple measures of school quality, develop shared accountability, and grow capacity for schools and districts to improve. In 2015, our team of researchers working with Policy Analysis for California Education (PACE) examined how administrators understood, implemented, and responded to CORE’s new accountability system (see Marsh et al., 2016, 2017). Overwhelmingly, we found that district and school administrators greatly appreciated the more holistic approach to measurement, the emphasis on support over sanctions, and the transparency of the CORE approach. Further, they reported using measurement system results to inform decision-making around resource allocation, stakeholder communication, and improvement planning. However, despite these benefits of a multiple measure accountability system, the districts reported several key challenges: First, we found that the CORE system needed to be modified to suit local needs. While this may have allowed districts to create a more relevant and meaningful accountability system, it also revealed a tension between shared accountability and local variation. While many agreed that locally defined and relevant measures were valuable, some reported prioritizing high-stakes standardized measures given their limited resources and time. Second, many in the districts were concerned with unintended consequences. Bae (2018) points to the value of transparency as a means of “incentiviz[ing] the behaviors we seek to encourage (e.g., a focus on a holistic view of student learning and development, the incorporation of noncognitive indicators) and safeguard[ing] against the behaviors we seek to discourage (e.g., a Education Policy Analysis Archives Vol. 26 No. 10 SPECIAL ISSUE 4 shortsighted focus on standardized test scores in two subject areas, the narrowing of the curriculum to tested subject areas)” (pp. 21-2). Our research indicates that CORE’s multiple-measure system still evoked strategic behaviors that might invalidate measurement and preclude learning, among some actors. For example, some district leaders reported that administrators asked them to share the “right” answers on social-emotional skills and school culture-climate surveys, in order to achieve a higher school rating. Others worried that the methods of reporting of student suspension were altered to lower suspension rates. Moving away from compliance mindsets towards a learning orientation proved challenging. Finally, new measures take a long time to be fully understood and acted upon. While CORE’s measurement system holds promise for improving local systems, efforts to grow capacity around the measurement system were still a work in progress. Administrators believed that additional capacity needed to be grown around understanding the new social-emotional learning and school culture measures, as well as how to actually develop and improve social-emotional learning and school culture. The following case presents insight into how districts can substantively shift mindsets to advance the intent of multiple-measure approaches. The quotes throughout this narrative come from our practitioner colleague and co-author David Montes de Oca, former Senior Deputy Chief of the Office of Post-Secondary Readiness for Oakland Unified and current Deputy Chief of Improvement for the CORE Districts. Changing Mindsets in Oakland Oakland Unified School District has embraced and expanded upon CORE’s multiple- measure accountability approach, including: broad outcome measures, a mix of state and local accountability, measuring Opportunities to Learn (OTL), data dashboards, and School Quality Reviews. Together, these measures, and the structures they are embedded within, are intended to help stakeholders in “seeing the whole elephant, taking the blindfolds off and seeing the schools.”1 OUSD worked to avoid the potential pitfalls identified by Bae (2018; e.g., low parent and community support, compliance-oriented staff responses, and inadequate capacity to address improvement needs): we ask, how? In executing local school improvement, OUSD took a proactive approach to redefining the school’s purpose, identifying highly localized, formative performance indicators in collaboration with all stakeholders, and responding to practitioner feedback on district hindrances to improvement. After exiting state receivership in 2009, leaders in Oakland took a bold move: to reimagine the district as a collection of full-service community schools committed to meeting the academic, social-emotional, and physical health needs of students and the community. Leaders met with about 400 stakeholders for over a year to understand how to measure quality in full-service community schools using explicit quality standards, measured in qualitative and quantitative ways. Engaging focus groups of students, parents, and educators, OUSD defined four values: 1) attention to and responsibility for the whole child, using multiple measures for a comprehensive view of how a child is doing; 2) impact, represented by calculating growth on every measure (including surveys, chronic absence, etc.); 3) equity, ensured by weighting groups of students, such as identifying the lowest-performing racial/ethnic group as an indicator across most metrics; and 4) college, career, and community readiness, represented by the linkages between measures and ultimate student readiness for life after high school. 1 This quote references a Buddhist parable memorialized in a poem by John Godfrey Saxe, in which six blind learned men observe an elephant and come to different conclusions about its nature, capturing parts but not the whole depending on which part of the elephant each has happened to touch. “Seeing the whole elephant” 5 This regular stakeholder engagement certainly meets Bae’s recommendation of meaningful stakeholder engagement, and likely enhanced buy-in and commitment to the reform. To break down the persistent compliance orientation of school staff, OUSD then began pivoting from accountability as a “hammer and feeling particularly judged” toward using measurement as a “flashlight” to promote “learning, improvement, growth and the idea that we’re constantly seeking to get better.” Specifically, OUSD meaningfully enhanced the annual “snapshot” measurement structure with more regular, formative metrics, while allowing schools to design their own indicators to use for engagement in cycles of inquiry and improvement that involved “testing” ideas. They also challenged school leadership teams to identify barriers to achieving their improvement goals. At times, this meant challenging district policy and practices. For example, school leadership teams identified the frequency and scheduling of certain assessments as a hindrance to their improvement efforts. As a result, district administrators modified the assessment calendaring process in response to this feedback. Together, these actions represented a substantial shift in mindsets for school staff: from top-down accountability metrics to locally defined formative indicators, from set-in-stone initiatives to trying out potential solutions, from being restricted by district policy and practice to being empowered to make the district work for schools. Of course, OUSD faces ongoing challenges in managing accountability and building capacity for improvement, which are likely to test all states and districts embracing multiple-measure approaches to accountability and support. Essentially, all such districts face a balancing act, addressing inherent challenges in the system: the tension between accountability and continuous improvement; the candor to admit that few individuals in the school system possesses a core understanding of equity and how to promote equitable outcomes; and the continual sense of “discomfort, trepidation, and risk” among stakeholders that is necessary to motivate continuous improvement. The aspiration is to distribute these challenges and the weight of this heavy lift across “many hands and hearts.” Oakland’s approach suggests that districts have substantial power to help modify the mindsets of practitioners and the community, but this involves modeling the values of inquiry, openness, and flexibility. By loosening the reigns, districts can give school stakeholders the space and authority to meaningfully manage their own accountability and improvement, to give them “empowered agency about being able to do something about it right now and every day.” References Bae, S. (2018). Redesigning systems of school accountability: A multiple measures approach to accountability and support. Education Policy Analysis Archives, 26(7). http://dx.doi.org/10.14507/epaa.26.2920 Marsh, J. A., Bush-Mecenas, S., & Hough, H. (2017). Learning from early adopters in the new accountability era: Insights from California's CORE waiver districts. Educational. Educational Administration Quarterly, 53(3), 327-364. - https://doi.org/10.1177/0013161X16688064 Marsh, J. A., Bush-Mecenas, S., & Hough, H. (2016). Local control in action: Learning from the CORE districts’ focus on measurement, capacity building, and shared accountability. [Policy Brief]. Stanford, CA: Policy Analysis for California Education. http://dx.doi.org/10.14507/epaa.26.2920 https://doi.org/10.1177/0013161X16688064 Education Policy Analysis Archives Vol. 26 No. 10 SPECIAL ISSUE 6 About the Authors Susan Bush-Mecenas University of Southern California sbush@usc.edu Susan Bush-Mecenas is a PhD candidate and Provost/Final Year fellow at the University of Southern California’s Rossier School of Education. Her research interests include organizational learning, district reform, district and school capacity building, and accountability policy. David Montes de Oca California Office to Reform Education (CORE) Districts david.m@coredistricts.org David Montes de Oca is the Deputy Chief of Improvement for the CORE Districts. Formerly he served as Senior Deputy Chief of the Office of Post-Secondary Readiness for Oakland Unified School District. Julie A. Marsh University of Southern California Julie.marsh@rossier.usc.edu Julie A. Marsh is an associate professor of education policy at the University of Southern California’s Rossier School of Education and specializes in rese arch on K-12 policy. Her research blends perspectives in education, sociology, and political science. Her research focuses on the implementation and effects of accountability and instructional reform policies, including the roles of central office administrators, intermediary organizations, and community members in educational reform and the use of data to guide decision making. Heather Hough Policy Analysis for California Education hjhough@stanford.edu Heather Hough is executive director of the CORE-PACE Research Partnership at Policy Analysis for California Education. Her area of expertise is in district - and state-level policymaking and implementation, with a particular focus on teacher compensation, support, and accountability; policy coherence; and system improvement. About the Guest Editors Elizabeth Leisy Stosich Fordham University estosich@fordham.edu Elizabeth Leisy Stosich is an Assistant Professor in Educational Leadership, Administration, and Policy at Fordham University. Previously, she was a Research and Policy Fellow at the Stanford Center for Opportunity Policy in Education. Her research interests include education policy, assessment and accountability, school and district leadership, school improvement, and teachers’ professional learning. Soung Bae Stanford University mailto:sbush@usc.edu mailto:david.m@coredistricts.org mailto:Julie.marsh@rossier.usc.edu mailto:hjhough@stanford.edu mailto:estosich@fordham.edu “Seeing the whole elephant” 7 soungb@stanford.edu Soung Bae is a Senior Learning Specialist and UDL Innovation Studio Manager at the Schwab Learning Center at Stanford University. Formerly, she was a Senior Research and Policy Analyst at the Stanford Center for Opportunity Policy in Education. Her research interests focus on school accountability, student engagement, and designing learning environments that appreciate and support learner variability. Jon Snyder Stanford University jdsnyder@stanford.edu Jon Snyder is the Executive Director of the Stanford Center for Opportunity Policy in Education (SCOPE). His research interests include teacher learning, conditions that support teacher learning, and the relationships between teacher and student learning. SPECIAL ISSUE Redesigning Assessment and Accountability education policy analysis archives Volume 26 Number 10 January 29, 2018 ISSN 1068-2341 Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for non- commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the author(s) or EPAA. EPAA is published by the Mary Lou Fulton Institute and Graduate School of Education at Arizona State University Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A1 (Brazil), SCImago Journal Rank; SCOPUS, SOCOLAR (China). Please send errata notes to Audrey Amrein-Beardsley at Audrey.beardsley@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. mailto:soungb@stanford.edu mailto:jdsnyder@stanford.edu http://www.doaj.org/ mailto:Audrey.beardsley@asu.edu https://www.facebook.com/EPAAAAPE Education Policy Analysis Archives Vol. 26 No. 10 SPECIAL ISSUE 8 education policy analysis archives editorial board Lead Editor: Audrey Amrein-Beardsley (Arizona State University) Editor Consultor: Gustavo E. Fischman (Arizona State University) Associate Editors: David Carlson, Lauren Harris, Eugene Judson, Mirka Koro-Ljungberg, Scott Marley, Iveta Silova, Maria Teresa Tatto (Arizona State University) Cristina Alfaro San Diego State University Gene V Glass Arizona State University Susan L. Robertson Bristol University, UK Gary Anderson New York University Ronald Glass University of California, Santa Cruz Gloria M. Rodriguez University of California, Davis Michael W. Apple University of Wisconsin, Madison Jacob P. K. Gross University of Louisville R. Anthony Rolle University of Houston Jeff Bale OISE, University of Toronto, Canada Eric M. Haas WestEd A. G. Rud Washington State University Aaron Bevanot SUNY Albany Julian Vasquez Heilig California State University, Sacramento Patricia Sánchez University of University of Texas, San Antonio David C. Berliner Arizona State University Kimberly Kappler Hewitt University of North Carolina Greensboro Janelle Scott University of California, Berkeley Henry Braun Boston College Aimee Howley Ohio University Jack Schneider College of the Holy Cross Casey Cobb University of Connecticut Steve Klees University of Maryland Noah Sobe Loyola University Arnold Danzig San Jose State University Jaekyung Lee SUNY Buffalo Nelly P. Stromquist University of Maryland Linda Darling-Hammond Stanford University Jessica Nina Lester Indiana University Benjamin Superfine University of Illinois, Chicago Elizabeth H. DeBray University of Georgia Amanda E. Lewis University of Illinois, Chicago Adai Tefera Virginia Commonwealth University Chad d'Entremont Rennie Center for Education Research & Policy Chad R. Lochmiller Indiana University Tina Trujillo University of California, Berkeley John Diamond University of Wisconsin, Madison Christopher Lubienski Indiana University Federico R. Waitoller University of Illinois, Chicago Matthew Di Carlo Albert Shanker Institute Sarah Lubienski Indiana University Larisa Warhol University of Connecticut Sherman Dorn Arizona State University William J. Mathis University of Colorado, Boulder John Weathers University of Colorado, Colorado Springs Michael J. Dumas University of California, Berkeley Michele S. Moses University of Colorado, Boulder Kevin Welner University of Colorado, Boulder Kathy Escamilla University of Colorado, Boulder Julianne Moss Deakin University, Australia Terrence G. Wiley Center for Applied Linguistics Melissa Lynn Freeman Adams State College Sharon Nichols University of Texas, San Antonio John Willinsky Stanford University Rachael Gabriel University of Connecticut Eric Parsons University of Missouri-Columbia Jennifer R. Wolgemuth University of South Florida Amy Garrett Dikkers University of North Carolina, Wilmington Amanda U. Potterton University of Kentucky Kyo Yamashiro Claremont Graduate University “Seeing the whole elephant” 9 archivos analíticos de políticas educativas consejo editorial Editor Consultor: Gustavo E. Fischman (Arizona State University) Editores Asociados: Armando Alcántara Santuario (Universidad Nacional Autónoma de México), Jason Beech (Universidad de San Andrés), Ezequiel Gomez Caride (Pontificia Universidad Católica Argentina), Antonio Luzon (Universidad de Granada), Angelica Buendia (Metropolitan Autonomous University), José Luis Ramírez (Universidad de Sonora) Claudio Almonacid Universidad Metropolitana de Ciencias de la Educación, Chile Juan Carlos González Faraco Universidad de Huelva, España Miriam Rodríguez Vargas Universidad Autónoma de Tamaulipas, México Miguel Ángel Arias Ortega Universidad Autónoma de la Ciudad de México María Clemente Linuesa Universidad de Salamanca, España José Gregorio Rodríguez Universidad Nacional de Colombia, Colombia Xavier Besalú Costa Universitat de Girona, España Jaume Martínez Bonafé Universitat de València, España Mario Rueda Beltrán Instituto de Investigaciones sobre la Universidad y la Educación, UNAM, México Xavier Bonal Sarro Universidad Autónoma de Barcelona, España Alejandro Márquez Jiménez Instituto de Investigaciones sobre la Universidad y la Educación, UNAM, México José Luis San Fabián Maroto Universidad de Oviedo, España Antonio Bolívar Boitia Universidad de Granada, España María Guadalupe Olivier Tellez, Universidad Pedagógica Nacional, México Jurjo Torres Santomé, Universidad de la Coruña, España José Joaquín Brunner Universidad Diego Portales, Chile Miguel Pereyra Universidad de Granada, España Yengny Marisol Silva Laya Universidad Iberoamericana, México Damián Canales Sánchez Instituto Nacional para la Evaluación de la Educación, México Mónica Pini Universidad Nacional de San Martín, Argentina Ernesto Treviño Ronzón Universidad Veracruzana, México Gabriela de la Cruz Flores Universidad Nacional Autónoma de México Omar Orlando Pulido Chaves Instituto para la Investigación Educativa y el Desarrollo Pedagógico (IDEP) Ernesto Treviño Villarreal Universidad Diego Portales Santiago, Chile Marco Antonio Delgado Fuentes Universidad Iberoamericana, México Paula Razquin Universidad de San Andrés, Argentina Antoni Verger Planells Universidad Autónoma de Barcelona, España Inés Dussel, DIE-CINVESTAV, México javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/819') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/820') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/4276') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/1609') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/825') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/797') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/555') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/823') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/798') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/814') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/2703') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/801') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/816') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/826') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/802') Education Policy Analysis Archives Vol. 26 No. 10 SPECIAL ISSUE 10 arquivos analíticos de políticas educativas conselho editorial Editor Consultor: Gustavo E. Fischman (Arizona State University) Editores Associados: Geovana Mendonça Lunardi Mendes (Universidade do Estado de Santa Catarina), Marcia Pletsch, Sandra Regina Sales (Universidade Federal Rural do Rio de Janeiro) Almerindo Afonso Universidade do Minho Portugal Alexandre Fernandez Vaz Universidade Federal de Santa Catarina, Brasil José Augusto Pacheco Universidade do Minho, Portugal Rosanna Maria Barros Sá Universidade do Algarve Portugal Regina Célia Linhares Hostins Universidade do Vale do Itajaí, Brasil Jane Paiva Universidade do Estado do Rio de Janeiro, Brasil Maria Helena Bonilla Universidade Federal da Bahia Brasil Alfredo Macedo Gomes Universidade Federal de Pernambuco Brasil Paulo Alberto Santos Vieira Universidade do Estado de Mato Grosso, Brasil Rosa Maria Bueno Fischer Universidade Federal do Rio Grande do Sul, Brasil Jefferson Mainardes Universidade Estadual de Ponta Grossa, Brasil Fabiany de Cássia Tavares Silva Universidade Federal do Mato Grosso do Sul, Brasil Alice Casimiro Lopes Universidade do Estado do Rio de Janeiro, Brasil Jader Janer Moreira Lopes Universidade Federal Fluminense e Universidade Federal de Juiz de Fora, Brasil António Teodoro Universidade Lusófona Portugal Suzana Feldens Schwertner Centro Universitário Univates Brasil Debora Nunes Universidade Federal do Rio Grande do Norte, Brasil Lílian do Valle Universidade do Estado do Rio de Janeiro, Brasil Flávia Miller Naethe Motta Universidade Federal Rural do Rio de Janeiro, Brasil Alda Junqueira Marin Pontifícia Universidade Católica de São Paulo, Brasil Alfredo Veiga-Neto Universidade Federal do Rio Grande do Sul, Brasil Dalila Andrade Oliveira Universidade Federal de Minas Gerais, Brasil