TOWARDS AN OBJECTIVE EVALUATION OF TEACHER PERFORMANCE Journal website: http://epaa.asu.edu/ojs/ Manuscript received: 11/5/2014 Facebook: /EPAAA Revisions received: 9/6/2015 Twitter: @epaa_aape Accepted: 9/29/2015 education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 23 Number 109 November 2nd, 2015 ISSN 1068-2341 Research and evidence in education decision-making: A comparison of results from two pan-Canadian studies Gerald Galway Memorial University & Bruce Sheppard Memorial University Canada Citation: Galway, G., & Sheppard, B. (2015). Research and evidence in education decision-making: A comparison of results from two pan-Canadian studies. Education Policy Analysis Archives, 23(109). http://dx.doi.org/10.14507/epaa.v23.1905 Abstract: In this paper we compare the use of research and other evidence in the decision-making practices of two groups of education policy elites, situated in different contexts – provincial education ministries and school districts. Data are derived from two pan-Canadian studies: Galway (2006) and Sheppard, Galway, Brown and Wiens (2013). The findings show that policy decisions at the ministry level are informed primarily by political and pragmatic factors, personal and professional beliefs and local knowledge. The role of external research is shown to be relatively marginal and confined to quantitative studies and performance assessments. Decision makers at the school district level attend less to political and pragmatic influences relying more on personal beliefs, values and experiential factors supplemented by the advice of professional staff and in -house research/indicators. Results from both studies demonstrate limited reliance on external data and university-based research – the latter ranking 15th of 20 influencing factors. http://epaa.asu.edu/ojs/ Education Policy Analysis Archives Vol. 23 No. 109 2 Consistent with Beck’s (1994; 1997) risk theory, we theorize that education po licy making in both contexts is influenced by both macro- and micro-level factors, where choice of policy evidence is mediated by personal considerations and political risk factors. This suggests a weak policy development paradigm that is resistant to inde pendent research- informed evidence. Keywords: education; school boards; ministries of education; government; research; evidence; decision-making; policy; risk theory. Investigación y evidencia científica en la toma de decisiones educativas: una comparación de los resultados de dos estudios pan-canadienses Resumen: En este artículo comparamos el uso que se le ha dado a la investigación y otras evidencias científicas en la toma de decisiones de dos grupos de élite de política educativa, provenientes de dos contextos distintos –los ministerios de educación provinciales y los distritos escolares. Los datos se derivan de dos estudios pan -canadienses: Galway (2006) y Sheppard, Galway, Brown and Wiens (2013). Los resultados muestran que las decisiones políticas a nivel ministerial están principalmente influenciadas por factores políticos y pragmáticos, creencias personales y profesionales y conocimiento local. El papel de la investigación externa se demuestra que es relativamente marginal y se reduce a estudios cuantitativos y evaluaciones de desempeño. Los responsables de la toma de decisiones a nivel de distrito escolar están menos influenciados por los factores políticos y pragmáticos y confían más en las creencias, valores y experiencias personales junto con el asesoramiento de profesionales y los resultados de investigaciones internas. Los resultados de ambos estudios demuestran una escasa confianza en los datos externos y la investigación universitaria –que ocupa el puesto número 15 de 20 en el ranking de facto res influyentes. En consonancia con la teoría del riesgo de Beck (1994; 1997), sostenemos que la formulación de políticas educativas en ambos contextos se ve influenciada por factores macro y micro, donde la elección de la evidencia política está mediatiza da por consideraciones personales y factores de riesgo político. Esto supone un débil desarrollo del modelo político que es resistente a las pruebas procedentes de investigación independiente. Palabras-clave: educación; consejos escolares; ministerios de educación; gobierno; investigación; evidencia; toma de decisiones; política; teoría del riesgo. Pesquisa e dados científicos na toma de decisões educacionais: uma comparação entre os resultados de dois estudos pan-canadenses. Resumo: Neste trabalho, comparamos o uso que tem sido dado à investigação e outras evidencias científicas na tomada de decisões de dois grupos da elite política em educação, de dois contextos diferentes -o ministérios provinciais de educação e distritos escolares. Os dados foram obtidos de dois estudos pan-canadenses: Galway (2006) e Sheppard, Galway, Brown e Wiens (2013). Os resultados mostram que as decisões políticas a nível ministerial são influenciados principalmente por fatores políticos e pragmáticos, crenças pessoais e profissionais e conhecimento local. O papel da investigação externa mostra que é relativamente marginal, reduzida para estudos quantitativos e avaliações de desempenho. Os responsáveis pela tomada de decisões em nível de distrito escolar são menos influenciados por fatores políticos e pragmáticos e confiam mais nas crenças, valores e experiências pessoais com aconselhamento profissional e os resultados das investigações internas. Os resultados de ambos estudos demonstram uma falta de confiança nos dados externos e pesquisa acadêmica, que é classificada como a No. 15 de 20 no ranking dos fatores influentes. Em Research and evidence in education decision -making 3 consonância com a teoria do risco Beck (1994; 1997), que argumentam que a formulação de políticas educacionais em ambos os contextos é influenciada por fatores macro e micro, onde a escolha de evidência política é influenciada por considerações pessoais e fatores de risco política. Este é um fraco desenvolvimento do modelo político que é resistente a evidência de pesquisa independente. Palavras-chave: educação; conselhos escolares; Ministérios da Educação; governo; pesquisa; provas; tomada de decisão; política; teoria do risco. Introduction We begin this paper with a short account of three incidents, which brought us to the realization that education policy making is a very public enterprise, where beliefs, opinions and politics are significant policy considerations. Several years ago, at the beginning of the school year, the front-page story in a major provincial newspaper bemoaned the fact that Francophone students had performed very poorly in provincial examinations and there was some speculation that the school board was placing linguistic concerns ahead of subject area competence in the recruitment of teachers. As the issue unfolded, and despite the absence of any supporting research, this ‘conventional wisdom’ seemed to become accepted as professional knowledge within the K-12 education community where we both held senior administrative positions. Some time later, one of us was purchasing symphony tickets when the artistic director of the orchestra came rushing out of his office with program notes for the season’s inaugural concert. In the conversation that ensued, he highlighted the British Prime Minister’s plan to redouble Britain’s investment in music education. He then volunteered that music education was dying in Canadian schools, a message that later appeared in the symphony’s program booklet and was reported in the local media. Then, a few weeks later, at a meeting of provincial education ministers, a colleague commented on a story from a regional newspaper in France. The article described the French Education Minister’s vision for his country’s education system, which emphasized the continuing relevance of core subjects and traditional methods, such as reading, writing and dictation. He noted that the shopkeeper who sold him the paper engaged in a lengthy monologue on what he felt was wrong with the Minister’s plan. While, in and of themselves, neither of these is particularly unusual, we were struck by the fact that all three represent very public expressions of interest in the education policy process. These incidents highlight the public standing of education in Western society; everyone seems to care about education and everyone has an opinion – from parents to shopkeepers, and yes, to symphony conductors. But, how do policymakers negotiate these often divergent and competing representations? In Canada’s global-competitive environment the ‘performance’ of the education system seems to have become a standard against which special interests and the public measure the effectiveness of government in addressing public concerns relating to regional and national economic downturns, competitiveness and community/regional sustainability. These arrangements draw attention to education policy making as a high-stakes undertaking. However, the question of what policy makers consider to be valid evidence for policy development has not been fully examined and questions continue to be raised about the extent to which government policy is grounded in research-based knowledge. In this article we begin with a literature review on research evidence and policy/decision- making in education, focusing on British, Australian and Canadian contexts. In the second section we describe the methodologies from two pan-Canadian studies that examine the use of research and other evidence in the policy formation practices of two groups of education policy elites, situated in different contexts. The first study examined the use of research and evidence by ministry level decision makers: education ministers and senior ministry bureaucrats (Galway, 2006). The second Education Policy Analysis Archives Vol. 23 No. 109 4 study considered the factors and evidence used by decision makers (school board trustees and superintendents) at the school district level (Sheppard, Galway, Brown & Wiens, 2013). In the last section we discuss and compare the findings and, following on earlier work in this area, offer suggestions for (1) improving the integration of research into the policy practices of education decision makers, and (2) strengthening the intersection of researchers, education practitioners, and policy makers. Research Evidence and Education Policy Since the 1970s there has been considerable focus on the impact of research on the production of social, health and educational policy (e.g., Caplan 1979; Cooper, Levin & Campbell, 2009; Cooper, 2013, 2014; Edwards, Sebba & Rickinson, 2007; Honig & Coburn, 2008; James & Pollard, 2011; Landry, Amara & Lamari, 2001; Lavis, Lomas, Hamid & Sewankambo, 2006; Levin & Riffle, 1997; Lindblom, 1980; Lindblom & Woodhouse, 1993; Majone, 1989; Martin, Downe, Grace & Nutley, 2010; Moat, Lavis, Wilson, Rottingen & Bärnighausen, 2013; Nutley, Jung & Walter, 2008; Nutley, Walter & Davis, 2003, 2009; Pollard, 2007, 2008; Qi & Levin, 2013; Rosenblatt & Tseng, 2010; Stone, 2002; Strike, 2006; Tseng & Seidman, 2007; Weiss, 1979; Yohalem & Tseng, 2015). While many social science researchers perceive their raison d’être to be connected to the development of sound public policy decisions, policy-making in education is seen to be somewhat resistant to the influence of research-based knowledge and, driven by other, more immediate political and pragmatic factors (Davis, 1999; Honig & Coburn, 2008; Ince, 2008; Levin, 2004a; Reimer & McGinn, 1997). The problem of the so-called research-policy divide has been approached from several perspectives and while it has become widely accepted that educational research has improved in volume and quality, difficulties with effective dissemination and integration into practice and underutilization by policy elites remain vexed problems (Cooper, 2013, 2014; Honig & Coburn, 2008; Qi & Levin, 2013; Yohalem & Tseng, 2015). Policy theorists have long argued that decisions about education should be tied more closely to research evidence. There is a growing push for ministers and other public sector policy makers to refer more of their policy decisions to independent research and these calls have been growing louder in recent years (DeBroucker & Sweetman, 2001; Fitz-Gibbon, 2003; Honig & Coburn, 2008; Naughton, 2005; Slavin, 2002, 2004; Thomas & Pring, 2004; Zussman, 2003). The challenge has been to effectively convey this idea to politicians, the media and the public, who often seem to base decisions on their own, different forms of evidence – popular sentiment or their own experiences and perceptions – rather than on research evidence (Levin, 2004b). Some researchers have accused policymakers of taking their lead from local pressures and political and practical considerations such as public opinion, conventional professional wisdom and media reports (e.g., Black, 2001; Cooper & Levin, 2010; Kwok, 2003; Leicester, 1999; Levin, 2003a; Ungerleider, 2003; Weissberg, 2002). Others have mentioned idiosyncratic factors such as dependencies, loyalties, associations, self-interest, personal values, beliefs, experiences, biases and fears (e.g., Honig & Coburn; 2007; Reimer & McGinn, 1997; Stone, 2002). Reimer and McGinn (1997, p. xiv) describe some of the problems they experienced in their policy work with senior education decision-makers in the international context: We started as researchers, eager to get developing countries to improve their education systems. We were convinced that our skills in research and analyzing data would make an important difference. To our great dismay and chagrin, we found it difficult to get decision makers to listen to us. When they did listen, they often were critical of what we had to say and seldom followed our advice, it seemed. […] Almost all their comments were negative, denying the validity and usefulness of the Research and evidence in education decision -making 5 reports. … [T]hey drew up recommendations for action, many of which directly contradicted what we had recommended. While many in the academy will identify with the frustrations expressed in this excerpt, it may be a little naive for researchers to expect policymakers to abandon their own perceptions and agendas in favour of a strictly linear technocratic approach to policy development. The seminal work of policy researchers such as Weiss (1979) and Lindblom (1980) raised questions about the direct or ‘instrumental’ influence of research-based evidence on policy, claiming instead that research utilization is indirect, long term and circuitous. As Levin (2001) comments, “[t]he political world is …shaped by beliefs more than facts” (p. 14). A number of scholars have observed that in cases where research does impact policy, the effect normally takes place after an extended period – a phenomenon described by Weiss (1979) as the ‘enlightenment’ function of research. Much of the critique of so-called “evidence-free” policy making can be theorized through the notion of a chasm between the academic and political/policy contexts. Nathan Caplan’s (1979) early work on “two communities” theory suggested that researchers and policymakers operate in entirely different contexts with “different and often conflicting values, different reward systems and different languages” (p.459) Caplan (1979, p. 468) argued that the coupling of existing knowledge to user needs requires collaborative arrangements whereby the researchers and policy makers come to an understanding on “which aspects of a [policy] problem are to be decided on the basis of data-based knowledge and which ones are to be decided on the basis of non-research knowledge.” Because university-based researchers operate according to their own, different accountability and reward systems, this has typically required that they follow their own research agendas while maintaining an appropriate level of independence from the government policy environment – even publicly criticizing government policy in some circumstances. As Levin (2004b) notes, “university-based research is driven by the university culture and reward system; it is primarily aimed at communication with other scholars and the rewards are related to research grants and peer recognition” (p. 9). Moreover, the engagement of academics in research on specific policy problems, as in the case of commissioned research brings its own set of problems including serious ethical dilemmas relating to authenticity and independence. Strike (2006) observes that the relationship between government and researchers is “fraught with moral peril” (p. 67). On one hand, there is the hope that researchers can advance social progress when they serve the needs of policy makers. On the other hand, commissioned studies are not subject to the same rigorous review standards as independent research: The research done at the behest of policymakers is often not refereed. It is often “quick and dirty” – done under time pressures and poorly funded. Moreover, policymakers often ask researchers to evaluate or assist in the implementation of policies and programs in which the policymakers have a vested interest. They may be uninterested in funding research that might fail to support favoured programs. The questions may be structured so as to assume the merits of current policy and policy aspirations. Most troubling is that, at the state level, those policymakers who fund research often have influence over the budget and reputation of the public universities in their states. Researchers may fear offending public officials, lest their access to funds and influence dry up. University administrators may be reluctant to have their faculties assume a critical or oppositional role, and policymakers are likely to look for sympathetic researchers. (Strike, 2006, p. 67) Government actors, for their part, have charged researchers with ignoring important policy areas in education (Blunkett, 2000; Galway, 2006). In the past, some critics have suggested that, with the Education Policy Analysis Archives Vol. 23 No. 109 6 possible exception of commissioned research, much of the independently-produced university research may be inaccessible or does not provide the answers decision makers are looking for when deciding among policy options (Cooper & Levin, 2010; Lindblom & Woodhouse, 1993; Neilson, 2001; Pring, 2000; Stone, 2002). In the mid-1990s an intense debate began in the UK around the quality and utilization of educational research. The ruckus was propelled by an article in the Times Education Supplement, wherein Alan Smithers (1995) argued that the only criterion for judging the value of educational research should be the extent of its usefulness to policy makers and practitioners (Dunseath, 2000). Hargreaves (1996, p. 7) later declared that education research was ‘second-rate’ and decried its inability to “make a serious contribution to fundamental theory or knowledge,” charging instead that it cluttered up obscure academic journals, was irrelevant to practice and was uncoordinated with antecedent research. Two controversial British reviews of the quality and utilization of educational research followed. Tooley and Darby (1998) evaluated the nature and quality of educational research in Britain; based on a sampling of articles published in four British educational journals, the researchers concluded that only about one third of the articles could be said to have been based on sound research practice. Citing weaknesses in methodology, policy relevance, objectivity and quality of argument, the authors suggested that education research must become non- partisan, pragmatic and relevant in its design, include a plan for dissemination, and be guided by an overall research strategy. The Tooley and Darby study was followed closely by a second review by Hillage, Pearson, Anderson and Tamkin (1998). Hillage and his colleagues put forward the widely contested view that the research-policy divide was due, in large part to an education research agenda that was ‘supply- driven’; that is, the research originates with researchers and not policymakers and practitioners, a model described by Landry and his colleagues as the ‘science push model’ (Landry et al., 2001). They argued further that education research tended to be backward looking and evaluative, rather than forward-looking and exploratory, resulting in negligible policy impact. This idea overlaps with the notion that education research lacked a ‘cumulative character;’ whereby new studies help to construct a knowledge base by building effectively on earlier work (Davies 1999; Hammersley, 1998; Ince, 2008). While some of the observations in the Tooley and Darby and Hillage reports have been without serious challenge several have been contested. Hodgkinson (1998) criticized Tooley and Darby (1998) for making a fundamental distinction between empirical research and non-empirical research and for evaluating qualitative studies by predominantly quantitative, positivist-realist criteria. He further charged that the review ignored basic ontological and epistemological precepts of research conducted in post-industrial traditions; for example, the fact that many qualitative researchers do not claim that their work is generalizable in a statistical sense, nor do they claim objectivity. Several reviews describing the state of education research in Australia have been less critical of the research community. Bates (2001) undertook a meta-analysis of research reviews in Great Britain and Australia over the preceding ten-year period. He concluded that the Australian reviews, in comparison to those conducted in Great Britain, tended to deliver much more positive assessments of the value of education research, both in terms of its international reputation and its use among teachers and policymakers. Ribbins, Bates and Gunter (2003) account for this difference, in part, by the nature of the reviews. While the Australian work has tended to begin with educational practice to inform research, the UK reviews have tended to begin with the research and work forward to practice. Between 1999 and 2000 five reviews focusing on research and practice in the Australian context were undertaken as part of an overall initiative sponsored by the Australian Department of Education, Training and Youth Affairs (DETYA) and the Australian Research Council. The first Research and evidence in education decision -making 7 concluded that much of the Australian research base was devoted to issues of teaching and learning (Holbrook, Ainley, Bourke, Owen, McKenzie, Mission & Johnson, 2000). Connections between university researchers and the educational community were considered to be robust and included ministry-faculty research committees, the publication of research in teacher-focused publications and the secondment of academics to ministry staff positions. There was additional evidence, based on claims by practitioners, that findings from education research had permeated practice, although the review acknowledged that research utilization was often mediated through individual policy advocates. In a second study, Figgis, Zubrick, Butorac and Anderson (2000) examined the impact of research on policy, starting at the policy/program level and working backwards to determine whether selected policies/programs were founded on a research base. Their work put forward the notion that practitioners and policymakers were connected to research through ‘connecting webs’ or networks with multiple nodes. They suggested that the application of research in an educational context requires human intervention of motivated actors and the active marketing of research knowledge by its producers. A third study examined teacher practice to determine the extent to which it was reflective of a research base and concluded that research was, in fact, a substantial influence among several ‘proximal’ sources of influence on teaching practice (McMeniman, Cumming, Wilson, Stevenson & Sim, 2000). The fourth study examined citation rates of Australian researchers in international journals (Phelan, Anderson & Bourke, 2000). This bibliometric analysis demonstrated that Australian research was extensively cited in a range of knowledge domains including pedagogy, administration and policy development. A fifth study examined the influence of research on policy decision-making within the vocational education and training system in Australia (Selby-Smith, 1999). Of the senior education decision makers surveyed, two thirds claimed that strategic and political considerations were most influential in decision-making while research was often used to justify or support a decision already taken, variously described as the symbolic use of research (Landry et al., 2001), use of research to mobilize support (Nutley et al., 2003; Weiss, 1998) and use of research to build political support, (Honig & Coburn, 2008). Selby Smith (1999) made several conclusions with respect to the authenticity of research for policy decision-making in education. First, research does impact decision-making, but it exerts its influence on the broader long-term education agenda – extending to the so-called ‘big ideas’ in education. He suggested that researchers be modest in their expectations about the impact of research on decision-making noting that “research is only one of a number of sources of information available to decision-makers and information from all sources is only one of many inputs into the decision-making process” (p. 556). Second, because decision makers bring their own experiences and backgrounds to bear on their practice, “the extent of research impact is affected by the knowledge, attitudes and experience of decision-makers” (p. 565). Finally, to be influential, research must be conducted in an environment of collaboration where robust linkages exist between decision makers and researchers at all stages of the research process. An “emphasis on linkages rather than dissemination alone increases the mutual responsibility of the parties, since enduring linkages are based on sustained mutual esteem, and understanding of the potential contribution of each party, and a commitment to collaboration for the good of the system” (p. 565). Prior to the mid-2000s there were relatively few references to the quality of educational research within the Canadian context. Most of the discussion is framed around the scope of the research effort, which was generally found to be somewhat lacking. Levin (2003a), for example, acknowledged pockets of strength, but claimed that the educational research enterprise generally, is small, short-term, narrow and fragmented. Although the research effort within faculties of education was growing, he concluded that there were few organized mechanisms to enable education research to be translated into knowledge for policy. Applied research in Canada, according to Levin (ibid.) had remained within the purview of government departments and third parties, such as privately funded Education Policy Analysis Archives Vol. 23 No. 109 8 institutes and think tanks. Willinsky (2003) similarly observed that the influence of academic research in Canada “pales before the impact of the government’s own sources of information, whether from politicians, bureaucrats, or government professionals” (p. 2). He noted, however, that the availability of academic articles through electronic means has facilitated research access among policy advisors and decision makers and this has acted as a “counterforce to policymakers’ reliance on a small number of academic consultants as gatekeepers and sources for research” (p. 1). According to Landry, et al., (2001) part of the problem was that the body of work on research utilization was disparate – scattered across different social science disciplines and recorded in disconnected journals. There were also problems with narrow and overly prescriptive definitions of research utilization which had come to be associated with direct instrumental or linear uses of research rather than conceptualizing research use as a broader, more interactive and circuitous process. In 2003, deBroucker and Sweetman compiled a series of papers by well-known Canadian educators and researchers with the goal of examining and evaluating the impact of Canadian education research impact to date. The authors drew three conclusions about the state of education research as policy evidence. First, compared to the health-care sector in Canada, education research is used to a far lesser extent; second, there is a pressing need to involve a broader set of people than researchers in considering research issues, and finally, there is a need to expand the indicator set to include a broader range of measures, including those indicators that are more difficult to measure. Shortly thereafter the then federal (Liberal) government established the Canadian Council on Learning (CCL) as an independent national agency with a mandate to produce, promote and disseminate educational research (Mason, 2012). Before funding was cut by a successor government in 2009, the CCL had made good progress in drawing attention to several key policy issues in education by completing systematic reviews and disseminating the research findings through a number of channels, most notably, an on-line repository of thematic reviews, research and data. Cooper et al. (2009) acknowledge a strengthening of capacity and cite several provincial initiatives aimed at increasing support for evidence-based policy and practice, but point out that integrated pan- Canadian strategies are more difficult because education in Canada is not a federal responsibility. From the perspective of raising awareness of research in education, the fallout from these reviews and from the discussions and debates that followed was both positive and progressive. More recent work in this field has left behind the divisive debates around research quality and attempted to focus on defining and structuring the field of evidence-informed practice as a research area (Cooper et al., 2009; Nutley et al., 2003) defining and promoting high quality and highly relevant educational research (Earl, 2011; James & Pollard, 2008; Pollard, 2007, 2008) and understanding the different forms and trajectories of research use both in education and other public sectors (Nutley et al., 2008; Rosenblatt & Tseng, 2010). A major advance in promoting this agenda in the UK was the Teaching and Learning Research Program (TLRC), a large-scale effort to spearhead a series of projects and thematic analyses intended to engage users in research-informed practice both to improve research quality and generate new knowledge (James & Pollard, 2011; Pollard, 2011). Pollard (2007) remarked that the program was intended to support high quality and highly relevant research that could “satisfy the ambition of ‘Pasteur’s Quadrant’” (p. 640), a reference to Stokes (1997) categorization describing a form of research that marries pure basic research to applied research. The ten-year £40 million program, which concluded in 2011, generated a series of ten principles of effective pedagogy in four broad areas and is widely regarded to have been successful in translating academic research into user- friendly forms and engaging users in culturally and politically current educational discourse (Earl, 2011; Galvin, 2008; Hogan, 2011; James & Pollard, 2011; Pollard, 2008). Other researchers have been working on understanding the different forms and pathways of research use. While those working in this field have argued about different perspectives on research Research and evidence in education decision -making 9 use (for example, rationalist-decisionist/rational linear versus more ambiguous and interactive approaches) there is some agreement that there is a diversity of avenues through which research permeates and influences educational practice and pedagogy (Nutley et al., 2008). Landry et al. (2001) examined the utilization of social science research in Canada across four disciplines and conceptualized the process as cumulative rather than instrumental; dependant on the engagement of researchers with end users rather than the quality of the research products. Drawing on research in the social care field and cross-sector reviews, Nutley and her colleagues describe several models of research use, noting that both rational-linear and interactive perspectives could be found operating in each. This suggests that there is value in conceptualizing such models and perspectives as iterative and overlapping; their use being appropriate at different times and in different contexts (Nutley et al, 2008; Nutley et al., 2009). A number of recent articles have begun to concentrate on research utilization as ‘evidence into practice’ with emphasis on the intersection of researchers and practitioners (Earl, 2011; Edwards et al., 2007; Nutley et al., 2003) including linkage mechanisms, research mediation efforts and new dissemination strategies (Cooper, 2013; Landry et al., 2001; Martin et al., 2010; Qi & Levin, 2013). We have already mentioned the work of Landry and his colleagues. Based on their study of more than 1200 social science researchers in Canada, they concluded that the most important determinants of utilization are “the mechanisms linking the researchers to the users, the users’ context and the publication assets of the researchers” (p. 334). Edwards et al. (2007) similarly found that establishing reciprocal relationships with users as co-researchers could enhance practitioner value for research. As the field has grown and expanded from health to education and to other social sectors new terminology has come into usage: knowledge transfer, evidence-informed policy, data-driven decision-making and knowledge brokering, to name a few. Knowledge mobilization (KM) has been touted as a useful all-encompassing term because it conveys the notion of direction instead of random interaction and it “embodies the idea that the use of knowledge is a social process, not just an intellectual task” (Cooper et al., 2009, p.166). The more recent North American work on education research utilization/research impact has framed the connection of research and policy environments in terms of KM and knowledge brokering with governments, schools and school districts, and community organizations (Cooper, 2013, 2014; Cooper & Levin, 2010; Cooper et al., 2009; Jbilou, Amara & Landry, 2007; Knowles et al., 2013; Qi & Levin, 2013, Wilson, Lavis, Traverse & Rourke, 2010; Yohalem & Tseng, 2015). Honig and Coburn (2008) were interested in which forms of evidence, if any, were used in school district offices and the purposes for which the evidence was used. They conducted a comprehensive review of relevant research literature but found a large proportion of the material in circulation were advocacy pieces leaving only about fifty peer-reviewed publications. The major findings suggest that the process of evidence use is complex, highly political and based largely on local knowledge. Along the same vein, Cooper and Levin (2010) surveyed a sample of superintendents, principals and other leaders to study how research use was perceived, supported and integrated into the operations and activities of school districts and schools across Canada. They found a strong interest in research use, but a weak and loosely coupled institutional research capacity that depended on “a few interested people rather than being deeply embedded in daily practice” (p. 61). Since the early 2000s, there has been a proliferation of interest in KM including more publications, new networks of researchers and practitioners, and more outlets and spaces for interchange on evidence and practice, but also a number of lingering problems (Cooper et al., 2009; Qi & Levin, 2013). Notwithstanding the contribution of Nutley et al. (2003) in suggesting a framework for organizing the evidence-into-practice research agenda, Cooper and her colleagues Education Policy Analysis Archives Vol. 23 No. 109 10 complain that that conceptually and theoretically, the field is still in disarray, there are too few empirical studies on most of the issues, and efforts to increase research impact have not been properly assessed (Cooper et al., 2009). Some of the Canadian work by this group and others at Queen’s University and the University of Toronto is attempting to address some of these problems. In a recent study of research mediation in Canada, Cooper (2013) takes on the unenviable task of disentangling some of the cross-sectorial confusion around KM terminology and knowledge brokering models, hypothesizing about how research brokering organizations (RBOs) are situated in the education sector and creating a typology of Canadian RBOs (e.g., the Atlantic Institute for Market Studies and the Fraser Institute) in education. RBOs are grouped into four categories – government, not-for-profit, for-profit and member RBOs – and are defined by their orientation along seven common dimensions. This particular work is an important step towards redressing some of the conceptual problems in defining the field of KM raised by Cooper and her colleagues in their earlier work (Cooper et al., 2009). In an effort to assess KM efforts among knowledge producing organizations, Qi and Levin (2009) studied websites from 100 institutions and agencies (including governments, universities and faculties of education, and other organizations) in Canada and other English-speaking countries. The researchers judged the research sharing effort to be low to modest, product-oriented and passive (unidirectional) with no opportunity for two-way interaction. Cooper (2014) similarly looked at on-line strategies and social media as a linkage mechanism between producers and users of research. The websites of forty-four Canadian RBOs were examined to assess their on-line and social media dissemination strategies. The social media effort was considered to be modest – mostly small-scale Facebook pages and Twitter networks – and there were few avenues for two-way interchange. The foregoing discussion highlights the thorny questions of what constitutes evidence for education policy and how education research becomes introduced into the policy process. Researchers and social policy critics still have divergent and incomplete conceptualizations about the relationship between research and practice. While this is an intriguing area, it has not been fully interrogated, the literature is still somewhat disconnected, and there are gaps in our knowledge base. As Cooper et al. (2009) have observed, after all of the activity related to KM “very little is known about how practice organizations, whether governments or schools and school systems, find, share and use research.” The analysis presented in this paper attempts to address some of these blank spots by addressing three interconnected research questions: (1) what are the factors and influences that drive policy decision-making in Canadian education ministries as compared with school boards, (2) how important is each category of evidence in relation to the others, and (3) to what extent is education research valued as ‘authentic’ policy evidence? Research Design The 2006 Study The 2006 study was a two-phase mixed-methods design with two participant groups: Canadian provincial deputy ministers and equivalents (survey/questionnaires) and current or former provincial ministers of education (in-depth, semi-structured interviews). Survey method. A population of 95 senior education public servants was identified using the Canadian Education Association Handbook and contact lists provided by the Council of Ministers of Education, Canada (CMEC) and provincial departments of education. A questionnaire developed for this research was field tested and administered to deputy ministers and assistant deputy ministers of education and their equivalents working in Canadian ministries of education using direct and electronic mail. A total of 51 valid questionnaires were returned for a response rate Research and evidence in education decision -making 11 of 53 percent (Quebec was not represented). Demographic information such as age, gender, educational background and years of experience in public administration was also collected. In Phase II, in-depth, semi-structured interviews were conducted with eight (8) participants – five (5) current and three (3) former provincial ministers of education. Participants were selected and contacted based on language (English-speaking), probable access, and proximity to the researcher. Participants were asked to recall three instances of education policy decisions or policy advice they had provided to ministers during the previous 12 months (N=148). They were then asked to rate, on a seven point Likert scale, the extent to which each of a series of factors influenced the corresponding decision or recommendation. They were then asked to recall examples of research or indicators they had recently used in their work. Data analysis of numeric and categorical data involved the generation of means and standard deviations for Likert scale items and frequency distributions for categorical data, in accordance with generally established principles and consistent with those procedures outlined by Dillman (1999), Fowler (1998), and Henerson, Morris and Fitz-Gibbon (1987). Interview method. Interview questions focused on participant representations of research and evidence in education decision-making, in particular (1) how they conceptualize education research; (2) the extent to which research and other forms of evidence are utilized in decision-making; and, (3) perceptions of accessibility, relevance, utilization and value of education research. The interview protocol followed the proce dures described by Bok van Kammen and Stouthamer-Loeber (1998), Mangione (1998), Marshall and Rossman (1989), and Maxwell (1998). All, except one telephone interview, were face-to-face, private interviews and all were recorded and transcribed. Interviews ranged in length from approximately 60 minutes to more than two hours with an average interview length of approximately 90 minutes. The analysis of interview data was modeled after the principles and strategies for interview data described by Creswell (1998) and Quinn Patton (1987). First, there was a general review of the printed transcripts of each of the interviews and the combined interview data was grouped according to question. The transcripts were then analyzed for identifying themes, findings and relationships among the data, and notes are recorded in the margins of these texts. The second step of the analysis process involved examining the transcripts, and over a period of time, thinking about the data sets, and identifying coherent and important examples, themes and patterns in the data, a procedure described by Quinn Patton (1987) as content analysis. These ideas and relationships were further developed in the form of longer notes and memos that were sorted into categories and themes, forming the preliminary findings. As the organization of findings and themes matured, the third step of the process involved selecting samples of text from the interviews that were illustrative and confirmatory of the finding or theme. In cases where the same idea was expressed repeatedly, participant representations were summarized and paraphrased or used as exemplars of the consensus of the group. The final data reduction step involved creating displays of information using diagrams showing interconnections among ideas and findings while identifying larger global themes running through the representations of the participant groups. Finally the interview transcriptions were re- read carefully to validate or discard these preliminary findings. Following from Silverman’s (2001) discussion on avoiding anecdotalism, the data were searched for negative exemplars in the data that could be argued to falsify or refute the findings and the categories, themes and findings were revised accordingly. Education Policy Analysis Archives Vol. 23 No. 109 12 The 2012 Study For the 2012 study we collected data through the use of a survey/questionnaire and a series of focus group interviews. There were three participant groups representing Anglophone school boards in all Canadian provinces and territories: (1) school board trustees, (2) school district superintendents, and (3) executive directors of each provincial school boards association who were also members of the Canadian School Boards Association executive board. Participants. The selection of participants for both the survey/questionnaire and interviews was conducted with sensitivity to gender, experience, ethnicity and regional geography. A total of 369 individuals from across Canada (331 school board trustees and 38 district superintendents) completed questionnaires. As well, we cond ucted 21 focus group sessions with school board trustees, school district superintendents, and members of the Canadian School Boards Association executive board. Survey method. The survey/questionnaire was modelled after the one used in the 2006 study, with some variations. Prior to the distribution of the questionnaire, it was pilot tested with a convenience sample of six individuals who had extensive experience as either a superintendent of education or trustee. The research protocol was also presented a t a session with interested members of the Canadian Association for the Study of Educational Administration (CASEA) at its 2010 Congress. Minor adjustments to the research instruments were made on the basis of the resulting feedback from these validation e xercises. Each participant was asked to identify specific policy-relevant decisions or recommendations that s/he made during the past year. They were then asked to rate, on a seven point Likert scale, the extent to which each of a series of factors influenced the corresponding decision or recommendation. Our analysis of the survey data is limited to descriptive measures such as frequency counts and the calculation of mean scores and standard deviations. For the survey analysis, because of low return rates among school district superintendents, it was not possible to report separately for school board trustees and superintendents. Data from school board trustees and district superintendents are pooled. Focus group method. The focus group protocol, and analysis of the focus group data, followed the procedures described by Meijer, Verloop, and Beijaard (2002), Merriam (2009), and Miles and Huberman (1994). Focus group sessions were conducted separately for trustees and superintendents. To increase the validity of findings, we took considerable care to collect data from school board members and school district superintendents from all provinces and territories and from those holding office with the CSBA Board. We also triangulated data collection such that each of research team members conducted a number of individual 60-90 minute focus group sessions (5 to 10 participants per session) that occurred at different times and in differing locations over the eighteen-month data collection period. Each focus group session was recorded and transcribed. To ensure anonymity of each participant we assigned pseudonyms as required. We analyzed the transcribed focus group data through the application of a qualitative data analysis program (QDA Miner, 2011). Using this program, we coded separate data segments for school board trustees, executive directors of provincial school board associations and school district superintendents of education. Subsequently, we retrieved these coded segments and saved them as separate reports that facilitated the development of specific themes. Also, we were able to develop frequency counts and the percentage of respondents who provided particular responses. It is important to note, however, that all focus group participants did not respond to each question. As a consequence, our reporting that a specific percentage of the focus group participants held a particular view, for instance, should Research and evidence in education decision -making 13 not be interpreted that a remaining percentage held a contrary view; if no percentage is reported for the contrary view, it means the others were silent on the issue. Limitations. Although we employed a Likert scale instrument in addition to focus group interviews to collect data, this work is primarily dependent on naturalistic and interpretive methodologies (Cohen, Manion & Morrison, 2011) that acknowledge personal involvement of the researchers and anticipates subjectivity. Data from interviews are actively constituted by the intervention of the researcher with research participants. They would otherwise not exi st. From a theoretical perspective the responses of research participants in this study are seen more as representations of perspectives and ‘moral forms’ than simple reports on reality (Silverman, 2001). Given the pan-Canadian scope of this study and holding realistic expectations of how much time we could reasonably expect of study participants at any one point in time, we utilized a questionnaire/survey as a means of collecting individual participant perspectives relating to his or her role as either a trustee or superintendent. Our analysis of the questionnaire/survey data is entirely limited to the determination of frequency counts and mean scores. Accordingly, we consider this work to be solely naturalistic and interpretive . Ethical review. The research protocol for both studies was approved in advance by a university interdisciplinary committee on ethics in human research and followed the principles outlined in the Tri-Council Policy Statement on Ethical Conduct for Research involving Humans (TCPS2). Research Findings The 2006 Study Profile of participants: Senior bureaucrats. The data show that senior executive positions in Canadian education ministries are reserved for highly educated, mature individuals with extensive experience in education or public administration, but not necessarily within schools, school boards or other educational institutions outside of government. Deputy and assistant deputy minister positions tend to be male-dominated – females accounted for less than 40% of education executives – and are occupied by mid to late career bureaucrats, who have completed more than one advanced degree. As a group, senior bureaucrats have a broad range of administrative experience both within and outside of government, yet are generally new to their current positions as education executives. More than half the senior officials surveyed were in their current jobs for three or fewer years (Table 1). Table 1 Senior Bureaucrats’ Years of Experience in Present Position and in any Administrative Position (N=51) Years Experience Present Position Percent Any Admin. Position Percent 0-3 years 26 51.0 3 5.9 4-6 years 12 23.5 8 15.7 7-9 years 4 7.8 4 7.8 10-12 years 5 9.8 9 17.6 More than 12 years 3 5.9 26 51.0 Not reported 1 2.0 1 2.0 Total 51 100 51 100 Education Policy Analysis Archives Vol. 23 No. 109 14 Categories of policy issues. We first sought to understand the general categories of education policy issues addressed by policy actors at the government (ministry of education) level. Senior education executives were asked to describe three policy -relevant decisions or recommendations they recall having made during the previous 12 months 1 and to rate the significance of each policy decision or recommendation on a seven point Likert scale. Data describing ‘perceived significance’ are means and standard deviations where +3 represented ‘very significant,’ 0 was neutral, and -3 represented ‘not at all significant. Participants reported a total of 148 policy decisions or recommendations that could be assigned to 16 broad policy categories2. Results showing the frequency cited and perceived significance, by policy category , are presented in Table 2. Senior education bureaucrats reported their primary focus was on corporate matters – educational finance and resourcing policy, accountability policy and strategic planning. These three areas accounted for almost half of the policy decisions/recommendations recalled. The areas least cited were: school bussing, professional development, special education, social development and equity and education research, together comprising less than seven percent of all policy decision and/or recommendations reported. Program and policy issues, such as literacy and post-secondary education, aboriginal education, as well as finance and resource issues were judged to be the most significant. Also rated as highly significant were a series of related issues – programming, curriculum, instruction and assessment, as well as educational planning, governance and accountability. Table 2 Categories, Frequency and Perceived Significance of Policy Decisions Recalled by Senior Bureaucrats (N=148) Policy Category Frequency Cited Percent Perceived Significance Standard Deviation Finance/Education Resources 41 28.4 2.24 0.81 Accountability, Testing and Assessment 15 10.1 1.80 1.15 Post-secondary Programming 15 10.1 2.07 0.80 Educational (Strategic) Planning 14 9.5 2.00 0.96 K-12 Curriculum/Instruction 11 7.4 2.00 1.10 Career/Labour Market Development 10 6.8 1.80 1.14 Literacy / Adult Education Policy 9 6.1 2.44 1.01 Education Governance 8 5.4 1.87 0.83 Francophone Education/Governance 5 3.4 1.80 0.45 Aboriginal Policy 5 3.4 2.20 0.83 Education Infrastructure 4 2.7 1.50 0.90 Education Research 3 2.0 1.67 1.53 Social Development and Equity 3 2.0 -1.00 0.00 Special Education 2 1.4 2.00 0.00 Professional Development 1 0.7 2.00 0.00 School Bussing 1 0.7 2.00 0.00 Total 148 100.0 1.99 0.94 1 Or since they had been in their current position, if it had been less than 12 months. 2 In several instances some of the data were combined to form broader categories of factors/influences. These are footnoted where they appear in the presentation of results. Research and evidence in education decision -making 15 Evidence and policy practice: Survey data. Data presented in Table 3 were generated from participant representations of their recent policymaking practice. Again, senior public servants were asked to describe three policy-relevant decisions or recommendations that they recall having made during the previous 12 months,3 and in the context of a given list of factors/evidence, they were asked to indicate the relative influence of each of the factors on each decision or recommendation. Data were pooled and, for each factor, expressed as the mean plus or minus the standard deviation where -3 represented “did not at all influence”, 0 was neutral and 3 represented “influenced to a great extent”. Findings reveal that greatest importance was placed on internal advice provided by staff and colleagues, followed by past experience, personal and professional beliefs, corporate/pragmatic evidence4, and political-democratic considerations5. Within the broad category of political considerations the data show that the representations of special interest groups was a moderately important consideration while claims about the influence of public opinion and the representations of the mass media were less definitive. Among the factors that were rejected by senior bureaucrats as having any substantial impact on policy development practice were anecdotal stories, the representations of individuals, and externally produced research. Research generally, and particularly, external research studies were considered to be relatively minor considerations as evidence for policy decisions or recommendations, although it is highly probable that at least some reference to research would be incorporated in some forms of staff advice. Participants were then asked to list the factors or evidence that should be considered in developing education policy. The results generated a list of 151 suggested factors covering a wide range of potential forms of evidence; these were assigned to one of 20 response categories we developed during the course of data analysis. Senior bureaucrats placed highest value on corporate/pragmatic evidence – including financial considerations, efficiency and policy implementation factors, and the direction of government’s strategic plan – together comprising more than 42 percent of the factors cited. In order of descending frequency, the following categories of evidence were listed as important to policy decision-making: (a) financial considerations/ efficiency/capacity to implement (21%); (b) government/department’s strategic goals/direction (21%); (c) education research or indicators (15%); (d) impact on students/clients (13%); (e) potential for social and economic development of the province (9%), and; (f) representations/pressure from special interest/stakeholder groups (5%). Among the categories of evidence least cited were (a) professional and personal knowledge, values and beliefs; (b) government/minister’s reputation; (c) equity and fairness considerations; (e) linguistic and cultural considerations; (f) avoidance of negative media attention; (g) jurisdictional considerations; and, (h) the minister’s direction, together comprising about eight percent (8%) of factors/evidence cited. 3 Or since they had been in their current position, if it had been less than 12 months 4 Corporate/ pragmatic influences are defined as practical, ideological, strategic and financial considerations. This cluster includes: department’s strategic plan; budget considerations; cabinet/government’s overall direction, and; potential to directly contribute to economic development of province. 5 Political-democratic considerations are defined as those factors that relate to the status, image or reputation of the government or minister. This cluster includes: potential to advance the reputation of the minister/government; minister’s particular interest of direction; representations of special interest and advocacy groups; public opinion, and; the representations of the mass media. Education Policy Analysis Archives Vol. 23 No. 109 16 Table 3 Senior Bureaucrats’ Perceptions of the Relative Influence of Different Factors/Evidence on their Recent Policy-relevant Decisions/Recommendations (N=148) Factor/Evidence Mean Standard Deviation Advice of staff and/or colleagues 1.80 1.29 Past experience with this issue or with a related issue 1.34 1.59 Personal or professional beliefs and values 1.16 1.53 Department’s strategic plan 1.15 1.56 Budget considerations 0.86 1.92 Research or indicators prepared in-house by department staff 0.86 1.70 Cabinet/Government’s overall direction 0.73 1.69 Potential to advance the reputation of the Minister/government 0.72 1.64 Minister’s particular interest or direction 0.59 1.79 Representations of other (non-business) groups 0.22 1.95 Public opinion -0.13 1.75 Potential to directly contribute to economic development of Province -0.23 2.16 Provincial government commissioned research or assessments -0.37 2.20 Avoidance of negative media attention -0.54 1.96 Representations of individuals (students, parents, educators) -0.57 2.07 All Research6 -0.82 2.04 Representations of business groups or the private sector -1.19 1.82 External Research Studies7 -1.36 1.88 Situation or event someone told you about outside your professional environment -1.65 1.73 Finally, participants were asked to provide some examples of research or indicators that they remembered or had used in their policy development practice. The findings presented in Table 4 suggest that senior public servants value survey research and large-scale assessments over qualitative research. The vast majority of examples of research and indicators that they remember or have used in policy development are large-scale quantitative studies such as provincial, national and international assessments, educational finance studies, demographic research and labour market outcome studies. 6 All Research includes: University Research (Education); University Research (other than Education); Research studies by other agencies; CMEC Research and Assessments; Federal Government Research; Research or indicators prepared in-house by department staff; and, Provincial Government- commissioned research or assessments 7 The following categories were collapsed to form the External Research category: Research studies by other agencies (Conference Board, Fraser Institute, etc.); University Research (Education); University Research (other than Education); CMEC Research and Assessments; and, Federal Government Research Research and evidence in education decision -making 17 Table 4 Categories of Research/Indicators Senior Bureaucrats Recall or Have Used in Policy Development (N=51) Description Frequency Percent Provincial/national/international achievement and literacy assessments 15 29.1 Educational finance studies 9 11.4 Demographic studies (e.g., enrolments, capacity) 8 10.1 Government-commissioned research 8 10.1 Post-secondary labour market outcomes 7 8.9 Participation/graduation indicators 6 7.6 Social/financial Indicators (e.g., health/ income support/ employment) 5 6.3 Aboriginal achievement and attainment indicators 4 5.1 Minority language indicators 3 3.8 OECD research/indicators 2 2.5 Other external research 2 2.5 Busing studies 1 1.3 School or school district-based research 1 1.3 Total 79 100 Profile of participants: Ministers of Education. The participant group of ministers of education comprised two females and six males, all over 45 years of age and all veteran politicians, each having served more than two terms in office. In general, however, they had a relatively short tenure in their education portfolios – four or fewer years. Evidence and policy practice: Interview data. Ministers of education were asked a series of questions relating to their policy development practice and the role of research in decision-making. The findings suggest that political decision makers vary considerably in their perspectives, but generally have very limited direct knowledge of education research. Some ministers responded to questions that probed their understanding of research -based evidence by referring to the importance of experimental studies, using terms like ‘double -blinded’ and ‘solid data’. Others associated valid research with program evaluations. Most participants expressed the view that valid research should be objective and involve “many cases” so it can be generalizable to the population. Ministers’ responses to questions that probed their understanding of the relationship of current educational research to policy decisions were divergent and confused, suggesting that the political decision makers in this study have a very superficial awareness of current research and its potential use for educational decision-making. Ministers were clearer in their representations of the value of external or university-based research and expressed negative views with respect to accessibility. The consensus view of the participants Education Policy Analysis Archives Vol. 23 No. 109 18 interviewed was that university research is obscure and inaccessible to policymakers. The following exchange illustrates this point: M2: Once the research is being done it’s just sitting there; it’s staying in academia. It’s not moving out into government, into communities; into business, as it should be. Now I’ve seen some instances where it’s been successfully done, but I have to admit, that from the year and a half that I’ve been here I’m struggling very hard to find an instance where it’s been useful to me in my work. R: So, in order for it to be more useful you’d say it would have to be more relevant to the issues that are being dealt with by government? M2: You see it could be that the research they are doing is relevant, but we don’t know about it. They’re just not letting us see the results of their research; they’re not saying to us, “This is what we have done. Could it be of use to you?” We’re not involved, I guess. That’s what [the problem] would be. The knowledge generated by university research was also seen to be inconsistent with the information needs of government. Participants questioned the relevance of research to the day-to- day policy questions that are important to education ministers. As one minister noted, education research is ‘not a main reason for change’. The following excerpts are typical of these representations: M7: Well, I haven’t had a chance to look at anything [university researchers] are doing so I really don’t know, but I do find that sometimes, just from past experiences, that the research coming out of (names university) is not always; well not the research, but some of the conclusions they draw from their research is not always applicable…. M2: Well, I don’t really know about it, to be truthful. As I was saying earlier that’s one of the things that is most surprising to me; that with the amount of research that is done at the university, how little discussion or impact it’s had on my work as the Minister in this portfolio. Participants also perceived a wide separation between the education research community and government departments where education policies are conceived. Participants observed that education researchers operate in different operational environment with a vastly different set of goals and orientations. Some ministers charged researchers with criticizing the education policy directions of government instead of offering productive suggestions or engaging in constructive debate. Several ministers expressed the view that university researchers tend to either ignore important policy problems or simplify those problems by remaining focused at a theoretical or ideological level, while failing to account for real-life policy pressures: M1: I guess that what I find with university research is that those who are doing it appear to be somewhat removed from the immediacy of it all; it’s not engaged, as it were, and [university researchers are] somewhat reluctant to engage those who are involved in a particular issue on a daily basis. That’s being pretty general; that’s not the case with all university research. M3: University research? Well, I don't know… we had all the mathematics people look at the mathematics curriculum in (province). They did that in 1998, 1999, 2000, as a result of our kids doing poorly in mathematics and so we had all these pundits. They're all experts. They put together this curriculum. The curriculum came out and…some of the very professors who were involved in the generation of that Research and evidence in education decision -making 19 [curriculum] came out and criticized it in the papers. So I don't understand where they're coming from. In each of the interviews, ministers were then asked to identify from a list of possible influences or evidence, the five factors they consider to be most influential when making education policy decisions. Collectively, participants identified a total of 42 factors8 represented in Table 5. These data support the findings from the in-depth interviews. While ministers claim to value a broad range of policy evidence, there was a collective preference for policy evidence that falls within five principal categories of evidence: (1) political-democratic factors, which we frame as inclusive of public opinion, the representations of individuals, special interests/advocacy groups and the representations of the media; (2) pragmatic or practical considerations, such as financial concerns and governance ideology; (3) global (macro-level) competitive influences and pressures – in particular, evidence such as educational rankings or standings based on large-scale national and/or international assessments; (4) staff advice; and (5) personal and/or professional beliefs, values and experiences. Table 5 Education Ministers’ Perceptions of Most Important Influences or Evidence on which to base Policy Decisions (N=8) Factor/Evidence Number of References Government’ strategic direction9 8 Representations of individuals or special interests10 7 Advice of the deputy minister and senior staff 6 Department’s own research and indicators 5 Public opinion 5 Budget considerations 4 Personal or professional beliefs, values and experience11 3 Province’s goals for economic growth 1 Potential to advance the reputation of the government or the minister. 1 Minister’s own particular interest or emphasis 1 CMEC research and assessment(s) 1 Ministers compared with senior education bureaucrats. There are striking similarities in how politicians and bureaucrats represent the importance of different categories of policy evidence, but there are differences in the relative standing or ‘ranking’ of each category. In Table 6 the key policy drivers for politicians and senior public servants are grouped into five main categories, ranked, in order of influence. As noted, for politicians the most significant influence on policy development seems to be based largely on broadly defined 8 In two cases participants identified six (6) factors 9 The following categories were collapsed: Cabinet/Government’s overall direction and the Department’s strategic plan 10 The following categories were collapsed: representations of individual students, parents or constituents; representations of business groups and the private sector; representations of individual teachers/instructors/administrator, and; Representations of special interest stakeholder groups. 11 The following categories were collapsed: past experiences with related issues and personal or professional beliefs and values Education Policy Analysis Archives Vol. 23 No. 109 20 political-democratic influences: public opinion, advocacy, the mass media and other political and pragmatic pressures, including political ideology. While senior bureaucrats also consider political-democratic considerations to be important, these factors are ranked lower. Deputies and assistant deputies place highest value on the internal advice provided by staff and colleagues, followed by past experience, personal and professional beliefs and values, corporate and pragmatic evidence and, political considerations. While politicians deem staff advice to be important, it was considered to be less important than corporate and pragmatic considerations (such as financial exigencies and governance ideology), and global (macro -level) competitive pressures. Table 6 Comparison and Ranking of Most Important Categories of Evidence for Education Policy Decisions, Education Ministers and Senior Education Bureaucrats Education Ministers Senior Education Bureaucrats 1. Political Considerations 2. Corporate and Pragmatic Considerations 3. Global (Macro-level) Competitive Pressures 4. Advice of Staff 5. Personal /Professional Beliefs and Values 1. Advice of Staff and/or Colleagues 2. Past Experience 3. Personal /Professional Beliefs and Values 4. Corporate and Pragmatic Considerations 5. Political Considerations The 2012 Study Profile of participants: School trustees and superintendents. Most survey respondents (62%) were from Western Canada, with 22 percent from the Central region (Ontario and Quebec), and 13% from the Atlantic Provinces. Three percent of respondents did not identify their location. The balance of male and female respondents was similar with slightly more females (53%) than males. The age of respondents varied from under 30 years (1.5%) to over 60 (33%) with the majority (67%) over the age of 50. About two thirds of respondents reported that they had served in the role of trustee for more than five years. School trustee occupations were quite varied (e.g. stay-at-home parents, skilled tradespersons, farmers, fishers, educators, office workers, business owners, health care professionals, clerks, civil servants, managers/executives and retirees). Evidence and policy practice: Survey data. We first sought to understand the general categories of school board decisions made by trustees in the fulfillment of their governance roles. Their responses, summarized in Table 7 provide a glimpse of the decision -making priorities of school boards across the country. It appears that the primary focus of school board policy decision-making relates to programs, school development, and assessment. The second most commonly identified category was actual engagement in policy development and governance. Other common foci included the provision of safe caring schools, transport ation, school restructuring, human resources, technological innovation, strategic planning, engaging families and community, budgeting, and resource allocation. Participants were also asked to indicate their views on a series of statements relating to governance and decision-making by using a seven-point Likert scale (strongly disagree to strongly agree). Most trustees/superintendents (56%) either agreed or strongly agreed that “major decisions Research and evidence in education decision -making 21 in this school district are based on data,” while 33 percent were less definitive in their responses indicating that they“ somewhat agree” with the statement. These responses suggest that although Table 7 Categories and Frequency of Policy Decisions Recalled by School Trustees and Superintendents Category Frequency Percent Programs, school development, assessment 78 25.6 Policy development and governance 48 15.7 Safe, caring environments 28 9.2 Transportation 26 8.5 Restructuring and school closures 18 5.9 Internal board/district matters 17 5.6 Human resources 16 5.2 Technology and Innovation 16 5.2 Strategic planning and oversight 14 4.6 Family and community 13 4.3 Budgeting and finance Maintenance and infrastructure Ministry of Education Aboriginal/First Nations Religion Public relations Total 12 5 5 4 3 2 305 3.9 1.6 1.6 1.3 1.0 0.7 100 reliance on data may not be an established approach to all decision-making in school boards, a majority of participants feel that it has become a meaningful aspect of their decision-making practice. To obtain a more specific understanding of the factors that influence school board decision-making, we presented trustees with 20 potential factors/influences and asked them to indicate on a seven- point scale, the extent to which each has had an effect on specific decisions or recommendations that they have made as trustees (Table 8). The principal six factors are as follows: (1) personal or professional beliefs and values, (2) potential to directly influence student outcomes/student learning, (3) advice of district staff and/or colleagues, (4) the school board's strategic plan and (5) past experience. Trustees and superintendents rated the following six factors as having the least influence on decision-making: (1) representations of business/private sector, (2) pressure from special interest or lobby groups, (3) a situation or event someone told you about, (4) pressure from government (ministry/department of education), (5) public opinion/avoidance of negative media attention, and (6) university-based research. Analysis of the data by province showed very little variance in the factors and evidence that influence school board decision-making across jurisdictions. Other factors and evidence with a moderate level of influence include (1) the potential to advance the reputation of the school district or board, (2) research or indicators prepared in-house by district staff, and (3) budget considerations while factors such as the representations of teacher and administrators, business/private sector interests, ministry of education assessments and other external research were seen as having only marginal influence on school board decisions. Education Policy Analysis Archives Vol. 23 No. 109 22 Table 8: School Board Trustee and Superintendent Perceptions of the Relative Influence of Different Factors/Evidence on their Recent Policy-relevant Decisions/Recommendations12 Factor/evidence N Mean Standard Deviation Personal or professional beliefs and values 337 1.70 1.39 Potential to directly influence student outcomes/student learning 333 1.66 1.81 Advice of district staff and/or colleagues 340 1.28 1.56 School board strategic plan 337 1.17 1.84 Past experience with this issue or with a related issue 334 0.96 1.74 Potential to advance the reputation of the school district or board 338 0.86 1.92 Research or indicators prepared in-house by district staff 334 0.74 1.89 Budget considerations 337 0.55 2.06 Representations of teachers and/or administrators 334 0.23 1.95 Representations of business/private sector 337 0.15 1.95 Ministry/department of education strategic plan/overall directions 335 -0.06 1.98 Research conducted by outside agencies 333 -0.29 2.18 Provincial government research or assessment information 335 -0.35 1.96 Decisions of other school boards 337 -0.69 1.88 University-based research 333 -1.08 1.97 Public opinion/avoidance of negative media attention 335 -1.09 1.83 Pressure from government (ministry/department of education) 336 -1.15 1.96 Situation or event someone told you about outside your professional environment 323 -1.24 1.90 Pressure from special interest or lobby groups 338 -1.26 1.77 Potential to advance reputation of district/board 330 -1.53 1.68 Evidence and policy practice: Focus groups. The focus-group protocol was designed as an in-depth probe of the factors influencing policy decision -making. Accordingly, we were able to assess this measure separately for school board trustees and superintendents . In each focus group session we presented each participant with a list of potential factors and influences on decision-making (similar, but not identical to the list of factors provided on the questionnaire). The list was a slightly modified form of the instrument used in Galway’s (2006) study. We asked trustees and superintendents to identify and rank, in order of importance, the top five factors/influences on their own policy practice. Although the two lists were not identical, findings from the focus group data with trustees and superintendents generally paralleled the findings based on the survey data with some minor variations . Trustees tended to place more value on the “representations of students, parents and constituents” than did superintendents while superintendents ranked “past experience” and “external research” somewhat higher than did trustees. Otherwise there was close alignment between the rankings of trustees and superintendents with the following factors all ranked high: (1) advice of the CEO (superintendent) and senior staff, (2) school board’s own research and indicators, 12 Mean values for each of the statements were converted from a 1 to 7 interval scale to a -3 to +3 interval scale, where -3 represented 'did not at all influence', 0 was neutral and 3 represented 'influenced to a great extent'. Research and evidence in education decision -making 23 (3) budget considerations, and (4) government/department of education’s overall direction. The data derived from the focus group responses were somewhat similar to the questionnaire data, with some notable exceptions. For example, the questionnaire data showed the influence of the ministries of education to be a relatively minor influence while the focus group participants considered it to be much more significant. The lowest ranked factors and evidence for both groups were (1) representations of individual teachers and administrators, (2) representations of special interest groups and (3) factors such as “anecdotal reports” and “economic goals for the region.” With regard to research utilization, our focus group sessions with school board members in some regions revealed a somewhat different perspective – not entirely consistent with the pan- Canadian survey data. In Quebec, commissionaires (trustees) noted that frequently they work closely with university researchers in determining policies and programs. The following response to our question, “What kind of research, if any, do you value when making policy decisions?” illustrates: It depends on the policy. Our board has a policy on policy development. We don’t research that very much. But when we have our policy for dealing with children with special needs then we use a lot of research of our university partners, McGill and Concordia. Doing our policy on digital citizenship…we took a lot of research from the Internet [and] we went out to people who implemented things before. Our policy on fieldtrips …is based on the law and insurance requirements, so it really depends on the nature of the policy, but we will do a fair amount of research, as appropriate. Another focus group of school board trustees in Saskatchewan echoed these views; their perspective suggesting that they place higher value on research evidence for decision-making than would be indicated by the data derived from the questionnaire: I think that’s what validates. I mean that’s where…you can separate the politics and [decision-making] becomes evidence-based…. When you do that then you’ve got substance based on evidence and people can stand behind it. It’s not so much public opinion and private opinion; it’s more a matter of some form of evidence-base. In one of the Atlantic Provinces, one participant responded, “[i]t depends on the issue. If it is something to do with student achievement then I think we look at university research, scholarly work and evidence more closely than we ever have.” These responses suggest that school board trustees, like education ministers and senior education bureaucrats place high value on political considerations in decision-making, such as the representations of students, parents and constituents. Their responses also correspond to the representations of ministry of education policy-makers in that they place high priority on staff advice and past experience. The responses of the participants from the two studies are also similar in that they say they place substantial importance on personal and professional beliefs and values, corporate and pragmatic influences, such as budget considerations and the government’s “overall direction.” The value placed on external research as policy evidence was mostly consistent among the participants in the two studies. Generally, the questionnaire data indicates a low level of research utilization among Canadian education policy elites. The consensus view of the political decision- makers interviewed in the 2006 study is that external research, and particularly university research is somewhat obscure and inaccessible to policymakers – a scenario where education researchers make a very marginal contribution to education policy decisions. The representations of school board trustees and superintendents were generally consistent with this perspective, although less definitive. Education Policy Analysis Archives Vol. 23 No. 109 24 The mean scale score13 representing the influence of university research on policy decision-making within ministries and school boards was -1.36 and -1.08, respectively. Based on focus group data, there was some variation in these findings for certain regions of the Country where school board trustees and superintendents said they relied on universities to undertake policy research and were likely to utilize research evidence if they were aware of its existence and deemed it relevant to their circumstances. It also appears that for policy decision-making, not all sources of research evidence are given equal sway. In both studies, internal research and indicators prepared either by ministry staff or school district staff was ranked the same – in the middle of the positive range of responses (mean scale scores of 0.86 and 0.86, respectively). The data show that external (university-based) research, however, did not rank highly in either study. Discussion Theorizing Low Research Impact Several educational researchers have described the educational landscape in Canada over the past quarter century as turbulent and uncertain as policymakers have tried to negotiate the problems of fewer students, unstable budgets, and new expectations for schools (Levin, 2003b; Levin & Riffel, 1997; Sheppard, Galway, Brown & Wiens, 2013; Ungerleider, 2003). There is widespread agreement that decision-making in education now occurs under a microscope; it has become highly politicized and subject to intense media attention (Levin, 2004a; Lingard & Rawolle, 2004; Thomson, 2004; Ungerleider, 2004). As Beck (1997) and Giddens (1990; 1994; 2003) argue, cultural changes and the growing complexity of society creates greater levels of uncertainty, resulting in people becoming more restless, taking less for granted, and initiating action to reduce their perceptions of risk. Canadians are passionate about education, and are unabashed about raising the bar for what they expect from schools. But public expectations for education are constantly shifting, creating policy challenges and dilemmas for decision makers. More than in the past, citizens and education lobby groups are better positioned with knowledge and expertise; they are better equipped to challenge institutional experts within school boards and ministries of education and have a greater capacity to lobby governments and force change. As Guppy and Davies (1999, p. 278) point out, parents and educational interest groups are “less deferential towards the educational establishment and are more likely to feel entitled to challenge experts.” Based on a review of two decades of polling evidence, Guppy and Davies (1999) showed that public confidence in Canadian schooling had declined in all population subgroups. A more recent survey, undertaken by the Canadian Education Association, showed that between 1984 and 2007 the percentage of Canadians who reported a great deal or fair amount of confidence in schools dropped by 31 percentage points from 76 to 45 percent (Canadian Education Association, 2007). This dissatisfaction has been expressed in myriad ways – from a startling increase in private tutoring (Davis, Aurini & Quirke, 2002) – particularly among wealthy Canadians (Cain, 2013) – to a rise in the number of legal cases against school boards (Sheppard, 2012). A 2012 Ipsos-Reid poll revealed that Canadians gave a poor rating to the “overall state” of the public elementary school system. More than half of those polled (58%) held the view that a private school education is better than a public school education and about two thirds (63%) said they would choose a private school education for their children, if money were no concern. At the same time, virtually all regions of the country have been experiencing major demographic changes leading to shifts in student populations, profound enrolment decline and 13 where -3 represented 'did not at all influence', 0 was neutral and 3 represented 'influenced to a great extent'. Research and evidence in education decision -making 25 greater diversity in the school-aged population. Census data show that Canada recorded the highest population growth among G8 nations in 2013-14; however, net international migration accounted for almost two-thirds (66.5%) of the total increase. In the last several decades there have also been profound changes in proportion of Canadians leaving smaller towns and settlements for urban areas – from 62% in 1951 to 81% in 2011 (Statistics Canada, 2011). Within the country the population share of some regions, notably Quebec and the Atlantic Provinces has decreased over the past three decades while Ontario and the Western Provinces has increased (Statistics Canada, 2014). For, example, over the twenty-year period, 1993-94 to 2013-14, the number of school-aged children in the Atlantic region fell by 30 percent or about 1.6 percent per year and enrolment losses between one and two percent per year are expected for at least the next decade (Galway, 2015). Schools are among the first public institutions to be impacted by socio-cultural and demographic changes. Addressing sustained enrolment changes (influx or decline) and responding to more diverse student populations require considered policy responses. Other social tensions and technological changes are similarly first felt in schools and colleges, leaving policy actors to negotiate everything from school closures and realignment to safe school protocols and diversity policies, often with a restless and impatient public. Ministries of education and school boards are expected to base their policy decisions on the best evidence available to them. The findings from these studies, however, portray the policy development process as ambiguous and political; driven by informal evidence that is only loosely connected to the educational research community. These findings are consistent with the theoretical orientations of Beck (1994; 1997), Majone (1989), Kingdon (1995), Levin (2003b; 2004b) and Stone (2002), all of whom conceptualize public policy-making as divergent, uncertain and risky – driven by personal, political, and experiential influences. In a politicized environment, independent university research, especially that which might be tacitly critical of existing policy, may either represent additional political risk or be perceived as inaccessible or irrelevant to settling urgent policy questions. Under such conditions, the values, reward systems and accountabilities against which social scientists operate are likely to differ substantively from those of system-level decision makers, thereby creating a policy environment that might be antagonistic to external research. These conditions may well force policymakers to set aside research-informed policy (which is normally a longer-term proposition) in order to attend to less risky and more immediate pragmatic and democratic/political considerations. Making Research Matter More These studies raise questions about whether the existing decision-making approaches are useful in advancing a sustainable policy agenda for Canadian schools. They also underscore the importance of undertaking policy work in consideration of the best available research. Decision makers need ready access to research evidence to guide their practice in the public sphere, but perhaps more pointedly, to refute incorrect information and hearsay that can so often form the substance of unqualified demands for educational change. Following Edwards et al. (2007), Jbilou et al. (2007) Landry et al. (2001), Nutley et al. (2008) and others, we propose that to attenuate the risk associated with greater reliance on education research stronger connections must be fostered between the research community and several levels of the bureaucracy within school districts and ministries of education. In short, to become a more significant consideration in the policy advice senior advisors provide to politicians, research must become more familiar and trustworthy. The importance of trust and confidence has been discussed elsewhere in the literature (e.g., Nutley et al., 2009; Tseng, 2012). One of the key questions raised by researchers is how do policymakers and other information consumers sort through competing claims in guiding their practice (Lubinenski, Scott & Debray, 2014). While the studies compared here have focused on the Education Policy Analysis Archives Vol. 23 No. 109 26 standing of research and other factors as policy evidence, several researchers have begun to conceptualize research utilization as a social process: “one that unfolds within a complex ecology of relationships, organization and political and policy contexts” (Yohalem & Tseng, 2015). One recent US study of school board decision makers found that the value of research could not be decoupled from trustworthiness of its sources and the motives of the person conveying the evidence (Asen, Gurke, Donners, Solomon, & Gumm, 2012). Rosenblatt and Tseng (2010, p. 201) have argued for the need to understand the importance of the demand side of research-policy equation, including “user settings and perspectives, political, economic and social contexts and the various uses of research.” Based on large-sample studies in the Canadian health care sector several strategies have been suggested to improve reciprocal understandings between the research and policy communities. Landry et al. (2001) identified several determinants of research utilization including the mechanisms linking researchers to users and the user’s operational context. Jbilou et al. (2007) added that communication networks, partnerships and other links between researchers and decision-makers could help effect a cultural change in governmental organizations – whereby there is greater support and value for research into practice. Programs such as the former TLRP in the UK and the former Community-University Research Alliance (CURA) program in Canada represented efforts to support alliances between community organizations and universities to foster innovative research, training and the creation of new knowledge (James & Pollard, 2011; Social Sciences and Humanities Research Council, 2013). More to the issue of engendering trust and confidence, Edwards et al. (2007) describes such programs in terms of a reciprocity or co-construction of research-based knowledge that serves to connect organizational boundaries and strengthen the warrant for research studies. In the following we draw upon the findings from these studies and the work of others researching this area to suggest some modest proposals for how such evidence could become a more integral aspect of education policy and practice. Engaging ministry and school board advisors. Following on Jbilou et al. (2007), we see research collaborations, research forums and regular formal (or informal) meetings with mid- and senior-level school board and ministry officials as important mechanisms for interchanges between knowledge producers and the education policy community. There are few opportunities for external research to reach policymakers, except through the bureaucracy and, to lesser degree, through the efforts of special interest groups and the mass media. Reimer and McGinn (1997) wrote about the difficulty they experienced in communicating research findings to government decision makers. As a dissemination strategy, direct access to school board trustees and/or ministers of education is problematic for several reasons. Access to policy elites is difficult at the best of times and focusing the attention of elected officials on complex teaching and learning issues is a longer-term proposition. Even in instances where researchers are successful in engaging elected decision makers, due to their crowded schedules, such meetings are likely to be short, infrequent and limited in their long -term impact. A more effective strategy would involve the active engagement of mid to senior policy professionals who prepare the information – informal e-mails or briefings, decision notes and policy papers – that is circulated ‘up the line.’ Bah Diallo (2001), a former minister of education in Guinea, says that politicians can only effectively initiate policy change with the cooperation of the bureaucracy. Administrators identify for ministers not only what is needed to effect change, but also the appropriate pathways to implementation: Administration constitutes the passage oblige, the gateway, the bridging link between policy and practice, between political intent and the hard reality of day-to-day business. As such it Research and evidence in education decision -making 27 plays the difficult role of reconciling the need to maintain the system and the need to lead the changes and the reforms. […] administrators have a tremendous influence on the system processes and the policy-makers’ capacity to decide because administrators can influence action where it counts the most, that is, in the field, where things happen. (p. 22) The policy actors who participated in the 2006 and the 2012 study indicated they place high value on knowledge produced or validated by insiders – ministry and/or school district policy staff, a finding consistent with Majone’s (1989) suggestion that the policy analyst plays a pivotal role in determining what evidence is considered by policy elites and the manner in which policy advice is constructed. While mid-level bureaucrats and district program specialists represent the entry point for external knowledge, senior bureaucrats mediate the cascade of information to elected decision makers by distilling and validating evidence and bringing together research with other forms of evidence to generate ‘trustworthy’ staff advice. Intersecting communities of practice. There are several small-scale efforts that could be undertaken to create intersections between the individual communities of practice of researchers and policy advisors/practitioners. Formal and informal professional exchanges cou ld create reciprocal space for discussion of policy issues and problems and the kinds of research that could contribute to public discourse and/or policy deliberations. In our earlier work on researcher perceptions of evidence used in policy making (Galway, 2006), participants referenced the challenges in sustaining regular professional exchanges between academics, senior public servants and ministry advisors, but also their effectiveness in connecting the three communities of practice. Another point of interchange could be the establishment of rotating research chairs within education ministries and faculties of education – whereby mid-to-senior level public servants could become immersed in the research culture of the university while academics could experience the policy environment of government. There is already some precedent for this kind of exchange. In at least one Canadian province there is a standing research fellowship that enables senior executives of the government to work in a university setting and to teach and research in their professional discipline. Through the fellowship, there have been similar moves in the opposite direction, whereby faculty members take up residence in government departments. For example, the former Canadian Council on Learning had conceived a plan centered on the establishment of ‘researchers in residence’ as a means of infusing research into policy discussions at the level of education ministries14. Under such an arrangement the ministry and/or the university would underwrite or partially underwrite the associated costs of the exchange; the participants would then focus on one or more policy questions occupying the district/ministry agenda. This research would imply that these and similar opportunities for researcher-bureaucrat exchange should be encouraged as a means of creating common understandings among school districts, government departments and education faculties. Another approach formalizes the involvement of educational leaders in research through the promotion of advanced professional degrees aimed at administrators, bureaucrats and senior educators. Now reasonably well established in many faculties, the professional doctorate requires that candidates undertake research relating to their professional practice. While relatively few professional doctorates are awarded each year in relation to the number of professional staff working in education, the connection of policy analysts and program specialists to the research community through professional learning opportunities would appear to be an effective, albeit 14 Personal communication Education Policy Analysis Archives Vol. 23 No. 109 28 small-scale means of infusing research into policy practice. School boards, post-secondary institutions and ministries of education could provide incentives to encourage professional staff to undertake research-based professional degrees as a means of professional development. Financial incentives such as grants to assist with tuition and research expenses, sabbatical leaves for senior public servants and/or promotion incentives could entice more senior education officials to begin doctoral or professional masters programs. Moreover, the design of many professional doctorate programs affords the student considerable flexibility. The completion of degrees, partially through Web-based study, the ability to study part-time over extended periods, and the offering of compressed didactic components using itinerant professorial instruction all contribute to the professional degree option as a viable means of making research more familiar to the policy community. Moderating the critical stance of the academy. We have argued that research will become more entrenched in policy development if it can become more familiar and less risky for policy makers. The data from the 2006 study show that ministers of education repres ented researchers as sometimes adversarial – either indifferent to the pressing educational problems on the government’s policy agenda, or too often, critical of education policies and practices. Recognizing the cautions of Pirrie, Adamson and Humes (2010) regarding the spread of bureaucratic rationality and the responsibilities of academics to speak truth to power and preserve freedom in academic life, we believe some consideration should be given to the ways in which academics represent their research in the public sphere, and how they frame policy critiques. While certain traditions of inquiry lend themselves to critical commentary on social institutions and processes (Creswell, 1998), the findings from this study suggest that sustained critique of policy decisions may serve to position academic research as foreign and threatening to policymakers. When academic researchers publicly argue for a strict (research) evidence - informed approach to policy, they may inadvertently overlook the political dynamic of a policy situation, and situate themselves as adversaries. As Caplan (1979, p. 461) writes , [t]he need for reciprocal relationships between knowledge producers and knowledge users in policy -making is clear, but the problem of achieving effective interaction of this sort necessarily involves value and ideological dimensions as well as technical ones. ” The studies discussed here show that decision-makers make choices in the context of an overwhelming and widely diverse set of public expectations. Politicians, in some instances, seem to identify those in the academy with risk and uncertainty. For some researchers – particularly younger scholars – the contextual background to the policy process may represent uncharted territory. In order for education research to begin to make a difference in the policy decisions of policymakers, academics may wish to evaluate how far critical approaches have advanced the influence of research on the public policy agenda and consider different ways and means of drawing attention t o their policy concerns. Researchers may better situate themselves to influence policy decisions by accounting for the ambiguities and political pressures of decision -making in their critique, while advocating for reasoned, research-informed policy. Balancing academic interests with research impact. The findings from these studies also suggest that attempts to stimulate more policy -relevant research will also come with the need to re-evaluate traditional ideas about the academic utility of research in relat ion to its practical value to consumers and policy makers. From the perspective of academic progression, researchers have good reason to doubt the value of undertaking commissioned or government/district-sponsored research. Levin (2004b, p. 9) has remarked that, “university- based research is driven by the university culture and reward system; it is primarily aimed at Research and evidence in education decision -making 29 communication with other scholars and the rewards are related to research grants and peer recognition.” In her case study of a large Canadian community-university research collaboration, Tapp-Neville (2015) reported that researchers were concerned that certain forms of collaborative and commissioned research have a lower standing than independent research as evidence of contribution to scholarly inquiry. If university communities, even passively assign lower status to commissioned policy work, they may, unintentionally, restrict the likelihood that collaborative research programs will attract young, able scholars. Moreover, if commissioned education research has limited standing within the academic community it might only be conducted in the ‘sidelines’ – something done to sustain a stream of research funding or otherwise left to researchers who have advanced in their careers and have already esta blished themselves as credible academics. From the perspective of enhancing research impact, there is reason to be concerned about a culture that draws a distinction between the importance of research that is needed and valued by ministries of education and school districts and other ‘legitimate’ research that is valued within the academy. For education research to become a more visible presence in day-to-day policy discussions, and for the quality and amount of commissioned research to improve, consumer-driven research needs be recognized for the purposes of academic reputation and tenure-track promotion. Finally, if commissioned research is assigned a lower status with respect to promotion and tenure, it may unintentionally restrict the likelihood that such research opportunities will attract young, able scholars. This topic will not be further addressed here, although it is important area for future research. Conclusions On balance, the findings from the 2012 study of school board trustees and superintendents are generally consistent with those of the 2006 study of education ministers and senior education bureaucrats. We found that elected officials and their advisors in both contexts use a variety of forms of evidence in their decision-making, including non-formal sources of evidence such as local (internal) knowledge and experience. Policy elites, particularly at the ministry level, tend to place considerable reliance on political-democratic forms of policy evidence, followed by staff advice, corporate and pragmatic considerations, personal and professional beliefs and values and past experience. Both participant groups also confirm the influence of formal sources of information, such as research evidence; however the form of such research matters, and appears to be largely limited to internally generated research and indicators. Findings from the 2006 study show that ministry policymakers express largely negative perspectives about the availability and value of certain forms of research-based knowledge. The ministers interviewed were either oblivious to the research undertaken in universities or deprecating in their appraisal of its availability and utility. Both political actors and senior bureaucrats say that they place greatest value on research-based evidence that originates or has been vetted by staff and policy advisors within the ministries themselves. Trustees and superintendents expressed a similar perspective placing greater value on the school board’s own research and indicators, while downplaying university research as less influential to school board decision-making – this form of evidence ranking 15th of 20 possible decision-making factors. Still, school board trustees and superintendents were more predisposed to the notion of “data-driven” decision-making and some focus group participants mentioned collaboration with university researchers; however, these references were few and limited in scope. Education Policy Analysis Archives Vol. 23 No. 109 30 We conceptualize use of low research impact partly in terms of trust vs. risk avoidance. In both policy contexts – provincial departments of education and school boards – we conclude that decision-making is ambiguous and risky. Policies and practices are developed and enacted in an uncertain policy context, often with diverse and impatient constituencies. Following Beck (1994, 1997), in uncertain times, decision-makers value knowledge that is familiar and emerges from their own community. Such knowledge, while frequently originating in external studies, is contextualized against other policy evidence, and judgments are made regarding its relative value within the larger policy context. This information – now repackaged in the form of staff advice – becomes privileged and trusted by ministers and senior bureaucrats as authentic knowledge. The problem inherent in a decision-making paradigm that is overly dependent on informal, populist and experiential knowledge is that it is haphazard and fundamentally reactive, thereby inviting continuous reform and readjustment. The evidence and arguments presented herein support calls from others for a more open, collaborative, familiar and trusted relationship between education practitioners and researchers. Such a change could begin to immunize schools against frequent changes in practices and policies until they have been referred for evaluation against the best information available. Following on other recent work in this area, several tangible strategies are suggested that might be helpful in effecting such a change. These representations also raise policy issues for administrators in some Canadian universities, both from the perspective of enhancing the formal and informal linkages between the research and policy communities, and in re-imagining the standing of user-defined (commissioned) research within the academy. If education research is to become a staple at the table of policy decision makers, those in authority within the academy would do well to examine and debate these issues and, where necessary, make appropriate adjustments. References Asen, R., Gurke, D., Conners, P., Solomon, R., & Gumm, E, (2013). Research evidence and school board deliberations. Lessons from three Wisconsin school districts. Educational Policy, 27(1), 33-63. http://dx.doi.org/10.1177/0895904811429291 Bah Diallo, A. (2001). Finding the right balance. In G. Hermes, & Y. Haq (Eds.), Organization of ministries of education. International Institute for Educational Planning Policy Forum, 13, (pp. 20- 33). Paris: UNESCO. Barber, M., & Sebba, J. (1999). Reflections on progress towards a World Class Education System. Cambridge Journal of Education, 29(2), 183-193. http://dx.doi.org/10.1080/0305764990290202 Bates, R. (2001). Reviewing research in education in Australia and the UK: Evaluating the evaluations (unpublished manuscript). Retrieved from http://www.deakin.edu.au/education/RADS/staff_folder/bates/ Beck, U. (1994). The reinvention of politics: Towards a theory of reflexive modernization. In U. Beck, A. Giddens & S. Lash, (Eds.), Reflexive modernization: Politics, tradition, aesthetics in the modern social order. Stanford, CA: Stanford University Press. Beck, U. (1997). The reinvention of politics: Rethinking modernity in the global social order. Cambridge, UK: Blackwell Publishers Inc. Black, Nick (2001). Evidence-based policy: Proceed with care. British Medical Journal, 323(7307), 5-13. http://dx.doi.org/10.1136/bmj.323.7307.275 http://dx.doi.org/10.1177/0895904811429291 http://dx.doi.org/10.1080/0305764990290202 http://www.deakin.edu.au/education/RADS/staff_folder/bates/ http://dx.doi.org/10.1136/bmj.323.7307.275 Research and evidence in education decision -making 31 Blunkett, D. (2000). Influence or irrelevance: Can social science improve government? Economic and Social Research Council, British Educational Research Association (invited speech). Retrieved from http://www.bera.ac.uk/ri/no71/ri71blunkett.html Bok van Kammen, W., & Stouthamer-Loeber, M. (1998). Practical aspects of interview data collection and data management. In L. Bickman & D. Rog (Eds.), Handbook of applied social research methods (pp. 375-399). Thousand Oaks, CA: Sage. Burchett, H. E. D., Mayhew, S. H., Lavis, J. N., & Dobrow, M. J. (2012). When can research from setting be useful in another? Understanding perceptions of the applicability and transferability of research. Health Promotion International, 28(3), 418-430. http://dx.doi.org/10.1093/heapro/das026 Cain, P. (2013, April 19). Sixty percent (60%) more Toronto students have private tutors – in wealthy areas most of all. Global News Toronto. Retrieved from http://globalnews.ca/news/492413/private-tutoring-rising-sharply-in-toronto-map-shows- strong-link-to-income/ Canadian Education Association (2007). Public education in Canada: Facts, trends and attitudes. Retrieved from http://www.cea-ace.ca/sites/default/files/cea-2007-public-education-in-canada.pdf Caplan, N. (1979). The two-communities theory and knowledge utilization. American Behavioral Scientist, 22(3), 459-470. http://dx.doi.org/10.1177/000276427902200308 Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education. New York, NY : Routledge. Cooper, A. (2013). Research mediation in education: A typology of research brokering organizations that exist across Canada. Alberta Journal of Education Research, 59(2), 181-207. Cooper, A. (2014). The use of online strategies and social media for research Dissemination in Education. Education Policy Analysis Archives, 22(88), 1-26. Cooper, A., & Levin, B. (2010). Using research in secondary schools: Education leaders respond. Education Canada, 50(4), 59-62. Cooper, A., Levin, B., & Campbell, C. (2009). The growing (but still limited) importance of evidence in education policy and practice. Journal of Educational Change, 10(2-3), 159-171. http://dx.doi.org/10.1007/s10833-009-9107-0 Creswell, J. (1998). Qualitative inquiry and research design. Thousand Oaks, CA: Sage. Davies, P. (1999). What is evidence- based education? British Journal of Educational Studies, 47(2), 108- 121. http://dx.doi.org/10.1111/1467-8527.00106 Davies, S., Aurini, J., & Quirke, L. (2002). New markets for private education in Canada. Education Canada, 42 (3), 36-38. DeBroucker, P., & Sweetman, A. (Eds.) (2003). Towards evidence-based policy for Canadian education. Montreal, Canada: McGill-Queens University Press. Dillman, D. (1999). Mail and electronic surveys: The tailored design method (2nd ed.). New York, NY: J. Wiley. Dunseath, K. (2000). Defending educational research in changing times. Proceedings of the 2000 Conference of the Australian Society for Educational Technology and the Higher Education Research and Development Society of Australia. Retrieved from http://www.aset.org.au/confs/aset- herdsa2000/procs/dunseath-k.html Earl, L. (2011). A response from Canada to TLRP's ten principles for effective pedagogy. Research Papers in Education, 26(3), 381-389. http://dx.doi.org/10.1080/02671522.2011.595549 Edwards, A., Sebba, J., & Rickinson, M. (2007). Working with users: Some implications for educational research. British Educational Research Journal, 33(5), 647-661. http://dx.doi.org/10.1080/01411920701582199 http://www.bera.ac.uk/ri/no71/ri71blunkett.html http://dx.doi.org/10.1093/heapro/das026 http://globalnews.ca/news/492413/private-tutoring-rising-sharply-in-toronto-map-shows-strong-link-to-income/ http://globalnews.ca/news/492413/private-tutoring-rising-sharply-in-toronto-map-shows-strong-link-to-income/ http://www.cea-ace.ca/sites/default/files/cea-2007-public-education-in-canada.pdf http://dx.doi.org/10.1177/000276427902200308 http://dx.doi.org/10.1007/s10833-009-9107-0 http://dx.doi.org/10.1111/1467-8527.00106 http://www.aset.org.au/confs/aset-herdsa2000/procs/dunseath-k.html http://www.aset.org.au/confs/aset-herdsa2000/procs/dunseath-k.html http://dx.doi.org/10.1080/02671522.2011.595549 http://dx.doi.org/10.1080/01411920701582199 Education Policy Analysis Archives Vol. 23 No. 109 32 Figgis, J., Zubrick, A., Butorac, A., & Alderson, A. (2000). Backtracking practice and policies to research. In Department of Education, Training and Youth Affairs, The impact of educational research (pp. 279-274). Commonwealth of Australia, Canberra: Australia. Fitz-Gibbon, C. (2003). Milestones en route to evidence-based policies. Research Papers in Education, 18(4), 313-329. http://dx.doi.org/10.1080/0267152032000176837 Fowler, F. (1998). Design and evaluation of survey questions. In L. Bickman & D. Rog (Eds.), Handbook of applied social research methods (pp. 343-374). Thousand Oaks, CA: Sage. Galway, G. (2006). An investigation of the uses of evidence and research in the education decision making of policy elites. (Unpublished Ph.D. dissertation). University of South Australia, Adelaide , SA. Galway G. (2015). Is teaching still a viable option in Atlantic Canada. In N. Maynes & B. E. Hatt (Eds.) The complexity of hiring, supporting, and retaining new teachers in Canada (pp. 1-21). Canadian Association for Teacher Education/Association canadienne pour la formation à l'enseignement. Galvin, C. (2008). But all the wrong people are here…: The emerging policymaking mode and its challenge for academic policy research: a commentary. Cambridge Journal of Education, 38(1), 121-130. http://dx.doi.org/10.1080/03057640801890020 Giddens, A. (1990). The consequences of modernity. Stanford, CA: Stanford University Press. Giddens, A. (1991). Modernity and self-identity. Self and society in the late modern age. Stanford, CA: Stanford University Press. Giddens, A. (1994). Beyond left and right: The future of radical politics. Stanford, CA: Stanford University Press. Giddens, A (2003). Runaway world: How globalization is shaping our lives. New York, NY: Routledge. Guppy, N., & Davies, S. (1999). Understanding Canadians' declining confidence in public education. Canadian Journal of Education, 24(3), 265-280. http://dx.doi.org/10.2307/1585875 Hargreaves, D. (1996). Teaching as a research-based profession: Possibilities and prospects. Teacher Training Annual Lecture, London, UK: TTA. Hammersely, M. (1998). The confessions of a “methodological purist”. In G. Walford (Ed.), Doing research about education (pp.139-153). Bristol: Falmer Press, Taylor and Francis. Henerson, M., Morris, L., & Fitz-Gibbon, C. (1987). How to measure attitudes. Thousand Oaks, CA: Sage. Hillage, J., Pearson, R., Anderson, A., & Tamkin, P. (1998). Excellence in research on schools. Institute for Employment Studies, Research Brief, 74. University of Sussex. Retrieved from: http://www.dfes.gov.uk/research/data/uploadfiles/RB74.doc Hodgkinson, P. (1998). Naiveté and bias in educational research: The Tooley report. BERA Research Intelligence, 65. Retrieved from http://www.bera.ac.uk/ri/no65/ri65hodkinson.html Hogan, D. (2011). Yes Brian, at long last, there is pedagogy in England- and in Singapore too. A response to TLRP's Ten Principles for Effective Pedagogy. Research Papers in Education, 26(3), 367-379. http://dx.doi.org/10.1080/02671522.2011.595548 Holbrook, A., Ainley, L., Bourke, S., Owen, J., McKenzie, P., Mission S., & Johnson, T. (2000). Mapping educational research and its impact on Australian schools. In Department of Education, Training and Youth Affairs, The impact of educational research (pp. 15-278). Commonwealth of Australia, Canberra: Australia. Honig, M., & Coburn, C. (2008). Evidence-based decision-making in school district central offices: Toward a policy and research agenda. Journal of Educational Policy, 22(4), 578-608. http://dx.doi.org/10.1177/0895904807307067 http://dx.doi.org/10.1080/0267152032000176837 http://dx.doi.org/10.1080/03057640801890020 http://dx.doi.org/10.2307/1585875 http://www.dfes.gov.uk/research/data/uploadfiles/RB74.doc http://www.bera.ac.uk/ri/no65/ri65hodkinson.html http://dx.doi.org/10.1080/02671522.2011.595548 http://dx.doi.org/10.1177/0895904807307067 Research and evidence in education decision -making 33 Honig, M. I., & Coburn, C. (2008). Evidence-Based Decision-making in School District Central Offices: Towards a Policy and Research Agenda. Educational Policy, 22(4), 578-608. http://dx.doi.org/10.1177/0895904807307067 Ince, M. (2008). Knowledge transformation and impact: A commentary. Cambridge Journal of Education, 38(1), 131-134. http://dx.doi.org/10.1080/03057640801890038 Ipsos-Reid (2012, September 3). Canadians give overall state of public elementary school system mediocre marks. Retrieved from http://www.ipsos-na.com/news-polls/pressrelease.aspx?id=5742 James, M., & Pollard, A. (2011). TLRP's ten principles for effective pedagogy: rationale, development, evidence, argument and impact. Research Papers in Education, 26(3), 275-328. http://dx.doi.org/10.1080/02671522.2011.590007 Jbilou, J., Amara, N., & Landry, R. (2007). Research-based-decision-making in Canadian Health Organizations: A behavioural approach. Journal of Medical Systems, 31(3), 185-196. http://dx.doi.org/10.1007/s10916-007-9054-3 Kingdon, J. (1995). Agendas, alternatives and public policy (2nd ed.). Boston: Little, Brown. Knowles, T., Krober, N., Ludwig, J., Petrilli, M.J., Slavin, R., & Tseng, V. (2013). Leveraging learning: The evolving role of federal policy in education research. The Aspen Institute Education & Society Program. Kwok, R. (2003). From administrative state to ministerial system: The quest for accountability in Hong Kong. The Journal of Commonwealth and Comparative Politics, 41(1). http://dx.doi.org/10.1080/713999608 Landry, R., Amara, N., & Lamari, M. (2001). Utilization of social science research knowledge in Canada. Research Policy, 30(2), 333-349. http://dx.doi.org/10.1016/S0048-7333(00)00081-0 Lavis, J., Lomas, J., Hamid, M., & Sewankambod, N. (2006). Assessing country-level efforts to link research to action. Bulletin of the World Health Organization, 84, 620-628. http://dx.doi.org/10.2471/BLT.06.030312 Leicester, G. (1999). The seven enemies of evidence-based policy. Public Money and Management, 25(1), 5-7. http://dx.doi.org/10.1111/1467-9302.00145 Levin, B. (2001). Knowledge and action in educational policy and politics. Conference on Empirical Issues in Canadian Education. Ottawa, Ontario: Canada. Levin, B. (2003a). Helping research to matter more. A brief discussion and proposals for action (discussion paper). Fall 2003 Meeting of the Advisory Committee of Deputy Ministers of Education (ACDME), Canada. Levin, B. (2003b). Knowledge and action in educational policy and politics. In P. DeBroucker & A. Sweetman, (Eds.) (2003). Towards evidence-based policy for Canadian education (pp. 15-33). Montreal: McGill-Queens University Press. Levin, B. (2004a). Media-government relations in education. Journal of Educational Policy, 19(3), 271- 283. http://dx.doi.org/10.1080/0268093042000207610 Levin, B. (2004b). Making research matter more. Education Policy Analysis Archives, 12(6), 1-21. http://dx.doi.org/10.14507/epaa.v12n56.2004 Levin, B., & Riffel, J. (1997). Schools and the changing world: Struggling towards the future. London: Falmer. Lindblom, C. (1980). The policy-making process. New York, NY: Prentice Hall. Lindblom, C., & Woodhouse, E. (1993). The policy making process (3rd ed.). Englewood Cliffs, NJ: Prentice Hall. Lingard, B., Rawolle, S., & Taylor, S. (2005). Globalizing policy sociology in education: Working with Bourdieu. Journal of Education Policy, 20(6), 759-777. http://dx.doi.org/10.1080/02680930500238945 http://dx.doi.org/10.1177/0895904807307067 http://dx.doi.org/10.1080/03057640801890038 http://www.ipsos-na.com/news-polls/pressrelease.aspx?id=5742 http://dx.doi.org/10.1080/02671522.2011.590007 http://dx.doi.org/10.1007/s10916-007-9054-3 http://dx.doi.org/10.1080/713999608 http://dx.doi.org/10.1016/S0048-7333(00)00081-0 http://dx.doi.org/10.2471/BLT.06.030312 http://dx.doi.org/10.1111/1467-9302.00145 http://dx.doi.org/10.1080/0268093042000207610 http://dx.doi.org/10.14507/epaa.v12n56.2004 http://dx.doi.org/10.1080/02680930500238945 Education Policy Analysis Archives Vol. 23 No. 109 34 Lubinenski, C., Scott, J., & Debray, E. (2014). The politics of research production, promotion, and utilization in educational policy. Education Policy, 28(2), 131-144. http://dx.doi.org/10.1177/0895904813515329 Majone, G. (1989). Evidence, argument and persuasion in the policy process. New Haven, CT: Yale University Press. Mangione, T. (1998). Mail Surveys. In L. Bickman & D. Rog (Eds.), Handbook of applied social research methods. Thousand Oaks, CA: Sage. Marshall, C., & Rossman, G. (1989) Designing qualitative research. Thousand Oaks, CA: Sage. Martin, S., Downe, J., Grace, C., & Nutley, S. (2010). Validity, utilization, and evidence- based policy: The development and impact of performance improvement regimes in local public services. Evaluation, 16(1), 31-42. http://dx.doi.org/10.1177/1356389009350119 Mason, G. (2012, September 6). Behind the political decision to cut the lifeline: Tories didn't like CCL's message or its independence. Globe and Mail. Retrieved from http://www.theglobeandmail.com/globe-debate/behind-the-political-decision-to-cut-the- lifeline-tories-didnt-like-ccls-message-or-its-independence/article1364410/ Maxwell, J. (1998). Designing a qualitative study. In L. Bickman, & D. Rog (Eds.), Handbook of applied social research methods (pp. 69-100). Thousand Oaks, CA: Sage McMeniman, M., Cumming, J., Wilson, J., Stevenson, J., & Sim, C. (2000). Teacher knowledge in action. In DETYA, The impact of educational research (pp. 375-550). Commonwealth of Australia: Canberra. Meijer, P., Verloop, N., & Beijaard, D. (2002). Multi-method triangulation in a qualitative study on teachers' practical knowledge: An attempt to increase internal validity. Quality & Quantity, 36, 145-167. http://dx.doi.org/10.1023/A:1014984232147 Merriam, S. (2009). Qualitative research: A guide to design and implementation. San Francisco: Jossey Bass. Miller D. (1991). Handbook of research design and social measurement (5th ed.). Thousand Oaks, CA: Sage. Miles, M., & Huberman, A. M. (1994). Qualitative data analysis, (Second edition). London: Sage. Moat, K.A., Lavis, J.N., Wilson, M.G., Røttingen, J.A., & Bärnighausen, T. (2013). Twelve myths about systematic reviews for health system policymaking rebutted. Journal of Health Services Research & Policy, 18(1), 44-50. http://dx.doi.org/10.1258/jhsrp.2012.011175 Naughton, M. (2005). 'Evidence-based policy' and the government of the criminal justice system – only if the evidence fits! Critical Social Policy, 25(1), 47-69. http://dx.doi.org/10.1177/0261018305048967 Neilson, S. (2001). Knowledge utilization and public policy processes. IDRC Institution: Evaluation Unit Retrieved from http://www.idrc.ca/evaluation/ Nutley, S., Jung, T., & Walter, I. (2008). The many forms of research-informed practice: a framework for mapping diversity. Cambridge Journal of Education, 38(1), 53-71. http://dx.doi.org/10.1080/03057640801889980 Nutley, S., Walter, I., & Davies, H. T. O. (2003). From Knowing to doing: A framework for understanding the evidence-into-practice agenda. Evaluation, 9(2), 125-148. http://dx.doi.org/10.1177/1356389003009002002 Nutley, S., Walter, I., & Davies, H. T. O. (2009). Promoting evidence- based practice: Models and mechanisms from cross-sector review. Research on Social Work Practice, 19(5), 552-559. http://dx.doi.org/10.1177/1049731509335496 Phelan, T., Anderson, D., & Bourke, P. (2000). Educational Research in Australia: A Bibliometric Analysis. In DETYA, The impact of educational research (pp. 575-671). Commonwealth of Australia, Canberra: Australia. http://dx.doi.org/10.1177/0895904813515329 http://dx.doi.org/10.1177/1356389009350119 http://www.theglobeandmail.com/globe-debate/behind-the-political-decision-to-cut-the-lifeline-tories-didnt-like-ccls-message-or-its-independence/article1364410/ http://www.theglobeandmail.com/globe-debate/behind-the-political-decision-to-cut-the-lifeline-tories-didnt-like-ccls-message-or-its-independence/article1364410/ http://dx.doi.org/10.1023/A:1014984232147 http://dx.doi.org/10.1258/jhsrp.2012.011175 http://dx.doi.org/10.1177/0261018305048967 http://www.idrc.ca/evaluation/ http://dx.doi.org/10.1080/03057640801889980 http://dx.doi.org/10.1177/1356389003009002002 http://dx.doi.org/10.1177/1049731509335496 Research and evidence in education decision -making 35 Pirrie, A., Adamson, K., & Humes, W. (2010). Flexing academic identities: Speaking truth to power. Power and Education, 2(1), 97-106. http://dx.doi.org/10.2304/power.2010.2.1.97 Pollard, A. (2008). Knowledge transformation and impact: aspirations and experiences from TLRP. Cambridge Journal of Education, 38(1), 5-22. http://dx.doi.org/10.1080/03057640801889949 Pollard, A. (2007). The UK's teaching and learning research programme: findings and significance. British Educational Research Journal, 33(5), 639-646. http://dx.doi.org/10.1080/01411920701582173 Pring, R. (2000). Philosophy of educational research. London: Continuum. QDA Miner (Version 3.2.12) [Software] (2011). Montreal: Provalis. Retrieved from: http://provalisresearch.com/products/qualitative-data-analysis-software/ Qi, J., & Levin, B. (2013). Assessing organizational efforts to mobilize research knowledge in education. Education Policy Analysis Archives, 21(2), 2-20. Quinn Patton, M. (1987). How to use qualitative methods in education. Thousand Oaks, CA: Sage. Reimers, F., & McGinn, N. (1997). Informed dialogue: Using research to shape education policy around the world. Westport, CT: Praeger. Ribbins, P., Bates, R., & Gunter, H. (2003). Reviewing research in education in Australia and the UK: Evaluating the evaluations. Journal of Educational Administration, 41(4), 423-444. http://dx.doi.org/10.1108/09578230310481649 Rosenblatt, A., & Tseng, V. The demand side: Uses of research in child and adolescent mental health services. Administration and Policy in Mental Health, 37(1-2), 201-204. http://dx.doi.org/10.1007/s10488-010-0300-5 Sabatier, P. (Ed.) (1999). Theories of the policy process: Theoretical lenses on public policy. Boulder, CO: Westview Press. Selby-Smith, C. (1999). The relationship between research and decision-making in education: An empirical investigation. In DETYA, The impact of educational research (pp. 551-572). Commonwealth of Australia, Canberra: Australia. Sheppard, B. (2012). Systems challenges to the sustainability of meaningful school reform. In G. Galway & D. Dibbon (Eds.), Educational reform: From rhetoric to reality (pp. 193-216). London: The Althouse Press. Sheppard, B., Galway, G., Wiens, J. & Brown, J. (2013, January). School boards matter: Report from the pan-Canadian study of school board governance. Canadian School Boards Association. Montreal. Retrieved from http://www.schoolboardsnl.ca/pdf/School-Boards-Matter-Report- Small.pdf Silverman, D. (2001). Interpreting qualitative data: Methods for analyzing talk, text and interaction. London: Sage. Slavin, R. (2002). Evidence-based education polices: Transforming educational practice and research. Educational Researcher, 31(7), 15-21. http://dx.doi.org/10.3102/0013189X031007015 Slavin, R. (2003). A reader's guide to scientifically based research. Educational Leadership, 60(5), 12–16. Smithers, A. (1995, September 8). Let usefulness be our yardstick for research. The Times Educational Supplement, 25. Statistics Canada (2011). Population, urban and rural, by province and territory. Retrieved from http://www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/demo62a-eng.htm Statistics Canada (2014). Headcount enrolments in public elementary and secondary schools, Canada, provinces and territories, 2007/2008 to 2011/2012. Retrieved from http://www.statcan.gc.ca/daily-quotidien/131204/t131204c001-eng.htm http://dx.doi.org/10.2304/power.2010.2.1.97 http://dx.doi.org/10.1080/03057640801889949 http://dx.doi.org/10.1080/01411920701582173 http://dx.doi.org/10.1108/09578230310481649 http://dx.doi.org/10.1007/s10488-010-0300-5 http://www.schoolboardsnl.ca/pdf/School-Boards-Matter-Report-Small.pdf http://www.schoolboardsnl.ca/pdf/School-Boards-Matter-Report-Small.pdf http://dx.doi.org/10.3102/0013189X031007015 http://www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/demo62a-eng.htm http://www.statcan.gc.ca/daily-quotidien/131204/t131204c001-eng.htm Education Policy Analysis Archives Vol. 23 No. 109 36 Stewart, D., & Shamdasani, P. (1998). Focus group research: Exploration and discovery. In L. Bickman & D. Rog (Eds.), Handbook of applied social research methods (pp. 505-526). Thousand Oaks, CA: Sage. Stone, D. (2002). The policy paradox: The art of political decision-making. New York, NY: W.W. Norton and Company, Inc. Stone, D., Maxwell, S., & Keating, M (2001). Bridging research and policy. (Discussion paper). International workshop funded by DFID, Warwick University. Retrieved from http://www.gdnet.org/pdf/Bridging.pdf Stokes, D. E. (1997). Pasteur's quadrant: Basic science and technological innovation. Brookings Institution Press. Strike, K. (2006). The ethics of educational research. In J.L. Green, G. Camilli, & P. Elmore, Handbook of Complementary Methods in Education Research (pp. 57-73). American Educational Research Association. Tapp-Neville, R. (2015). Collaborative research and the scholarship of engagement: Challenges for academic researchers (Unpublished Ph.D. dissertation). Memorial University, St. John’s. Canada Thomas, G., & Pring, R. (Eds.) (2004). Evidence-Based Practice in Education. Maidenhead: Open University Press. Thomson, P. (2004). Introduction. Journal of Education Policy, 19(3), 251-252. http://dx.doi.org/10.1080/0268093042000207593 Tooley, J., & Darby, D. (1998). Educational research: A critique. London: OFSTED. Tseng, V. (2012). The uses of research in policy and practice. SRCD Social Policy Report, 26, 3-16. Tseng,V., & Seidman, E. (2007). A systems framework for understanding social settings. American Journal of Community Psychology, 39(3-4), 217-228. http://dx.doi.org/10.1007/s10464-007-9101-8 Ungerleider, C. (2003). Failing our kids: How we are ruining our public schools. Ottawa: McClelland and Stuart. Ungerleider, C. (2004). Changing expectations, changing schools. Education Canada, 44(3), 18-21. Weiss, C. (1979). The many meanings of research utilization, Public Administration Review, September- October. Weissberg, R. (2002). Polling, policy and public opinion: The case against heeding the voice of the people. New York, NY: Palgrave Macmillan. http://dx.doi.org/10.1057/9780230109179 Willinsky, J. (2003). The strategic education research program and the public value of research. Educational Researcher, 30(1), 5-14. http://dx.doi.org/10.3102/0013189X030001005 Wilson, M.G., Lavis, J.N., Travers, R., & Rourke, S.B. (2010). Community-based knowledge transfer and exchange: Helping community-based organizations link research to action. Implementation Science, 5, 1-14. http://dx.doi.org/10.1186/1748-5908-5-33 Yohalem, N., & Tseng, V. (2015). Commentary: Moving from practice to research and back. Applied Developmental Science, 19(2), 117-120. http://dx.doi.org/10.1080/10888691.2014.983033 Zussman, D. (2003). Good policy starts with the facts. Public Policy Forum. Retrieved from http://www.ppforum.com/ow/ow_p_06_16_2003.htm http://www.gdnet.org/pdf/Bridging.pdf http://dx.doi.org/10.1080/0268093042000207593 http://dx.doi.org/10.1007/s10464-007-9101-8 http://dx.doi.org/10.1057/9780230109179 http://dx.doi.org/10.3102/0013189X030001005 http://dx.doi.org/10.1186/1748-5908-5-33 http://dx.doi.org/10.1080/10888691.2014.983033 http://www.ppforum.com/ow/ow_p_06_16_2003.htm Research and evidence in education decision -making 37 About the Authors Gerald Galway Memorial University, St. John’s, NL, Canada ggalway@mun.ca Dr. Galway is an associate professor in the Faculty of Education at Memorial University and a former education executive in the Government of Newfoundland and Labrador. For much of his career, he worked as a senior policy advisor providing research support and advice to decision- makers, cabinet ministers and other political elites. He is a former Associate Dean of Education and holds a number of distinctions including two first-place research awards from AERA. Dr. Galway holds a PhD in educational policy and leadership from the University of South Australia. He is the co-author of two edited books and has written numerous articles and reports. His research interests include evidence-based policy/decision making, knowledge mobilization, planning and demography, public sector accountability and educational reform. His most recent (2013) co-edited book, Inspiration and Innovation in Teacher Education, is published by Lexington. Bruce Sheppard Memorial University, St. John’s, NL, Canada bsheppar@mun.ca Dr. Sheppard is a professor in the Faculty of Education, Memorial University. Prior to his current position, he had a lengthy career in the public school system and has served as Director of Education in two school boards, and Associate Dean of Graduate Programmes and Research in Education at Memorial. Dr. Sheppard has been a recipient of a number of distinguished awards throughout his career including the CEA-Whitworth Award for contribution to research and scholarship in Canada and the European Distance Education Network Best Research Paper Award. His research interests include educational leadership, educational change and computer technology integration into classroom learning. He is author or co-author of more than 100 peer-reviewed publications, including a 2009 book entitled School District Leadership Matters. mailto:ggalway@mun.ca mailto:bsheppar@mun.ca Education Policy Analysis Archives Vol. 23 No. 109 38 education policy analysis archives Volume 23 Number 109 November 2nd, 2015 ISSN 1068-2341 Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for non- commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the author(s) or EPAA. EPAA is published by the Mary Lou Fulton Institute and Graduate School of Education at Arizona State University Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A2 (Brazil), SCImago Journal Rank; SCOPUS, SOCOLAR (China). Please contribute commentaries at http://epaa.info/wordpress/ and send errata notes to Gustavo E. Fischman fischman@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. http://www.doaj.org/ mailto:fischman@asu.edu https://www.facebook.com/EPAAAAPE Research and evidence in education decision -making 39 education policy analysis archives editorial board Editor Gustavo E. Fischman (Arizona State University) Associate Editors: Audrey Amrein-Beardsley (Arizona State University) Jeanne M. Powers (Arizona State University) Jessica Allen University of Colorado, Boulder Jaekyung Lee SUNY Buffalo Gary Anderson New York University Christopher Lubienski University of Illinois, Urbana-Champaign Michael W. Apple University of Wisconsin, Madison Sarah Lubienski University of Illinois, Urbana- Champaign Angela Arzubiaga Arizona State University Samuel R. Lucas University of California, Berkeley David C. Berliner Arizona State University Maria Martinez-Coslo University of Texas, Arlington Robert Bickel Marshall University William Mathis University of Colorado, Boulder Henry Braun Boston College Tristan McCowan Institute of Education, London Eric Camburn University of Wisconsin, Madison Michele S. Moses University of Colorado, Boulder Wendy C. Chi Jefferson County Public Schools in Golden, Colorado Julianne Moss Deakin University Casey Cobb University of Connecticut Sharon Nichols University of Texas, San Antonio Arnold Danzig California State University, San Jose Noga O'Connor University of Iowa Antonia Darder Loyola Marymount University João Paraskveva University of Massachusetts, Dartmouth Linda Darling-Hammond Stanford University Laurence Parker University of Utah Chad d'Entremont Rennie Center for Education Research and Policy Susan L. Robertson Bristol University John Diamond Harvard University John Rogers University of California, Los Angeles Tara Donahue McREL International A. G. Rud Washington State University Sherman Dorn Arizona State University Felicia C. Sanders Institute of Education Sciences Christopher Joseph Frey Bowling Green State University Janelle Scott University of California, Berkeley Melissa Lynn Freeman Adams State College Kimberly Scott Arizona State University Amy Garrett Dikkers University of North Carolina Wilmington Dorothy Shipps Baruch College/CUNY Gene V Glass Arizona State University Maria Teresa Tatto Michigan State University Ronald Glass University of California, Santa Cruz Larisa Warhol Arizona State University Harvey Goldstein University of Bristol Cally Waite Social Science Research Council Jacob P. K. Gross University of Louisville John Weathers University of Colorado, Colorado Springs Eric M. Haas WestEd Kevin Welner University of Colorado, Boulder Kimberly Joy Howard University of Southern California Ed Wiley University of Colorado, Boulder Aimee Howley Ohio University Terrence G. Wiley Center for Applied Linguistics Craig Howley Ohio University John Willinsky Stanford University Steve Klees University of Maryland Kyo Yamashiro Los Angeles Education Research Institute Education Policy Analysis Archives Vol. 23 No. 109 40 archivos analíticos de políticas educativas consejo editorial Editores: Gustavo E. Fischman (Arizona State University), Jason Beech (Universidad de San Andrés) y Jesús Romero Morante (Universidad de Cantabria) Armando Alcántara Santuario IISUE, UNAM México Fanni Muñoz Pontificia Universidad Católica de Perú, Claudio Almonacid University of Santiago, Chile Imanol Ordorika Instituto de Investigaciones Economicas – UNAM, México Pilar Arnaiz Sánchez Universidad de Murcia, España Maria Cristina Parra Sandoval Universidad de Zulia, Venezuela Xavier Besalú Costa Universitat de Girona, España Miguel A. Pereyra Universidad de Granada, España Jose Joaquin Brunner Universidad Diego Portales, Chile Monica Pini Universidad Nacional de San Martín, Argentina Damián Canales Sánchez Instituto Nacional para la Evaluación de la Educación, México Paula Razquin Universidad de San Andrés, Argentina María Caridad García Universidad Católica del Norte, Chile Ignacio Rivas Flores Universidad de Málaga, España Raimundo Cuesta Fernández IES Fray Luis de León, España Daniel Schugurensky Arizona State University, Estados Unidos Marco Antonio Delgado Fuentes Universidad Iberoamericana, México Orlando Pulido Chaves Instituto para la Investigacion Educativa y el Desarrollo Pedagogico IDEP Inés Dussel DIE-CINVESTAV, Mexico José Gregorio Rodríguez Universidad Nacional de Colombia Rafael Feito Alonso Universidad Complutense de Madrid. España Miriam Rodríguez Vargas Universidad Autónoma de Tamaulipas, México Pedro Flores Crespo Universidad Iberoamericana, México Mario Rueda Beltrán IISUE, UNAM México Verónica García Martínez Universidad Juárez Autónoma de Tabasco, México José Luis San Fabián Maroto Universidad de Oviedo, España Francisco F. García Pérez Universidad de Sevilla, España Yengny Marisol Silva Laya Universidad Iberoamericana, México Edna Luna Serrano Universidad Autónoma de Baja California, México Aida Terrón Bañuelos Universidad de Oviedo, España Alma Maldonado DIE-CINVESTAV México Jurjo Torres Santomé Universidad de la Coruña, España Alejandro Márquez Jiménez IISUE, UNAM México Antoni Verger Planells University of Barcelona, España Jaume Martínez Bonafé, Universitat de València, España José Felipe Martínez Fernández University of California Los Angeles, Estados Unidos Mario Yapu Universidad Para la Investigación Estratégica, Bolivia Research and evidence in education decision -making 41 arquivos analíticos de políticas educativas conselho editorial Editor: Gustavo E. Fischman (Arizona State University) Editores Associados: Rosa Maria Bueno Fisher e Luis A. Gandin (Universidade Federal do Rio Grande do Sul) Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Brasil Jefferson Mainardes Universidade Estadual de Ponta Grossa, Brasil Paulo Carrano Universidade Federal Fluminense, Brasil Luciano Mendes de Faria Filho Universidade Federal de Minas Gerais, Brasil Alicia Maria Catalano de Bonamino Pontificia Universidade Católica-Rio, Brasil Lia Raquel Moreira Oliveira Universidade do Minho, Portugal Fabiana de Amorim Marcello Universidade Luterana do Brasil, Canoas, Brasil Belmira Oliveira Bueno Universidade de São Paulo, Brasil Alexandre Fernandez Vaz Universidade Federal de Santa Catarina, Brasil António Teodoro Universidade Lusófona, Portugal Gaudêncio Frigotto Universidade do Estado do Rio de Janeiro, Brasil Pia L. Wong California State University Sacramento, U.S.A Alfredo M Gomes Universidade Federal de Pernambuco, Brasil Sandra Regina Sales Universidade Federal Rural do Rio de Janeiro, Brasil Petronilha Beatriz Gonçalves e Silva Universidade Federal de São Carlos, Brasil Elba Siqueira Sá Barreto Fundação Carlos Chagas, Brasil Nadja Herman Pontificia Universidade Católica – Rio Grande do Sul, Brasil Manuela Terrasêca Universidade do Porto, Portugal José Machado Pais Instituto de Ciências Sociais da Universidade de Lisboa, Portugal Robert Verhine Universidade Federal da Bahia, Brasil Wenceslao Machado de Oliveira Jr. Universidade Estadual de Campinas, Brasil Antônio A. S. Zuin University of York