Evaluation of Aboriginal Programs: What Place is Given to Participation and Cultural Sensitivity? The International Indigenous Policy Journal Volume 4 | Issue 2 Article 1 April 2013 Evaluation of Aboriginal Programs: W hat Place is Given to Participation and Cultural Sensitivity? Steve Jacob Laval University, steve.jacob@pol.ulaval.ca Geoffroy Desautels Laval University, geoffroy_desautels@hotmail.com Recommended Citation Jacob, S. , Desautels, G. (2013). Evaluation of Aboriginal Programs: What Place is Given to Participation and Cultural Sensitivity?. Th e International Indigenous Policy Journal, 4(2) . DOI: 10.18584/iipj.2013.4.2.1 Evaluation of Aboriginal Programs: W hat Place is Given to Participation and Cultural Sensitivity? Abstract Aboriginal populations in Northern Canada have, for many years, been confronted with socio-economic problems affecting their development. In the early 1990s, the Royal Commission on Aboriginal Peoples (1996) report concluded that it was important to integrate Aboriginal people into the management of public policies that concern them and to encourage their autonomy. In order to produce a quality evaluation that is useful in particular cultural contexts, measures have been developed to assure that the evaluation highly regards cultural sensitivity while integrating local participants in the evaluation process. This study, based on the systematic analysis of a non-probability sample of 27 program evaluation reports, presents an inventory of evaluation practice in Aboriginal contexts and estimates in what measure a culturally sensitive and participatory approach was applied. It was apparent that cultural sensitivity is gradually being integrated into Aboriginal program evaluation and that certain indicators show that there has been a positive evolution in this direction. Finally, the study shows an occasional recourse to participatory approaches, but this is not a strong tendency as systematically technocratic approaches are more broadly employed. Keywords program evaluation, cultural sensitivity, culturally-competent evaluation, participatory evaluation, Aboriginal governance, Aboriginal program evaluation in Canada Acknowledgments The authors would like to thank the Social Sciences and Humanities Research Council of Canada (SSHRC), which sponsored our research. Creative Commons License This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License. http://creativecommons.org/licenses/by-nc-nd/4.0/ http://creativecommons.org/licenses/by-nc-nd/4.0/ http://creativecommons.org/licenses/by-nc-nd/4.0/ http://creativecommons.org/licenses/by-nc-nd/4.0/ E v a l u a t i o n o f A b o r i g i n a l P r o g r a m s : W h a t P l a c e i s G i v e n t o P a r t i c i p a t i o n a n d C u l t u r a l S e n s i t i v i t y ? C o n t e x t a n d R e s e a r c h P r o b l e m Aboriginal people of Canada have, for a long time, been confronted with major socioeconomic problems that constitute a hindrance to their development and the blossoming of their communities. On this subject, numerous statistics indicate that Aboriginal people are disproportionally affected by school drop-out, problems related to drug and alcohol use, delinquency, and domestic violence in comparison with the rest of the Canadian population (Health Canada, 1998). In the North, these persistent difficulties, which can be explained by specific environmental, economic, politico-historical, and cultural conditions, make the governance of Northern Canada a major issue for the federal government (Institute on Governance, 2011). Meanwhile, as Banting and Kymlicka (2010) explained, Canada’s multicultural diversity necessarily implies flexibility, prudence, and openness in governance on the part of the federal government. Diversity policies in Canada today typically operate within three distinct “silos,” with separate laws, constitutional provisions, and government departments dealing with multiculturalism in response to ethnic diversity arising from immigration; federalism and bilingualism in response to the French fact; and aboriginal rights for First Nations. (p. 63) The Royal Commission on Aboriginal Peoples (RCAP), held at the beginning of the 1990s in the wake of the Oka Crisis and rejection of the Meech Lake Accord, permitted Canadians to truly take into consideration the challenges and weaknesses concerning their relationship with Aboriginal people. Conducted by members of First Nations communities and renowned jurists, the basic approach of the RCAP (1996) was based on Canadian historical data and calibration measures developed in other countries to help better understand relationships with Aboriginal peoples. The study demonstrated that the Government of Canada needs to become more aware of the knowledge of Aboriginal peoples and underlined the relevance of changing the existing power relationship in order to permit Aboriginal peoples to participate in both reflection and the decision-making process regarding issues that concern them. Moreover, in January 1998, in response to the conclusions of the RCAP, the Canadian government produced a document entitled, Gathering Strength, which consisted of a strategic plan of action to address the new priorities established for Aboriginal peoples. Notably, the plan suggests that: The development of stronger relationships with Aboriginal organizations is critical to improving the design, development, and delivery of programs and policies; programs designed to strengthen Aboriginal Governance are dedicated to enabling Aboriginal communities to make the transition to stable and accountable self-government. (Indian Affairs and Northern Development [IAND], 2000, p. 5) Following the conclusion of the RCAP, the virtues of a consultative process that favours greater autonomy for Aboriginal peoples has gained more and more recognition in research on Aboriginal governance. As underlined by Abele in reference to policies in matters of Aboriginal health, On the one hand, it helps Aboriginal people to familiarize themselves with the workings of the Canadian political system and to formulate an opinion; on the other hand, this collaboration 1 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 helps managers to familiarize themselves with the culture, the beliefs and the specific needs of Aboriginal peoples in matters of health, which opens a new era of collaboration. (cited in the Institute for Governance, 2011, p. 6) Other work, including a study focusing specifically on the development of Aboriginal communities in the Northwest Territories, also insists on the advantages of Aboriginal autonomy. Our rich cultural diversity, combined with our history, has led to a unique and fairly complex system of governance. While the implementation of self-government agreements will add to the complexity of the system, it is important to the achievement of Aboriginal peoples’ aspirations. It is also a critical step toward building a solid foundation for the territories’ political, social, cultural, and economic development. (Northwest Territories Government, 2004, p. 5) In this way, whether in Gathering Strength or in the literature on Aboriginal governance, we observe awareness and a change in attitudes relating to the relationship between Aboriginal peoples and the Government of Canada. Notably, the new “paradigm” prioritizes local participation and cultural sensitivity; two governance issues that are no stranger to the domain of program evaluation. In this regard, Jacob and Daigneault (2011) confirm that in order to act on the will for participatory governance, it will be necessary to review the evaluative process in order to integrate the maximum number of stakeholders, which will require use of a more systematic, participatory evaluation. Indeed, apart from an Aboriginal context, participatory evaluation has developed based on three distinct but complementary aims, including: (a) improving evaluation use; (b) favouring social justice and democratic representation; and (c) strengthening the validity of an evaluation’s conclusions. First, participatory evaluation is presented as an answer to the recurring problem of under-used evaluation. For example, elected and public managers ignore reports produced by evaluators when the time comes to make program decisions (Patton, 2008). In response to this weakness, many authors have claimed that the integration of participants (either somewhat or greatly affected by the program and its evaluation) favours the use of results (Briedenhann & Butts, 2005; Cousins & Earl, 1992, 1995; Cousins & Whitmore, 1998; Jacob, Bélanger, & Ouvrard, 2011; Patton, 2008; Wholey, 1994). This collaboration contributes to the orientation of evaluation, selection of an adequate methodology, and, ultimately, to the production of results matching the expectations of involved parties and, therefore, greater use of the results (Burke, 1998; Cousins & Shulha, 2006; Patton, 2008). Second, participatory evaluation can present a fundamentally political orientation by giving a voice to participants traditionally left outside the process. The motivation behind participatory evaluation, which notably finds its incarnation through “empowerment evaluation,” involves a redefinition of the role of the evaluator who then becomes a facilitator. This favours the success of the evaluation project (Eliadis, Furubo, & Jacob, 2011; Fetterman, 2000), the idea being to permit participants to develop their skills relating to evaluation, emancipate themselves in relation to the program, and, eventually, reach self-determination (Fetterman et al., 1995; Macaulay et al., 1997). The third motivation of participatory evaluation is epistemological. Participation is considered a medium by which to mobilize the knowledge of the involved parties, which translates into an evaluation strongly guided by the true needs of the program and leads to more valid, trustworthy, and representative conclusions (Brandon, 1998; Cousins & Earl, 1992, 1995; Cousins & Whitmore, 1998). 2 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 Participatory evaluation is not the only evaluative approach that focuses on public interest and integrating different stakeholders (Alkin & Christie, 2008), but it differs from other approaches, notably due to its attention to empowerment. Thus, in the context of Aboriginal governance, where the ties between the central government and the local authorities are particularly fragile, participatory evaluation sets itself apart from other evaluative processes in that it promotes the sharing of knowledge and development of evaluative skills to the benefit of the program’s decision-makers (Rossman, 2000). In fact, participatory evaluation helps to give a voice to actors whose views are not traditionally echoed in Aboriginal governance. Specifically, in participatory evaluation, “participants determine the evaluation’s focus, design and outcomes within their own socioeconomic, cultural and political environments” (Zukoski & Luluquisen, 2002). It is from this perspective that this type of assessment differs from traditional approaches. It is in line with the document, Gathering Strength, in that it encourages citizen empowerment and, ultimately, contributes to the development of Aboriginal autonomy. As stated by Putnam, Measures of participation in civic associations and political life point to similar conclusions. Engagement in civic associations has been celebrated as a means of building trust and enhancing the capacity for collective action in contemporary democracies. (cited in Banting & Kymlicka, 2010, p. 56) Culturally sensitive evaluation (Culturally Competent Evaluation), which has much in common with participatory evaluation, was developed as a way to favour the quality and precision of evaluation in different cultural milieus (Chouinard & Cousins, 2007). This recent interest in culture flows, based on several observations, underlines the difficulties relating to the development of unique and universal research frameworks (Barbier & Hawkins, 2012; Chouinard & Cousins, 2007; Guba & Lincoln, 2005). This approach accentuates the fact that evaluators need to show evidence of a very particular sensitivity regarding local culture so as to maximize their understanding of the context (Ebbutt, 1998; Smith, 1990). As stated by SenGupta, Hopson, and Thompson-Robinson (2004), “more broadly, because values are so integral to culture they are also integral to cultural dimensions of program design” (p. 8). Recognizing the importance of the cultural context in evaluation, the American Evaluation Association (AEA) has published a formal declaration on the cultural competence in the context of the evaluation. Now, the cultural competence of the evaluator or the evaluation team is no longer a suggestion, but rather a professional and ethical obligation in order to produce evaluation: “To ensure recognition, accurate interpretation, and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence” (AEA, 2011). More precisely, the AEA puts an emphasis on the skills of the evaluator or the team: The culturally competent evaluator (or evaluation team) must have specific knowledge of the people and place in which the evaluation is being conducted – including local history and culturally determined mores, values, and ways of knowing. The culturally competent evaluator draws upon a wide range of evaluation theories and methods to design and carry out an evaluation that is optimally matched to the context. In constructing a model or theory of how the evaluand operates, the evaluator reflects the diverse values and perspectives of key stakeholder groups. (AEA, 2011, What is cultural competence?, para. 3) 3 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 Finally, the AEA statement highlights the importance of cultural contextualization based on ethical principles, validity, and theory, criteria similar to the pragmatic, political, and epistemological dimensions of participatory evaluation. Consequently, whether through training or involvement with key partners in the evaluation, an evaluator sensitive to the context helps to produce an evaluation that provides concrete results that are adapted to the expectations and interests of the parties concerned (Friersen, Hood, & Hughes, 2002; Mertens, 2008). SenGupta et al. (2004), quoting Madison, also observed the sensitivity to the cultural context by involving stakeholders at key moments in the evaluation, which can significantly modify the spectrum of analysis and effectiveness of the research: […] The most important role program participants can play is in the program design and planning, and preferably also in problem definition. She convincingly argues that problem definition, a core activity that drives ameliorative program development, is often a dominant culture’s interpretation of reality that perpetuates the myth of the deficit model. (p. 8) Also, consideration of the context helps to avoid factual errors and, in this way, leads to even more valid conclusions (Nelson-Barber, LaFrance, & Trumbull, & Aburto, 2005). As underlined by Chouinard and Cousins (2007), the unique cultural context of Aboriginal evaluation has largely contributed to the development of this evaluative approach. In fact, any research grounded in solid contextual relevance will make for better or stronger conclusions. However, the specific Canadian context, where Aboriginal and non-Aboriginal relationships have always been motivated by the will of members of the dominant culture to colonize Aboriginal people in a paternalistic way, justifies the recourse to this evaluative approach (McKenzie, 1997; Potvin, Cargo, McComber, Delormier, & Macaulay, 2003). This conclusion is consistent with the AEA declaration in respect to the motivation to produce an evaluation that takes into account the cultural context: “In many minority and indigenous communities there is a history of inappropriate use of research or evaluation in ways that violated basic human rights. Vigilance to securing the well-being of individuals and their communities is essential” (AEA, 2011, Cultural competence is an ethical imperative, para. 2). In this sense, using a culturally sensitive evaluation approach goes a long way to repairing the lack of trust that flows from this paternalistic approach. For their part, Thomas and Bellefeuille (2006), during the formative evaluation of the Cross-Cultural Mental Health Program for Aboriginal People, also remarked that Canadian evaluation is not sufficiently inclusive of First Nations peoples in spite of the fact that the will to involve them is there. The authors assume that traditional approaches are undergoing a mutation and that, gradually, Aboriginal people are passing from the role of the evaluated to that of the evaluators: Once understood more as the object of research, Aboriginal peoples are increasingly being seen as researchers conducting research within Aboriginal knowledge traditions, using Aboriginal methodologies as well as methodologies drawn from interaction with non-Aboriginal intellectual traditions. (Thomas & Bellefeuille, 2006, p. 10) It is possible to observe more and more examples of evaluations of Aboriginal programs using participatory or culturally sensitive methods. The rarity of empirical studies on the subject demonstrates an opening for potential research: “We need greater methodological and philosophical clarity surrounding the use of participatory and collaborative approaches to cross-cultural evaluation” (Chouinard & Cousins, 2007, p. 54). 4 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 R e s e a r c h O b j e c t i v e s The objective of this article is to determine the consequences that arose from the conclusions of the RCAP (1996), in the context of the evaluation of Aboriginal programs. More precisely, it aims to verify how the principles of new Aboriginal governance, notably those proposed in the document Gathering Strength, apply to Aboriginal program evaluation in Canada. The study examines how evaluation reports integrate participatory principles and those of cultural sensitivity. Do we observe a true willingness to have Aboriginal peoples participate in evaluations and, if so, how does this translate concretely? Finally, do the evaluation reports present diverse elements attesting to an approach sensitive to Aboriginal people’s cultural reality? T h e o r e t i c a l F r a m e w o r k A n a l y t i c a l G r i d f o r P a r t i c i p a t o r y E v a l u a t i o n Daigneault and Jacob (2009) have developed, following from the work of Cousins and Whitmore (1998), an analytical framework helping to conceptualize and measure the participatory nature of a given evaluation: the Participatory Evaluation Measurement Instrument (PEMI). The PEMI will help us to better understand Aboriginal program evaluations: “The potential contribution of the framework for research is also important. A promising area of research relates to predicting and explaining the consequences of participatory approaches” (Daigneault & Jacob, 2009, p. 346). This instrument of conceptualization is a valuable tool to better understand the different dimensions of participatory evaluation and measurement. The measure of participatory evaluation is founded on three dimensions, including the diversity of participants, the scope of their involvement, and the real control that they exercise on the evaluation process. In this way, at the diversity level, we can denote four groups of actors potentially involved in the framework of a participatory evaluation: a. Decision-makers and policy designers, b. Those responsible for implementation and program delivery, c. Direct and indirect beneficiaries, and others affected, and d. Civil society and citizens. The diversity of actors is calculated on a scale from 0 to 1, where each group involved in the evaluation process is worth 0.25. If no group is represented, the evaluation receives a score of 0 for diversity. If a single group is represented, a score of 0.25 is given, and so on. The scope of involvement is related to the steps of the process in which the evaluation participants are potentially involved: a. The definition of issues and the development of evaluation questions, b. Results collection and data analysis, c. Judgement formulation and recommendations, and 5 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 d. The report and the dissemination of results. To be considered participatory, an evaluation must integrate the participants in at least one of these four phases. In the fashion of participant diversity, the scope of the involvement is calculated on a scale from 0 to 1 where each step is valued at 0.25. The third and final dimension of participatory evaluation is control. This represents the real hand that participants may have in the framework of conducting an evaluation. This dimension, which is more difficult to operationalize than the two preliminary dimensions of participatory evaluation, is accessible in the following ways: a. Exclusive control of the evaluator; b. Limited control by the participants; c. Control shared equally between, on the one hand, the evaluation participants and, on the other, the evaluator; d. Substantial participant control; and e. Exclusive control of the participants. The measure of this dimension only takes into consideration the steps where participation is involved. It is then logically possible to obtain a control score of 1, even if participation is limited to a single step. Finally, in the case where the level of control varies from one step to another, it becomes necessary to determine a representative score for the control for evaluation as a whole, for example, by calculating the average of the different levels of control in order to arrive at a single value. Therefore, it is in combining the results obtained in the three previously defined categories that we obtain the overall score and can determine if the evaluation is participatory or not and to what degree. The overall participation score represents the most trustworthy value obtained for one of the three measured dimensions. In this way, if one of the dimensions obtains 0.25, the overall participation score could not be superior to 0.25, regardless of the scores in the other dimensions1. C u l t u r a l S e n s i t i v i t y The second dimension of the study focuses on cultural sensitivity. It is a facet of evaluation relatively difficult to observe with a single reading of a report because there is no consensus on the definition of a culturally sensitive evaluation (Guba & Lincoln, 2005). That being said, the operationalization of the concept of culturally sensitive evaluation proposed by Frierson et al. (2005) is no doubt the most complete. These authors have established a list of criteria to observe in order to guide the handling of this type of evaluation. These criteria are expressed in the various phases of the evaluative process: 1 The present study relies on the original version of the PEMI (Daigneault & Jacob, 2009). The procedure to 6 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 a. Preparing for the Evaluation: Given the important role of the evaluation team, care should be taken in selecting its members. Those members, whenever possible, should be individuals who understand or who at least are clearly committed to being responsive to the cultural context in which the project is based. b. Identifying the Purpose(s) and Intent of the Evaluation: Careful documentation of the implementation of program activities is critical to making sense of the subsequent summative evaluation results. c. Framing the Right Questions: For an evaluation to be culturally responsive, it is critical that the questions of significant stakeholders have been heard and, where appropriate, addressed. d. Collecting the Data: Consequently, when collecting qualitative data directly from individuals, e.g., via interviews or observations, if those who are collecting and recording the data are not attuned to the cultural context in which the program is situated, the collected data could be invalid. e. Analyzing the Data: Having adequate understanding of cultural context when conducting an evaluation is important, but the involvement of evaluators who share a lived experience may be even more essential. The analyst of data gathered in a culturally diverse context may serve as an interpreter for evaluators who do not share a lived experience with the group being evaluated. To this end, a good strategy is the creation of review panels principally comprising representatives from stakeholder groups to examine evaluative findings gathered by the principal evaluator and/or an evaluation team. f. Disseminating and Utilizing the Results: Further, communications pertaining to the evaluation process and results should be presented clearly so that they can be understood by all of the intended audiences. (Frierson et al., 2005, pp. 65 - 72) There are other ways to operationalize the principle of cultural sensitivity. For example, the RCAP ensured that it conducted its research using an ethical code in matters of Aboriginal research where the accent is placed on the importance of proving cultural sensitivity. This document explains that researchers must listen to Aboriginal people, assuring that they get their say, that their points of view are heard, and that they have the right to see the conclusions of the research that concerns them (see RCAP, 1996, Appendix E). We can also return to the criteria proposed by the Interagency Advisory Panel on Research Ethics (PRE) of the research of the Government of Canada, which explains research that is sensitive to Aboriginal culture needs to, notably, respect the culture, the traditions, and the knowledge of the Aboriginal group; consult the members of the group with the appropriate expertise; and have the group participate in the conception of the project (Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, Social Sciences and Humanities Research Council of Canada, 1998). These are the criteria set out by Frierson et al. (2005), as well as the RCAP and the PRE, which provided the theoretical framework that helped to develop a coding grid for the evaluation of Aboriginal programs. The “instrument” section presents, in detail, the criteria retained for the ends of the report analysis. 7 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 M e t h o d o l o g y D a t a S o u r c e s a n d S a m p l i n g Examination of the levels of participation and cultural sensitivity was founded on a non-probability sample of evaluation reports that best represented the evaluation of Aboriginal programs in Canada. A report was considered to be “Aboriginal” if it was produced in the public program evaluation framework in an Aboriginal collective as defined by Indian and Northern Affairs Canada (INAC) (2011)2. Aboriginal collectives include: Inuit collectives in Nunavut, the Northwest Territories, the Yukon, Northern Quebec (Nunavik), and Labrador; Métis collectives; towns or municipalities that are not part of reserves or other traditional territory (for example, the Aboriginal collective of Winnipeg) (INAC, 2011). In addition to the INAC website, we consulted the websites of provincial and territorial governments and various private evaluation firms3 in Canada to determine the outlines of the “population” of Aboriginal evaluation reports. Although the initial objective was to assemble diverse reports relating to the various levels of government represented (national, provincial, and territorial), it became obvious that INAC was the most important player in these matters in Canada, with more than 60 published reports since 2003. At first glance, given the exploratory nature of this study and the goal to draw an overall portrait of the practices used in Aboriginal evaluation, it appeared a sample of 20 reports would be sufficient. However, in light of preliminary results, a supplementary sample round was conducted to improve the representativeness of the final sample. Figure 1 illustrates the approach that led to the selection of the initial sample, while Figure 2 illustrates complementary research. Initially, the objective was to assemble a number of diverse reports according to the level of government concerned, year of publication, and type of evaluation (internal or external and summative or formative). Because of the rarity of the reports issued from provincial and territorial governments, these selection criteria were applied uniquely in the case of the reports from INAC. Unlike the criteria applied in a qualitative manner, the selection process is similar to a stratified sample. On the surface, the sample seems to provide an adequate representation of program evaluations conducted in Canada in an Aboriginal context (Table 1). The sample is composed of 27 evaluation reports on Aboriginal programs (Table 2). The majority of the reports retained were produced for the federal ministry, INAC. For INAC’s part, they frequently participated in the evaluation of programs in collaboration with external resources; this is why nearly a third of the reports are evaluations using both internal and external resources. Another note about the sample is that the vast majority of reports retained present summative evaluations (74%). Finally, a very large majority of reports were produced after 2005 (85%)4. 2 Of note, the Ministry of Indian and Northern Affairs Canada (INAC) changed their name at the end of 2011 to the Ministry of Aboriginal Affairs and Northern Development Canada (AANDC). 3 We consulted the websites of eight evaluation firms suggested by the Canadian Evaluation Society: Goss Gilroy, Science-Metrix, Raymond Chabot Grant-Thornton, Capra International, Universalia, Johnson Research, Cathexis Consulting Inc., and Harry Cummings & Associates Inc. 4 The oldest retained evaluation report dates back to 2001. 8 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 F i g u r e 1 . S a m p l e S e l e c t i o n P r o c e s s - P h a s e 1 December 2010: Research in the evaluation report section of the INAC website (n = 66). December 2010: Google search using the keywords “Aboriginal evaluation,” “Canadian Aboriginal evaluation,” and “Aboriginal evaluation”. Read the 50 first entries for each of the keywords (n = 200). Rejected reports on the INAC website after having read the abstract (for reasons of being repetitive in terms of year and/or type of evaluation) (n = 40). Entries rejected after analyzing the description on Google because they were not evaluation reports (n = 190). Reports from the INAC website retained for a complete reading (n = 26). Google entries retained from a detailed reading (n = 10). Retained evaluations from the INAC website (n = 16). Retained evaluations from Google (n = 4). Reports rejected on the INAC website because they were not relevant (program covered by another evaluation already selected or program similar to an already retained evaluation) (n = 10). Reports rejected from Google because they were not Aboriginal program evaluations (the description was not sufficiently exhaustive to reject them at an earlier step) (n = 6). 9 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 F i g u r e 2 . S a m p l e S e l e c t i o n P r o c e s s - P h a s e 2 Evaluations retained from the Yukon, Nunavut, and Northwest Territories websites (n = 5). Evaluations retained from Google (n = 2). Reports retained from the Yukon, Nunavut, and Northwest Territories websites for a complete reading (n = 10). Reports retained from Google search for a detailed reading (n = 5). Entries rejected following analysis of the research description from the websites for the Yukon, Nunavut, and the Northwest Territories since they were not evaluation reports (n = 90). Entries rejected following the analysis of the research description in Google because they were not evaluation reports (n = 145). February 2012: Search using the keyword “evaluation” on the websites for the Yukon (n = 3,050), Northwest Territories (n = 2,490), and Nunavut (n = 0) territorial governments. Read the first 50 results (n = 100). February 2012: New Google search for different key words, such as “Yukon evaluation” (n = 7,700,000), “Northwest Territories evaluation” (n = 1,253,000), “Nunavut evaluation” (n = 2,620,000). Read the first 50 results (n = 150). Reports rejected from the websites of the Yukon, Nunavut, and Northwest Territories websites because they were not relevant (n = 5). Reports rejected from Google because they were either not Aboriginal program evaluations or non- relevant (n = 3). 10 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 T a b l e 1 . C h a r a c t e r i s t i c s o f t h e S a m p l e o f R e p o r t s L e v e l o f G o v e r n m e n t N u m b e r o f R e p o r t s % Federal 18 67 Nunavut 1 4 Northwest Territories 5 19 Yukon 3 11 Total 27 100 T y p e o f E v a l u a t i o n N u m b e r o f R e p o r t s % Internal 4 15 External 15 55 Mixed 8 30 Total 27 100 T y p e o f E v a l u a t i o n N u m b e r o f R e p o r t s % Summative 20 74 Formative 7 26 Total 27 100 P u b l i c a t i o n D a t e N u m b e r o f R e p o r t s % Before 2005 4 15 From 2005 to 2009 11 41 Since 2010 12 44 Total 27 100 11 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 T a b l e 2 . L i s t o f D o c u m e n t s i n S a m p l e Evaluation Number Report 1 Evaluation, Performance Measurement and Review Branch & Cona, D. (2010). Impact Evaluation of Treaty Commissions. Ottawa: Indian and Northern Affairs Canada. 2 Evaluation, Performance Measurement and Review Branch of INAC’s Audit and Evaluation Sector. (2010). Formative Evaluation of the Elementary/Secondary Education Program On Reserve. Ottawa: Indian and Northern Affairs Canada. 3 Evaluation, Performance Measurement and Review Committee. (2010). Summative Evaluation of the Contribution for Inuit Counselling in the South. Ottawa: Indian and Northern Affairs Canada. 4 Hicks J., & Akeeagok, D., Wilson, C., & Lovely, K. (2002). Building Nunavut Through Decentralization: Evaluation Report. Iqaluit: Nunavut Executive and Intergovernmental Affairs. 5 Departmental Audit and Evaluation Branch assisted by Rosemary Trehearne and Associates. (2005). Evaluation of the Family Violence Prevention Program for First Nations. Ottawa: Indian and Northern Affairs Canada. 6 Hanson G., Lloyd, R., & Lorimier, B. Evaluation of the Social Housing Program. (2004). Whitehorse: Yukon Housing Corporation and Canada Mortgage Housing Corporation. 7 Engage Strategies. (2010). Program Evaluation of Temporary Assignment Activity between Yukon First Nations and the Government of Yukon 2006 - 2009. Whitehorse: Workplace Diversity Employment Office Government du Yukon. 8 KPMG. (2001). Evaluation of the P3 Pilot Initiative: Final Report. Yellowknife: Government of the Northwest Territories. 9 DPRA Canada in collaboration with T. K. Gussman Associates. (2009). Evaluation of Community-Based Healing Initiatives Supported Through the Aboriginal Healing Foundation. Ottawa: Indian and Northern Affairs Canada. 10 TNS Canada Facts, & Harvey McCue Consulting. (2010). Formative Evaluation of the Post- Secondary Education Program. Ottawa: Indian and Northern Affairs Canada. 11 Evaluation, Performance Measurement and Review Branch, & external consultants. (2008). Evaluation of the Implementation of INAC's Gender-Based Analysis (GBA) Policy. Ottawa: Indian and Northern Affairs Canada. 12 Evaluation, Performance Measurement and Review Branch. (2009). Evaluation of the Advocacy and Public Information Program. Ottawa: Indian and Northern Affairs Canada. 13 Centre for Public Management Inc., & Evaluation, Performance Measurement and Review Branch. The report was written by T. K. Gussman Associates Inc. (2010). Summative Evaluation of the Capital Facilities and Maintenance Program. Ottawa: Indian and Northern Affairs Canada. 12 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 Evaluation Number Report 14 Evaluation, Performance Measurement and Review Branch Audit and Evaluation Sector. (2010). Evaluation of Grants for the Promotion of Fire Protection Awareness in Band and Federally Operated Schools. Ottawa: Indian and Northern Affairs Canada. 15 PRA Inc. (2010). Summative Evaluation of the Contributions Made to Indian Bands for the Registration Administration. Ottawa: Indian and Northern Affairs Canada. 16 Centre for Public Management. (2010). Implementation Evaluation of the First Nation Infrastructure Fund. Ottawa: Indian and Northern Affairs Canada. 17 Consortium of Goss Gilroy Inc., & Hollett and Sons. (2011). Evaluation of the Miawpukek First Nation Grant Agreement. Ottawa: Indian and Northern Affairs Canada. 18 Evaluation, Performance Measurement and Review Branch. (2008). Evaluation of INAC's Contaminated Sites Management Policy and Programming. Ottawa: Indian and Northern Affairs Canada. 19 Human Resources and Social Development Canada. (2007). Evaluation of the National Child Benefit Reinvestments Initiative. Ottawa: Indian and Northern Affairs Canada. 20 Unidentified consulting firm. (2009). Evaluation of the First Nations SchoolNet Program. Ottawa: Indian and Northern Affairs Canada. 21 Wildeor Wildlife Research & Consulting. (2008). Evaluation of the Klondike Soft Gold Program. Whitehorse: Yukon Department of Environment. 22 Zapf, M. K. (2004). An Evaluation of the Whole Child Project. Whitehorse: RCMP National Youth Strategy. 23 Departmental Audit and Evaluation Branch, & Goss Gilroy. (2005). Evaluation of the Cultural/Education Centres Program. Ottawa: Indian and Northern Affairs Canada. 24 Terriplan Consultants. (2009). Territorial Medical Travel Program Northwest Territories Evaluation. Yellowknife: Steering Committee - Pan Evaluation Northern Medical Travel Programs. 25 J. Carey Consulting Evaluation Plus Ltd. (2010). NWMOG Project Summative Evaluation. Yellowknife: Status of Women Council of the Northwest Territories. 26 R. A. Malatest & Associates Ltd. (2011). Evaluation of the Protection Against Family Violence Act. Yellowknife: Department of Justice, NWT. 27 Evaluation Directorate Strategic Policy and Research Branch Human Resources and Skills Development Canada. (2009). Summative Evaluation of the Employment Benefits and Support Measures Delivered Under the Canada-NWT Labour Market Development Agreement. Retrieved from http://www.ece.gov.nt.ca/files/TAB%205_NWT%20LMDA%20- %20Sum%20Evaluation%20Report_ENG%20w%20MR.pdf 13 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 I n s t r u m e n t The analysis of the reports follows a coding system with the help of a grid that is divided into three distinct sections. The first section of the grid makes reference to more general aspects of the evaluation reports: a. Report’s year of publication, b. Internal or external evaluation, c. Summative or formative evaluation, d. The ministry that implemented the evaluated program, and e. The level of government (federal, provincial, or territorial). This first section of the grid presents the descriptive characteristics of the evaluation reports that were retained for the sample. The second section of the coding grid helped us to judge the integration of the principles of culturally sensitive evaluation into the evaluations. These principles included, notably, if the evaluators were from the Aboriginal community, familiar with the evaluated program, had received any particular training to better understand Aboriginal culture, and whether members of the Aboriginal community were involved when the objective was defined. The research questions were then drafted and the results analyzed. Also, this grid permitted us to determine whether the evaluated program was well documented and if there were recommendations specific to the evaluated program’s beneficiaries and Aboriginal authorities. Finally, the third coding section measures the participatory nature of the evaluations with the help of a tool developed by Daigneault and Jacob (2009). The tool was used in the manner outlined in the theoretical section. Thus, for each report, we identified the different participants, identified in which evaluative phases they were involved, and established the overall control they exerted on the evaluative process. Assigning a score (from 0 to 1) for each dimension, we were able to establish an overall participation score for all reports in the sample. C o l l e c t i o n a n d A n a l y s i s P r o c e d u r e s The second author, in consultation with the first author when necessary, performed the coding. To assure the reliability and the validity of the results, we followed a three-step coding process. A first reading allowed the author to become familiar with the style and content of the reports in the sample. During this stage, no notes were taken; the only objective was to understand the context of the evaluation. The coding grid was then rigorously applied to each of the evaluation reports during second reading. The objective was to identify all the information necessary to complete the grid. After reading and re-reading a report, if the necessary information proved to be missing, we reported it as not available. Finally, a few weeks after the first coding was done, each report was the object of a second round of coding, which was compared to the first. This “test-retest” validation procedure aimed to assure the stability of the coding over time. Coding took an average of 6.5 hours for each of the reports (4.5 hours for the first round of coding and 2 hours for the second). It is important to point out that certain elements, such as the members of the evaluation team, justification of the methodological choices, and progress of the evaluation, were not always presented in the evaluation reports. This observation also 14 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 reflects findings of the Treasury Board Secretariat in its examination of the quality of federal evaluations (cited in Daigneault, 2010). R e s u l t s S e n s i t i v i t y t o A b o r i g i n a l C u l t u r e Does Aboriginal program evaluation in Canada accord particular attention to the cultural realities of Aboriginal people? Table 3 presents the principal indicators of cultural sensitivity and their application to the reports in the sample. T a b l e 3 . P o r t r a i t o f A n a l y z e d R e p o r t s – S e n s i t i v i t y t o A b o r i g i n a l C u l t u r e C r i t e r i a o f S e n s i t i v i t y t o A b o r i g i n a l C u l t u r e N u m b e r o f R e p o r t s w h e r e t h e C r i t e r i a i s M e n t i o n e d % 1. Presence of an evaluator from the Aboriginal community 5 19 2. Presence of an evaluator familiar with the evaluated program 13 48 3. Presence of an evaluator having taken specific training to better understand Aboriginal culture 11 41 4. Consultation of participants familiar with the program at the point of defining objectives or evaluation questions 12 44 5. Well documented set-up of the program 25 93 6. Consultation with participants familiar with the program at the point of analyzing the results 8 30 7. Recommendations destined to Aboriginal authorities 16 59 8. Recommendations destined to evaluated program’s beneficiaries 9 33 9. Number of evaluations meeting Criteria 1, 2, 3, or 4 (criteria demonstrating preliminary knowledge of the environment or if approaches were taken with local participants so as to better understand the environment) 15 56 10. Number of evaluations meeting Criteria 1, 2, 3, or 4 A N D Criteria 5, 6, 7, and 8 (this conjunction of criteria illustrates the number of evaluations where there was preliminary knowledge of the context and where the approach was done while respecting the totality of the pre-defined criteria of a culturally sensitive evaluation) 5 19 15 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 We immediately noticed that a weak majority of evaluation reports (56%) respect Criteria 1, 2, 3, or 4. This suggests that, when all is said and done, evaluation frequently happens without the evaluators having any of the resources that foster their understanding of the Aboriginal context. In nearly half of all cases, the evaluator’s experience with the Aboriginal context is summed up in the initial meeting with the evaluation sponsors to discuss the project’s terms of reference. In return, the explanation of the activities of the evaluated program is well illustrated in the vast majority of the evaluation reports (93%). In this regard, the principal steps of program implementation are clearly described in practically all of the evaluation reports. Also, key participants are rarely consulted when the collected data are being analyzed. Finally, it must be noted that rarely do the evaluation reports provide recommendations focusing on the evaluated program’s beneficiaries (33% of the cases) and the recommendations aimed at the Aboriginal authorities are not systematic (59% of cases). This phenomenon can be explained by the fact that evaluation reports are more often aimed at providing accounts to the evaluation sponsors, such as the federal government. The preceding indicators sketch an overall picture of the more or less culturally sensitive nature of Aboriginal program evaluation, a portrait that may be completed with the help of report extracts. Further, certain evaluations are conducted by Aboriginal evaluators or evaluators who are particularly knowledgeable of the Aboriginal cultural context. This is notably the case for the evaluation of the Whole Child Project in the Yukon. In the report, the expertise of the principal evaluator is presented unequivocally: Principal Investigator for this study was Dr. Michael Kim Zapf (CRILF Board Member and Professor of Social Work at the University of Calgary) who has published extensively on issues of social service delivery in remote Northern regions. Having previously lived and worked in Whitehorse, he had direct experience in the relevant context. (Table 2, Evaluation 22, 2004, p. 4) This seasoned evaluator, perfectly conversant with the context in which the evaluation process would evolve, represents the evaluator-type for more than half of the analyzed evaluations. In other cases, evaluators have recourse to local resources that allow them to better understand the cultural context. In this way, evaluators frequently surround themselves with key actors at the heart of Aboriginal programs and communities, notably taking the form of a supervisory committee composed of the principal decision-makers and local personnel: The project was guided by a Steering Committee of territorial and Health Canada representatives, who met with the evaluation team and the project manager as required to clarify goals, provide direction, approve the methodology, and monitor evaluation activity. (Table 2, Evaluation 24, 2009, p. 2) Otherwise, when evaluators were not familiar with the program or the cultural context, and if they had not received training to aid their comprehension of the milieu, particular attention was given to the evaluation methodology so as to “hear” the Aboriginal cultural sensitivities. Often, this methodological sensitivity was expressed in the form of concern over providing easily understandable documentation for the population involved in the evaluation framework. As is the case, for example, in the evaluation on the initiatives of community healing offered by the Aboriginal Healing Foundation: 16 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 The evaluation methodology was adapted for this project to ensure cultural appropriateness of methods and the safety and wellbeing of participants. There were three primary ways of doing this: one was to provide for translation where needed to enable interviewees to participate in their Aboriginal language if desired. (Table 2, Evaluation 9, 2009, p. 12) Moreover, considering the language barrier is a recurring difficulty in the evaluation of Aboriginal programs, evaluators do not hesitate to seek out local resources to narrow this gap. For example, in the framework of a study on the decentralization of Nunavut, the evaluator explains his recourse to local experts: “The researcher was supported by local research assistants familiar with the issues and fluent in Inuktitut. As a result, all those who wished to be heard were able to share their views” (Table 2, Evaluation 4, 2002, p. 5). It must be understood that, in the Aboriginal context, the role of local expertise and sensitivity to local issues goes beyond simply finding an interpreter. It also encourages evaluators to adapt their speech, which contributes to greater trust on the part of the stakeholders involved at various stages of the evaluation process. Though not all evaluators use resources to help understand the cultural context, a very large majority of the evaluation reports analyzed demonstrated that the evaluator had a good understanding of the evaluated program. Moreover, most evaluations were founded on the reading and analysis of principal program documents: “The document review was used to familiarize the project team with the program and its context” (Table 2, Evaluation 23, 2005, p. 5). Nelson-Barber et al. (2005) confirm that it is important for the evaluator to have recourse to local experience and not to hesitate to immerse themselves in local culture: “If possible, evaluators should engage in community activities, such as graduation ceremonies and dinners for the elders in the tribe, or funerals for honoured tribal members” (p. 74). The idea behind this is that each evaluated environment possesses local particularities and that, for an evaluator unfamiliar with the context, it can be difficult to arrive at even-handed and appropriate conclusions. Moreover, the analyzed reports demonstrate that, very often, the availability of local skills is written into the terms of reference of the evaluation contract. Consequently, the work of the evaluator is facilitated and the evaluated organization is assured the work will be conducted in a context where there is a constant awareness of the local cultural realities. I n W h a t M e a s u r e i s A b o r i g i n a l E v a l u a t i o n P a r t i c i p a t o r y ? At first glance, we notice that a little more than half (52%) of the evaluations of Aboriginal programs integrate elements of the participatory approach that is operationalized in different ways at the practice level (Table 4). We note, however, that no evaluation is considered to be “very” or “totally” participatory. The dimensions of participant diversity and control over the evaluative process have indeed pulled down the overall participation scores. In a large majority of cases (86%), just one or two groups of actors were represented out of a possible four (Table 5). Among the groups represented, the decision-makers and policy designers (86% of cases), or even those responsible for implementation or program delivery (79% of cases), are those who are habitually associated with the evaluative process. 17 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 T a b l e 4 . D i s t r i b u t i o n o f t h e E v a l u a t i o n s a c c o r d i n g t o t h e L e v e l o f P a r t i c i p a t i o n L e v e l o f P a r t i c i p a t i o n N u m b e r o f R e p o r t s % Non-participatory 13 48 Somewhat Participatory 9 33 Average Participation 5 19 Very Participatory 0 0 Totally Participatory 0 0 Total 27 100 T a b l e 5 . P a r t i c i p a t i o n D i v e r s i t y i n A b o r i g i n a l E v a l u a t i o n s Number of Different Groups Represented in the Evaluation Number of Occurrences among Participatory Evaluations % of Participatory Evaluations 1 Group 4 29 2 Groups 8 57 3 Groups 1 7 4 Groups 1 7 Total 14 100 Type of Participants Represented in the Evaluation Number of Occurrences among Participatory Evaluations % of Participatory Evaluations Decision-makers and Policy Designers 12 86 Manager for Program Implementation and Delivery 11 79 Direct and Indirect Beneficiaries, Others Affected 1 7 Civil Society and Citizens 3 21 18 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 Table 6 illustrates that, when evaluations are participatory, the involved parties generally participate during the two first phases. More often than not, the formulation of recommendations and the dissemination of results are the responsibility of the evaluators. Finally, the control for the evaluative process is either limited for the involved parties (64% of cases) or shared equally between the evaluator and these participants (36% of cases). In the evaluations examined, not once did we observe a greater level of control for participants (Table 7). Indeed, the integration of participatory approaches is relatively common in Aboriginal program evaluation; half of the evaluation reports in the sample had only limited or moderate participatory approaches. As a general rule, it is the pragmatic and epistemological aspects – more than the political aspects – which seem to motivate the systematic recourse to participatory evaluation. Indeed, the perspective of favouring community development does not seem to be the principal motivation in the framework of Aboriginal program evaluation. For example, we noted a few passages from different reports illustrating the common relationship between evaluators and local participants: “… the evaluation team, in consultation with the client identifying key INAC staff along with the contact information of each identified participant,” (Table 2, Evaluation 16, 2010, p. 8) or even: “The key informers participating in this study were chosen on the basis of the recommendations of the consultative committee regarding the people that know the most on the programs and INAC educational policies and First Nations’ problems in this domain” (Table 2, Evaluation 2, 2010, p. 11). These two examples demonstrate the integration-type of the involved parties in the evaluation of an Aboriginal program. The objective is then to integrate them so as to favour the understanding of the program, bring together the key actors, and, ultimately, produce conclusions compatible with their efforts in relation to the evaluated program. In return, the perspective of giving a voice to participating actors, notably during the steps of defining objectives and developing evaluation questions, does not seem to be a priority for Aboriginal program evaluation. Beyond the observed levels of participation, we have brought to light two participatory types or modes: (a) the communication channel, and (b) the management and supervisory committee. In the first type, the participant acts as a facilitator serving the primary role as the transmission cog between the program’s actors and the evaluators. For example, a report on the Infrastructure Funds of First Nations summarizes well how this type of participation operates, explaining the role assumed by the participant: Identify key informants; Identify key documents, literature, and data sources; Determine the adequacy of data collection; Identify possible case study communities; Identify local/community contact to facilitate with application of lines of inquiry; Obtain appropriate insight into FNIF’s successes and limitations. (Table2, Evaluation 16, 2010, p. 8) 19 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 T a b l e 6 . E x t e n t o f I n v o l v e m e n t i n A b o r i g i n a l E v a l u a t i o n s S t e p s w h e r e I n v o l v e d P a r t i e s a r e R e p r e s e n t e d i n P a r t i c i p a t o r y E v a l u a t i o n s N u m b e r o f O c c u r r e n c e s % o f R e p r e s e n t a t i o n Definition of the Issues and the Development of Evaluation Questions 12 86 Results Collection and Data Analysis 12 86 Formulation of Judgments and Recommendations 8 57 Report and Results Dissemination 7 50 T a b l e 7 . D i s t r i b u t i o n o f C o n t r o l i n t h e F r a m e w o r k o f P a r t i c i p a t o r y E v a l u a t i o n s Level of Control Number of Occurrences % among Participatory Evaluations Exclusive Control given to the Evaluator N/A N/A Limited Control given to Evaluation Participants 9 64 Control Shared Equally between Evaluation Participants (on the one hand) and the Evaluator (on the other) 5 36 Substantial Control given to Evaluation Participants 0 0 Exclusive Control given to Evaluation Participants 0 0 In this case, the role is one of orienting, but especially supporting, the evaluation team in the framework of their approach. This participation has also been observed in the evaluation of the Klondike Soft Gold Program: “We particularly thank the YTA’s manager, Mara Spricenieks, who cheerfully responded to our requests, and made extra efforts to track down documents and information that helped us in this work” (Table 2, Evaluation 21, 2008, p. vii). It is also evident in the framework of the Whole Child Project: “Ms. Crystal Pearl-Hodgins, WCP Community Coordinator, found time in her busy schedule to gather written materials, share her perspective and experiences with WCP, and direct the researcher toward those community people whose stories were vital to the report” (Table 2, Evaluation 22, 2004, p. ix). It is the integration of the participants that is most commonly observed in the evaluation of Aboriginal programs. When the participants are considered more as facilitators for evaluators, their participation tends to be limited to the first two steps of the evaluative process. Indeed, in this type of evaluation, cases where the actors participated in the entire process were rare. When this was the case, participant control was relatively moderate. For example, the consultative committee for the evaluation framework on community healing initiatives, played a limited role that hinged more on revising the process than on 20 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 decision-making: “The evaluation team consulted a consulting group from the FADG, government experts and independent specialists who revised a detailed report on the methodology and all the data collection tools as well as the preliminary findings of the evaluation” (Table 2, Evaluation 9, 2009, p. 12). As a general rule, it is during these first two steps – either the preparation of the draft and research questions or data collection and analysis – where parties were most involved. The other type of participation observed is more oriented towards the needs of the program decision- makers. In these circumstances, the spread of participation is more complete, notably concerning Steps 3 and 4 of the evaluation process. In these cases, the reports justify collaboration with involved parties, not with a view to develop the skills of the actors involved in the program, but more to answer to the epistemological and pragmatic perspectives of evaluation. To this effect, the evaluation report of the postsecondary education program is unequivocal: “The purpose of the Advisory Committee was to work collaboratively to produce evaluation products, which are reliable, useful and defendable to both internal and external stakeholders” (Table 2, Evaluation 10, 2010, p. 10). The same holds true for the evaluation of the Protection Against Family Violence Act where a committee guarantees that the evaluation follows the right objectives: “The evaluation approach taken for the current project relied on multiple lines of evidence and followed the recommendation of the Advisory Committee to focus on the influence of social responses on the achievement of the goals and objectives of the PAFVA and the EPO process” (Table 2, Evaluation 26, 2011, p. 10). This involvement was also observed in the framework of the evaluation on the effect of the commission of treaties: “The broad mandate of the Advisory Group was to provide advice and guidance on the conduct of the study for EPMRB’s consideration at key stages of the evaluation process” (Table 2, Evaluation 1, 2010, p. 6). While such participation is relatively surprising, considering the efforts necessary to integrate involved parties possessing little or no experience in matters of evaluation, it is largely explained by the fact that it is most often program managers or personnel who are included. Consequently, these consultative committees do not attract many participants familiar with the implementation of the evaluated program. In other cases, however, the participatory aspect can be summed up in a simple final approval of the work of the evaluator, a sort of seal of quality, as in the framework of the evaluation concerning the Infrastructure Funds: In line with EPMRB’s Quality Control Process and Standards and its engagement policy, a working group established for the purposes of this evaluation assisted by facilitating access to information, reviewing and validating the draft final report. The Working Group comprised of managers and program officers (Headquarter and regional) from FNIF and from the AFN. A validation session of the preliminary findings and the final report occurred with program representatives. (Table 2, Evaluation 16, 2010, p. 13) Finally, on very rare occasions, the reports presented a participatory approach integrating the key participants throughout the process. This was notably the case in the evaluation of the Contribution for Inuit Counselling in the South: In line with EPMRB’s Engagement Policy, TI was actively engaged in the evaluation. TI provided information on, and access to, key informants and documentation. The organization also reviewed the evaluation issues and questions, methodology report and final report. Officials from both NAO and the IRS provided input on the Terms of Reference, as well as comments on 21 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 the methodology report, preliminary findings and the final report, in line with EPMRB’s Quality Assurance Strategy. (Table 2, Evaluation 3, 2010, p. 4) It is here, in one of these rare cases, where it became possible to observe a real integration of the parties involved in the evaluation process. The most common recourse to Aboriginal participants was more pragmatic in nature or was intended to confirm the validity of the process. D i s c u s s i o n The objective of this study was to examine to what extent, in Canada, the evaluation of Aboriginal programs integrates the elements of cultural sensitivity and is a participatory process. At first glance, the results suggest evaluations are of varying sensitivity to the local (Aboriginal) culture. Moreover, a minority of the criteria for cultural sensitivity is respected. For example, few Aboriginal evaluators or evaluators familiar with the program take part in program evaluation. This absence might be compensated for, in part, if the evaluators had undergone training to improve their understanding of the context or, even, if they had systematic recourse to local participants, but this is far from reality in most cases. Indeed, the approach of understanding the context essentially seems to operate in the framework of readings presenting the broad themes of the evaluated program. This documentary research contributes to a better understanding of evaluation issues, but does not suffice to guarantee a sensitive evaluation in the cultural context. Certainly, the evaluators adequately present the program they are charged to evaluate and demonstrate their understanding of said program (93%), but in conducting their analysis from a strictly external perspective, without recourse to informed participants, they increase their chances of producing an evaluation with erroneous conclusions. On this subject, Nelson- Barber et al. (2005) believe that a heightened cultural sensitivity helps to draw more valid conclusions and avoid the factual errors related to an evaluator in an unfamiliar environment: “Without specific understandings of the context in which a program is being implemented, for example, evaluators are likely to miss important information that can shed light on why a program has particular outcomes or impact on a community” (p. 62). In a parallel manner, we note the recommendations are essentially addressed to the governmental authorities, which generally sponsor the evaluation. In the view that the Government of Canada seeks to favour autonomy and collaboration with Aboriginal communities, it is surprising that a weak majority of evaluations present recommendations for the Aboriginal authorities (59%) and a minority of the evaluations make recommendations for the beneficiaries of evaluated programs (33%). It is all the more surprising considering it is an explicit criterion of the code of ethics in matters of Aboriginal research: “Whenever possible, research should support the transfer of skills to individuals and increase the capacity of the community to manage its own research” (RCAP, 1996, Appendix E, Community benefit, para. 3). Consequently, it is paramount that the evaluations present results and, therefore, recommendations that address local communities; otherwise, it seems difficult to anticipate the adoption of the results by the communities and, in this way, favour the autonomy of these people in view of potential research. At the moment, when we look at the recommendations, there still seem to be cleavages between program administrations that are sometimes centralized in the federal government and the local administration of Aboriginal communities. There is one essential condition in order to produce a culturally sensitive evaluation: “In addition to the need to recognize a paradigm shift in Aboriginal research, it is important that Aboriginal communities be given the opportunity to decide that the research priorities should be for their communities” (Thomas & Bellefeuille, 2006, p. 10). In the end, despite certain gaps, such as few recommendations to the Aboriginal administrators and limited 22 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 involvement of local participants, it is possible to observe an emerging cultural sensitivity in Aboriginal program evaluation. Though the use of culturally sensitive evaluation is not systematic, considering the importance given to a rigorous program’s documentation and the fact that we see, more often than not, the necessity to have recourse to local skills written in the form of an obligation in the terms of reference of the evaluation contract, we can conclude that some efforts seem to be undertaken in order to favour the autonomy and the integration of Aboriginal people. Following the example of cultural sensitivity, the integration of participatory approaches in the evaluation of Aboriginal programs is, in sum, relatively limited. Local populations hold little or no decision-making power over the evaluation process. It is true that the integration of the whole list of involved parties concerned with the evaluation process is sometimes difficult, whether for financial or logistical reasons (Bradley, Mayfield, Mehta, & Rukonge, 2002). However, it is nevertheless possible to observe the recourse to different participatory methodologies in Aboriginal programs. Indeed, Aboriginal participatory evaluation seems to favour the emergence of two participatory types or modes. First, the facilitating participant, who is essentially an interpreter at the service of the evaluator, is often represented by decision-makers or first-line program personnel, matching the theories stipulating that participatory evaluation aims to promote the use and the quality of the evaluation. Interestingly, these participants were frequently involved in the two first phases of the evaluation, which is surprising considering that many authors (Cousins & Earl, 1995; Cousins & Whitmore, 1998) say the involved parties have a tendency not to involve themselves in data collection, preferring instead to leave the more technical tasks to the evaluators. However, it is possible to assume that there is a particular reality in the context of evaluation of Aboriginal programs that can be largely explained by the numerous barriers, notably those of language and distance, which can exist between the evaluator and the respondents in the evaluated sphere. Furthermore, Brandon (1998) summarizes well how involving participating parties can contribute to the achievement of practical and epistemological objectives of participatory evaluation: By tapping the expertise of all appropriate stakeholder groups, […] evaluators can take steps calculated to enhance evaluation validity. Evaluators’ confidence in the credibility of their studies should be improved, and, most likely, they will find that their studies will receive greater use. (Brandon, 1998, p. 334) Otherwise, the second type of evaluator involved acts in a similar fashion to the decisional power. This active participant has, more often than not, the right to observe each step of the process. In this case, this is an actor interested principally in the results of the evaluation. The integration of this type of actor is a sufficient criteria to consider the evaluation to be participatory, although it is important to underline here that, in these circumstances, the interests represented are more those of program decision-makers and managers. It is, therefore, difficult to speak of a complete autonomy of Aboriginal communities during the evaluation process. In fact, even if there is a history of promoting initiatives to develop participatory approaches favouring the empowerment of Aboriginal peoples, it is only recently that this inclination has been officially listed as an objective for Aboriginal governance (Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, Social Sciences and Humanities Research Council of Canada, 1998; IAND, 2000). At this stage, it is, therefore, normal to observe that the great majority of evaluations are still managed by an evaluator. This might not be, except given the goal of the former is a collective aimed at self-determination. 23 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 In the end, the epistemological and pragmatic dimensions of participatory evaluation seem to occupy a place of choice in the Aboriginal context, although, at first glance, there seems to be little emphasis on the political dimension of participatory evaluation. Emancipation, self-determination, and autonomy do not emerge as priorities in Aboriginal program evaluation, in spite of the fact they are the priorities of Aboriginal governance and were, thereby, justifying the use of a participatory approach rather than traditional approaches. In the context where the political dimension of participatory evaluation does not appear at the heart of Aboriginal program evaluation, it must be concluded that the objectives of the RCAP and Gathering Strength have not yet been achieved. However, recourse to certain methods benefitting the understanding of the environment by the evaluator contributes, in an important way, to the production of a just and clarified evaluation that will ultimately serve the cause of local populations, assuring the provision of rigorously evaluated public programs. F u t u r e O r i e n t a t i o n s a n d L i m i t s The results observed through this research allow for a better understanding of how to articulate and develop evaluation in an Aboriginal context, where evaluation is traditionally carried out by the dominant culture seeking to better understand a minority culture. This inherent reality in any multicultural society means that our findings may apply to other societies with similar conditions. As mentioned by Kirkhart (1995), today’s evaluators must be aware of the cultural context in which they operate: “We as evaluators need to have a clearer understanding of the ways in which multicultural influences shape and are shaped by our work” (p. 1). We found that in Canada there appears to be a moderate, but growing, willingness to integrate the principles of culturally responsible evaluation, and, in parallel, we observed that participatory evaluation is mainly pragmatic and epistemological. Despite this, our results allow us to see the challenges that remain to be lifted for these different approaches to establish themselves in the context of the evaluation of Aboriginal programs. The challenges of multiculturalism are not limited to Canada; in the United States, the AEA deemed it necessary to publish an official declaration calling for cultural sensitivity on the part of evaluators and greater awareness of the virtues of involving local stakeholders. In multicultural contexts, where culturally responsible and participative approaches are increasingly encouraged and recognized as approaches that produce more ethical and valid results, it would be interesting to compare how they are integrated in comparison with the results of our study. Are there situations in which the political aspect of participatory evaluation and the ethical challenges of culturally responsible evaluation are favourable? Finally, it is important to underline the limits of this research. Indeed, despite the results that we were able to obtain and the qualitative observations that we conducted, this analysis is not supported by a large-scale sample, nor is it a statistically representative one. This research should be taken for what it is; that is to say, an exploratory step aimed at verifying how participatory and culturally sensitive approaches are integrated into the evaluation of an Aboriginal program. Moreover, even when the reports were available, they did not always reveal everything concerning the actors involved and the events that happened as a result of the evaluation. In many cases, the reports did not present the profiles of the evaluators, the details of the methodology used, or the contributors consulted. The results that we presented rely on the elements reported in the evaluation report. On certain occasions, the evaluation process happened in other circumstances and involved other actors who were not mentioned in the reports that we consulted. For future research, it would be useful to take into consideration the 24 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 recommendation of Daigneault (2010) who suggested, “contacting the evaluation services of the involved organizations, submitting to them, for example, the detailed evaluation of the reports so as to complete the missing information and validate the judgments made on the main people concerned” (p. 208, translated). In this way, a more thorough research integrating the interviews with the involved participants at the heart of these evaluations or even the authors of these reports could furnish us with the details that we were not able to obtain by relying solely on documentary sources. It would be interesting to know the extent of stakeholder involvement and the extent to which they benefitted from their involvement. A methodology based on interviews would promote this type of understanding, an option that is not possible using the current research design. Additionally, while the study sketches a portrait of the evaluation of Aboriginal programs, this is not compared to other evaluations of Canadian programs. In these circumstances, it seems relevant to pursue this work on a larger scale, notably proceeding to the analysis of reports at different levels of Aboriginal governments, but also in comparing the results with analyses on evaluation in other specific communities. What can be said of participatory Aboriginal evaluation in the United States and Australia? These are emerging research paths that remain very under-explored, and the evaluation community would benefit greatly from broader knowledge. 25 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 R e f e r e n c e s Alkin, M., & Christie C. (2008). An evaluation theory tree revisited. In M. Alkin (Ed.), Evaluation roots (pp. 381 - 392). Thousand Oaks, CA: Sage Publications. American Evaluation Association (AEA). (2011). Statement on cultural competence in evaluation. Retrieved from http://www.eval.org/ccstatement.asp Banting, K., & Kymlicka W. (2010). Canadian multiculturalism: Global anxieties and local debates. British Journal of Canadian Studies, 23, 43 - 72. Barbier, J.-C., & Hawkins, P. (Eds.). (2012). Evaluation cultures sense-making in complex times. New Brunswick, NJ: Transaction Publishers. Bradley, J. E., Mayfield, M. V., Mehta, M. P., & Rukonge, A. (2002). Participatory evaluation of reproductive health care quality in developing countries. Social Science and Medicine, 55(2), 269 - 282. Brandon, P. R. (1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations, American Journal of Evaluation, 19, 325 - 337. Briedenhann, J., & Butts, S. (2005). Utilization-focused evaluation. Review of Policy Research, 22(2), 221 - 243. Burke, B. (1998). Evaluation for a change: Reflections on participatory methodology. New directions for evaluation, 1998, 43 - 56. Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, Social Sciences and Humanities Research Council of Canada. (1998). Section 6 – Research involving Aboriginal peoples. In Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (pp. 6.1 – 6.3, with 2000, 2002 and 2005 amendments). Retrieved from http://www.pre.ethics.gc.ca/archives/tcps-eptc/docs/TCPS%20October%202005_E.pdf Chouinard, J, & Cousins, J. B. (2007). Culturally competent evaluation in First Nations and Inuit communities: A review of literature. Journal of Multidisciplinary Evaluation, 4(8), 40 - 57. Cousins, J. B., & Earl, L. M. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14, 397 - 418. Cousins, J. B., & Earl, L. M. (1995). Participatory evaluation in education: Studies in evaluation use and organizational learning. Washington, DC: Falmer. Cousins, J. B., & Shulha, L. M. (2006). A comparative analysis of evaluation utilization and its cognate fields of inquiry: Current issues and trends. In I. Shaw, M. M. Mark, & J. Greene (Eds.), Handbook of evaluation: Program, policy and practices. London: Sage. 26 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New directions for evaluation, 1998, 5 - 23. Daigneault, P.-M. (2010). L’examen de qualité des évaluations fédérales: une méta-évaluation réussie? Revue canadienne d’évaluation de programme, 23(2), 191 - 224. Daigneault, P.-M., & Jacob, S. (2009). Toward accurate measurement of participation: Rethinking the conceptualization and operationalization of participatory evaluation. American Journal of Evaluation, 30(3), 330 - 348. Daigneault, P.-M., & Jacob, S. (forthcoming). Unexpected but most welcome: Mixed methods for the validation and revision of the participatory evaluation measurement instrument. Journal of Mixed Methods Research. Daigneault, P.-M., Jacob, S., & Tremblay, J. (2012). Measuring stakeholder participation in evaluation: An empirical validation of the participatory evaluation measurement instrument (PEMI). Evaluation Review, 36(4), 243 - 270. Ebbutt, D. (1998). Evaluation of projects in the developing world: Some cultural and methodological issues. International Journal of Educational Development, 18, 415 - 442. Eliadis, P., Furubo, J.-E., & Jacob, S. (Eds.). (2011). Evaluation: Seeking truth or power? New Brunswick, NJ: Transaction Publishers. Fetterman, D. M. (2000). Foundations of empowerment evaluation. Stanford University: Sage Publications. Frierson, H. T., Hood, S., & Hughes, G. B. (2002). Strategies that address culturally-responsive evaluation. In J.F. Westat (Ed.), The 2002 user friendly handbook for project evaluation (pp. 63 - 73). Arlington, VA: National Science Foundation. Retrieved from http://www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf Frierson, H. T., Hood, S., & Hughes, G. B. (2005). Strategies that address culturally responsive evaluation: A guide to conducting culturally responsive evaluations. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 63 - 73). Thousand Oaks, CA: Sage. Guba, E. G., & Lincoln, Y. S. (2005). Paradigmatic controversies, contradictions and emerging confluences. In N. Denzin & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research (Vol. 8, pp. 191 - 215). Thousand Oaks: Sage Publications. Health Canada. (1998). Taking action on population health. A position paper for Health Promotion and Programs Branch Staff. Ottawa: Publications gouvernementales du Canada. Indian and Northern Affairs Canada. (2011). Aboriginal peoples and communities. Retrieved from http://www.aadnc-aandc.gc.ca/eng/1100100013785/1304467449155 27 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 Indian Affairs and Northern Development (IAND). (2000). Gathering strength: Canada’s Aboriginal Action Plan. Ottawa, ON: Author. Institute on Governance. (2011). Gouvernance autochtone, Institut sur la gouvernance. Retrieved from http://iog.ca/fr/domaines-de-comp%C3%A9tence/gouvernance-autochtone Jacob, S., Bélanger, J.-F., & Ouvrard, L. (2011). Participatory evaluation and process use within a social aid organization for at-risk families and youth. Evaluation and Program Planning, 34(2), 113 - 123. Jacob, S., & Daigneault, P-M. (2011). La gouvernance et l’implication des parties prenantes dans l’évaluation des politiques: panacée ou boîte de pandore? In C. Rouillard & N. Burlone (Eds.), L'État et la société civile sous le joug de la gouvernance (pp. 217 - 242). Québec, QC: Presses de l'Université Laval. Kirkhart, K. E. (1995). Seeking multicultural validity: A postcard from the road. Evaluation Practice, (1), 1-12. Macaulay, A. C., Paradis, G., Potvin, L., Cross, E. J., Saad-Haddad, C., McComber, A., Desrosiers, S., Kirby, R., Montour, L. T., Lamping, D. L., Leduc, N., & Rivard, M. (1997). The Kahnawake Schools Diabetes Prevention Project: Intervention, evaluation, and baseline results of a diabetes primary prevention program with a Native community in Canada. Preventive Medicine, 26, 779 - 790. McKenzie, B. (1997). Developing First Nations child welfare standards: Using evaluation research within a participatory framework. The Canadian Journal of Program Evaluation, 12(1), 133 - 148. Mertens, D. M. (2008). Stakeholder representation in culturally complex communities. In N. L. Smith & P. R. Brandon (Eds.), Fundamental issues in evaluation (pp. 41 - 60). New York, NY: Guilford Press. Nelson-Barber, S., LaFrance, J., Trumbull, E., & Aburto, S. (2005). Promoting culturally reliable and valid evaluation practice (Chapter 5). In S. Hood, R. Hopson, & H. Frierson (Eds.), The role of culture and cultural context: A mandate for inclusion, the discovery of truth, and understanding in evaluative theory and practice (pp. 61 - 85.). Greenwich, CT: Information Age Publishing. Northwest Territories Government. (2004). GNWT Strategic Plan: Self-reliant people, communities and Northwest Territories - A shared responsibility. Yellowknife: Author. Patton, M. Q. (2008). Utilization focused evaluation (4th ed.). California, CA: Sage. Potvin, L., Cargo, M., McComber, A. M., Delormier, T., & Macaulay, A.C. (2003). Implementing participatory intervention and research in communities: Lessons from the Kahnawake schools diabetes prevention project in Canada, Social Science & Medicine, 56, 1295 - 1305. 28 The International Indigenous Policy Journal, Vol. 4, Iss. 2 [2013], Art. 1 http://ir.lib.uwo.ca/iipj/vol4/iss2/1 DOI: 10.18584/iipj.2013.4.2.1 Rossman, G. (2000). Participatory monitoring and evaluation. The Center for International Education. Retrieved from http://www.umass.edu/cie/Themes/participatory_evaluation.htm Royal Commission on Aboriginal Peoples. (1996). Report of the Royal Commission on Aboriginal Peoples. Retrieved from http://www.collectionscanada.gc.ca/webarchives/20071115053257/http://www.ainc- inac.gc.ca/ch/rcap/sg/sgmm_e.html SenGupta, S., Hopson, R., & Thompson-Robinson, M. (2004). Cultural competence in evaluation: An overview. New Directions for Evaluation, 102, 5 - 19. Smith, T. (1990). Policy evaluation in third world countries: Some issues and problems. The Asian Journal of Public Administration, 12, 55 - 68. Thomas, W., & Bellefeuille, G. (2006). An evidence-based formative evaluation of a cross-cultural Aboriginal mental health program in Canada. Australian e-Journal for the Advancement of Mental Health, 5(3), 1 - 14. Wholey, J. S. (1994). Assessing the feasibility and likely usefulness of evaluation. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 15 - 39). San Francisco: Jossey-Bass. Zukoski, A., & Luluquisen, M. (2002). Participatory evaluation. What is it? Why do it? What are the challenges? Community-Based Public Health Policy & Practice Partnership for the Public’s Health, 5. Retrieved from http://depts.washington.edu/ccph/pdf_files/Evaluation.pdf 29 Jacob and Desautels: Evaluation of Aboriginal Programs Published by Scholarship@Western, 2013 The International Indigenous Policy Journal April 2013 Evaluation of Aboriginal Programs: What Place is Given to Participation and Cultural Sensitivity? Steve Jacob Geoffroy Desautels Recommended Citation Evaluation of Aboriginal Programs: What Place is Given to Participation and Cultural Sensitivity? Abstract Keywords Acknowledgments Creative Commons License Evaluation of Aboriginal Programs: What Place is Given to Participation and Cultural Sensitivity?