Program Evaluation as Community-Engaged Research Challenges and solutions Much of the thinking that has shaped our understanding of community-engaged research has its roots in the intellectual résistance of the early 20th century to the domination and permeation of the positivist imagination into our ways of knowing, talking about and interacting with the world. Since then, positivism’s vision of a singular source and mode of knowledge, accessed, secured and enriched only through carefully guarded instruments and methodologies, has been systematically assaulted by social theorists at two major points. At the first of these, adherents of the critical theory tradition have attacked the tendency of positivist models to reduce diversity and complexity in the social world to rationalised taxonomies and empirically observed social ‘facts’, querying the possibility of their ‘objective’ observation and even disputing their very existence. Elsewhere, cultural and political theorists have exposed the myriad ways in which positivist epistemologies preserve and rely upon systems of bureaucratic control, sustaining the privileges of the elite and entrenching social stratification and inequality. These critiques, intellectual progenitors of post-modernist epistemologies, instead seek to promote plural and local forms of knowledge and envisage sociological research as a function of emancipation. Their authors, from Michel Foucault to Edward Said, have inspired a re-imagination of research as a collaborative, inclusive and impact-driven process that acknowledges the role of knowledge creation in the establishment, preservation or disestablishment of the latent power relationships that sustain disadvantage and social division. In the hands of critical pedagogists such as Paulo Freire and Henry Giroux, this epistemological tradition became a tool for those committed to the empowerment of the disempowered, the granting of voice to the unheard, and the inclusion of the excluded. Theories of community-engaged research emerged from within this tradition as an approach to research conducted in community contexts, and encouraged the development of collaborative strategies for advancing community wellbeing, in so doing seeking to foster Gateways: International Journal of Community Research and Engagement Vol 8/No 1 (2015): 118–38 © UTSePress and the author ISSN 1836-3393 Richard Reed Macquarie University 119 | Gateways | Reed and support partnerships between ‘researchers’ and ‘researched’ characterised by two-way learning built on a commitment to knowledge exchange and mutual respect and recognition. Despite being a tool for understanding impact and effectiveness, evaluative research into community-based programs has not always taken this approach. Indeed, for those whose programs or activities are ‘under evaluation’ it might give the impression of reinforcing control rather than advancing collaboration, equity, mutual learning or any other emancipatory outcome. In part, this is an inevitable consequence of differences and disagreements vis-a-vis evaluation terminologies, methodologies and strategies. On the face of it, evaluation has a number of broad goals, serving principally to guide program development, support institutional planning and enhance accountability. In practice, the weighting or priority these goals receive depends hugely on context. In considering the relationship between evaluation and the principles of community-engaged research, we also need to think briefly about an important semantic question. The extant body of literature on evaluation practice implies some degree of consensus that evaluation, in non-academic contexts, employs unique techniques that set it apart from other forms of social research – and thus that there may be a clear difference between evaluation and evaluative research. To an extent this is undeniably true – the methods evaluation deploys are often focused first and foremost on delivering findings that are useable and have practical applications; this often requires ‘compromising’ on traditional concerns for research quality vis-a-vis data validity in order to address certain professional and practical expediencies. Notwithstanding the emergence of quite different languages and approaches, however, to some degree this is also something of a false dichotomy, one that has emerged from and speaks to differences in professional positioning and structures rather than fundamentally different methods, values or goals. While recognising this is still a live debate, for the purposes of this present article I assume a degree of interchangeability between the terms evaluation and evaluative research. I therefore urge the reader to see beyond any language that suggests an alignment with one mode over the other to the conceptual and practical issues that are important considerations for all modes and forms of evaluation. From this premise I advance the contention that evaluative research into social interventions can – and indeed should – be both conceptualised and operationalised as engaged research committed to effecting positive social change. The article frames its reflections in the context of evaluative research into a number of outreach programs at Macquarie University, Sydney, targeted at school students and community members from backgrounds that are underrepresented within Australia’s higher education student population. The article illustrates ways in which evaluative research can be conceived of in terms of community 120 | Gateways | Reed engagement by mapping research strategies against two influential models of community engagement. It then reflects on some of the challenges in implementing these ideal practices within the context of evaluative research, recognising that the challenges and opportunities that have arisen during this evaluative research reinforce the conclusion that such models can only provide aspirational targets. In this sense, the appropriate moral and professional framework for such research is one that combines a commitment to engagement with a reflexive, adaptable, pragmatic and above all iterative approach to methodology and stakeholder relations. BACKGROUND: EVALUATING OUTREACH ACTIVITIES The evaluative research upon which the reflections in this article are based focused on seven educational interventions led by Macquarie University professional and academic staff. These programs are part of a broad portfolio of activities at Macquarie funded by the Federal Government’s Higher Education Participation and Partnerships Program, a tranche of bespoke funding initially conceived as a response to a review of Australia’s higher education system in 2008 that identified the disproportionately low participation of a number of societal groups in higher education, including individuals from low socioeconomic, Aboriginal, and culturally and linguistically diverse backgrounds. Collectively, the Macquarie programs have identified a number of critical objectives to assist the goal of ensuring that the representation of these groups in higher education corresponds to their demographic representation within the broader population, including raising ambitions and aspirations, increasing capacity and skills, and tackling structural disadvantage. In order to assess progress against these aims, in 2012 Macquarie commissioned evaluative research to complement existing reporting and monitoring exercises and develop strategic understanding of the programs’ impacts. This evaluation consisted of both formative and summative components and produced five principal outcomes: 1 In its formative aspect, the evaluation provided a ‘reflective space’ in which evaluators were able to work with program facilitators to help identify appropriate, evidence-based improvements. The evaluation sought to explain not just what the impact had been, but also how that impact was achieved, allowing program facilitators to build on the successes and strengths of individual programs and to address critical weaknesses. 2 The research findings supported strategic decision-making around the University’s social inclusion and widening participation strategies. 3 By contributing to various reporting processes, the evaluation supported efforts to ensure the university meets its legal accountability and transparency requirements. 121 | Gateways | Reed 4 The evaluation helped to construct an evidence base on the impact of social inclusion activities, thus building a case for the continuation of funding and public support and interest. 5 The evaluation contributed to general knowledge concerning disadvantaged students from a range of backgrounds and the barriers they face in accessing higher education. The collection and analysis of data was guided by a theoretical and methodological framework that fused current best practice in evaluative research with the specific aims and objectives of the program. The evaluation was holistic in scope, intended to gather triangulated data but also to uncover evidence of how impact is often mediated by the relationship between and contributions of the various stakeholders in the program. EVALUATION AS COMMUNITY-ENGAGED RESEARCH Although the increasing demand for accountability and transparency of publicly funded social interventions has inspired a focus on summative evaluation (Shah, Nair & Wilson 2011), the relatively developmental character of many of the programs under assessment at Macquarie (as well as the portfolio to which they belonged) meant it was important to give equal attention to formative aspects of the process. Formative evaluation accrues a number of benefits above and beyond those of summative evaluation; in the present case it was anticipated that it would facilitate learning and program development (Harris, Jones & Coutts 2010; Nesman, Batsche & Hernandez 2007), as well as build evaluative and reflexive capacity – critical given the limited duration of the evaluative research (Preskill & Boyle 2008; Smeal, Southwell & Locke 2011). Formative evaluation, of course, is also particularly conducive to participatory methods and models of engaged research (Harris, Jones & Coutts 2010; Hashimoto, Pillay & Hudson 2010; Stoner et al. 2012). However, the research also advanced from a recognition of the potential for top–down disengaged research to extend and deepen entrenched disadvantage, and thus a failure to engage would risk working against the goals of the programs it was seeking to assess. There were a number of additional reasons why the principles of engaged research were seen as crucial to this evaluation project. The exchange of knowledge facilitated by collaborative partnerships is critical to ensuring that evaluative practice is directly informed by the reality of conditions on the ground. Procuring the wisdom of multiple perspectives and the intimate knowledge of program development is key to the development of effective, efficient, ethically sound and minimally disruptive evaluation instruments. It should also not be overlooked that the trust engendered by the good relationships created as part of engaged research practice ultimately enhances the possibility of more intimate and accurate testimony from research participants who understand and buy into the function and purposes of the research. However, we would do well to recognise that, while 122 | Gateways | Reed partnerships characterised by high levels of trust are likely to provide more intimate testimony, better understanding of the rationale and direction of the research project may also lead to a bias in results as research participants provide answers in interviews and focus groups they feel the researcher wants to hear. In creating the conditions and building the relationships that enable and sustain knowledge exchange, engaged research also facilitates the building of capacity (evaluative capacity, in this case). External evaluation is rarely cheap, even where it is readily available. Moreover, whatever expertise or putative objectivity external evaluation brings to the table, it takes time for the evaluators to familiarise themselves with evaluated projects, and requires effective communication on an ongoing basis to ensure researchers are aware of developments in the program under evaluation. The inevitable distance of the external researcher from the program also means that there is a constant risk that evaluation findings are misaligned with program objectives or otherwise misrepresent the work under evaluation. Internal evaluation, by contrast, drastically reduces the risk of this disconnect because it is informed by professional experience in the program. For these reasons a number of authors have argued that best evaluative practice by necessity should include a capacity-building component. Again, engaged research with a focus on the building of collaborative relationships characterised by trust, mutual learning and knowledge exchange are plainly the best vehicles for this process (Nesman, Batsche & Hernandez 2007; Oliver et al. 2002; Preskil & Boyle 2008; Ryan, Chandler & Samuels 2007; Smeal, Southwell & Locke 2011). From the outset, therefore, the evaluative research at Macquarie envisaged engaged methodologies as critical to its success, thus embedding a commitment to collaboration in the key research stages of design, implementation, iteration, analysis and dissemination. The research identified individual program staff, principals and senior teaching and support staff in schools, and parents and community representatives as key stakeholders and, as far as possible, encouraged and presented opportunities for these stakeholders to contribute to research processes. The inclusion of stakeholder voices at every stage of the research was also intended to ensure that the findings of the evaluation were a product of the authentic integration of multiple areas of expertise and local knowledge, brought together within and through the research process. This input would help to maximise the research’s potential to empower its participants to make informed decisions about program improvements and professional or pedagogic practices, and to equip its participants with a collaboratively constructed evidence base to support program sustainability and advocacy efforts. With these considerations in mind, due thought was given from the outset to how theories of community engagement might support the development of engaged and inclusive research strategies and methodologies for the Macquarie evaluation. 123 | Gateways | Reed This was not a simple proposition: the current literature on community engagement is emergent rather than established, and the frameworks that do exist are varied in quality, detail, scope and applicability. Nevertheless, it is possible to discern some points of consensus within the extant body of scholarship, and evaluative research has integrated three core objectives within its various stages sourced from this literature: (1) empowerment of program stakeholders; (2) democratisation of knowledge; and (3) effecting social change. Additionally, two of the better known community- engaged research models have been useful for conceptualising and operationalising the engaged intent at the heart of evaluative research: Arnstein’s Ladder of Citizen Participation (Arnstein 1969; see Figure 1), and the more recent five-point framework developed by the International Association for Public Participation (IAPP) 2014; compare also Sarrami Foroushani et al. 2012; see Figure 2). Both models share a common vision of community-engaged research falling somewhere along a continuum – from top–down, autocratic and non-participatory approaches at one end to fully democratic, inclusive and synthetical modes of research at the other (Bowen, Newenham-Kahindi & Herremans 2010; Russell et al. 2008). Citizen power Citizen control —Enabling informed decision- making to enhance future practice —Facilitating advocacy —Evaluative capacity Delegated power —Joint decision-making —Inclusion in methodology —Responsibility for some dissemination Partnership —Collaborative design —Presentation of findings —Joint responsibility —Shared credit Tokenism Placation —Agreement on methodology (e.g. timing, location, extent) Consultation —Ascertain boundaries, existing practices, potential challenges —Agree consent Informing —Nature of research project, aims, objectives, scope, anticipated roles Non- participation Therapy Manipulation Figure 1: National Indigenous Science Education Program evaluation mapped against Arnstein’s Ladder (1969) 124 | Gateways | Reed In combination, these models were used in the Macquarie evaluation as frameworks to guide engagement and inform matters of timing, scope, audiences and methods for engagement. Collaboration was key throughout: during the design phase, for instance, stakeholders were initially identified and consulted to ensure the process of framing the project was informed by a range of stakeholder views (Harris, Jones & Coutts 2010) and the particular requirements and reality of the program (Lawrenz & Huffman 2003). This was followed by an initial design phase that involved further consultations around matters of methodology and anticipated implementation; during this stage, and subsequent iterative design stages, stakeholders were asked to comment on proposed interview schedules and to contribute questions of their own that would provide useful information in their professional development. The stakeholders were also critical in the implementation stage. Not only did they broker contacts between the researchers and the program participants, but in line with the ambition to raise evaluative capacity they were also directly involved in the data collection (though decisions in this regard also had to be balanced with matters of research ethics and data integrity). Finally, stakeholders were involved on an ongoing basis in the dissemination of the research findings. In one case, for example, the researchers worked closely with both the Macquarie-based program team and principals and head science teachers in a number of schools in Western Sydney involved in a science outreach program to devise strategies for disseminating the research findings as professional development for teaching staff. Elsewhere, during evaluations of the University’s scholarship schemes and a mentorship program in partnership with a national Figure 2: Evaluative research against the five ‘goals’ of community engagement 125 | Gateways | Reed media organisation, program staff were able to identify critical audiences for dissemination, in addition to leading or advising dissemination practices as appropriate. MITIGATING IMPACT Notwithstanding these aspirations, there were a number of challenges that inhibited the evaluation’s success in realising this idealised framework on the ground. Figure 3 provides a visual representation of some of the key issues: it imagines a series of challenges and opportunities that collectively constitute ‘intervening factors’ that mediated the impact of the evaluation as an engaged project. The challenges the evaluative research faced varied tremendously, from those that were relatively easily overcome to those that required significant compromises. Many of these problems were anticipated as part of an initial risk identification and management process, but this itself became an evolving strategy as the initial approaches adopted to minimise these risks proved unsuccessful or insufficient. These challenges may be usefully considered within the framework of five broad categories: definition, paradigm, participation, resources and governance. Challenges of definition concerned issues of boundaries and inclusion. One of the core challenges of engaged research into the sorts of large multi-agency projects included in this evaluation was the difficulty in identifying the full range of stakeholders (that is, the targets for engagement). The programs under evaluation all involved a number of groups and individuals who could be legitimately considered stakeholders based on their capacity to affect or be affected by the programs (e.g. see Bryson 2004, p. 22), foremost among whom were program funders and coordinators, in addition to a range of teachers, school administrative and support staff, community representatives, parents, school and university student participants, and volunteers. In some cases, however, identifying the stakeholders was less straightforward. The National Indigenous Science Education Program (NISEP), for instance, involves training school students to present scientific Figure 3: The mitigation of impact: challenges and opportunities 126 | Gateways | Reed experiments to their peers, which raised the questions: How should we be thinking about the stake in the program of those observing the experiments? What about the teachers responsible for these students? More problematically, as the programs under evaluation evolved and developed during the course of the evaluation, so too did the size and nature of individuals’ stakes in the programs, resulting in arguments for the inclusion of others in this circle who were not part of the initial engagements. This prompted the broader question: how do we respond when the size of the stake changes as individuals or groups join or leave programs, or as the program develops? Paradigm challenges are those that arise as an inevitable result of the need to synthesise divergent forms of knowledge and professional practice. They are perhaps the most complex and intractable issues to negotiate, as they can involve issues of longstanding unequal power relationships between alternative sets of beliefs and professional or community practices. Many of these concerned bringing together the standards and expectations governing data validity, research practice or ethics protocols as understood in the academy with the alternative models of professional practice or ‘valid’ knowledge that are often prevalent in the schools and community contexts within which programs operate. The complexity of this synthesis was particularly evident for programs delivered in Indigenous communities alongside Indigenous partners, where questions of diverse paradigms also touched on far broader clashes between the Western epistemologies that frame the academy’s ethical oversight and research processes and the local epistemologies of Australian Aboriginal groups. More than a mere intellectual dilemma, this had a genuine consequence. Concerns over conflicts of interest, for example, and the prevalent view in the academic research paradigm that distance and value neutrality are central to the production of valid research findings often precluded – or certainly complicated – the complete inclusion in the research of stakeholders who had either little understanding of, or actively rejected, the dominant ways of thinking about knowledge creation within the academy. Just as difficult to overcome were problems of participation. As Hashagan (2002) recognised, the reality of stakeholders’ interest and willingness to engage in community-based research rarely matches the idealised plan. The differential in stakeholder predisposition towards engagement means that it can sometimes be very difficult – if not impossible – to ensure that all stakeholders are equally included and involved. This throws up numerous professional and ethical dilemmas. Should the researcher focus his/her efforts on those with whom effective connections have been established, or on those with whom the engagement is less solid? How does the researcher contend with different ways of understanding or ‘doing’ collaboration or engagement (the latter was a frequent challenge during the Macquarie evaluation)? 127 | Gateways | Reed The evaluative research at Macquarie evidenced a number of reasons why stakeholder engagement might be limited. For instance, through the course of the evaluation, evidence emerged of stakeholders having previously had ‘bad’ or difficult experiences with research or researchers. Again, there is a particular weight to this ‘burden of distrust’ (Brenner & Manice 2011) for some Indigenous stakeholders, who are understandably suspicious of the intent of external researchers. For some older participants, the bureaucratic cloak of institutional research recalled memories of the welfare men in white coats who, in the name of protection, provided the academic rationale for shocking acts of state- sponsored racism that led to the forced separation of Aboriginal children from their families and country. This legacy haunts even the most democratically intended research project, though the impact of that distrust, of course, varies from person to person, from community to community, and is intricately tied up with local collective memories. Even those stakeholders not scarred in the same way can be inclined to a certain passivity and disinterest towards university- sponsored research that can significantly hinder attempts to involve and include. Sometimes the research is simply not seen as important or capable of addressing key needs within the community – a reflection both of the diverse needs and pressures within community settings and of a perception (fair or unfair) that research tends to be low impact, disengaged, or even parasitic. But this can also manifest as a perception that the researcher is an expert, who not only knows the best way to conduct such research, but is also ultimately responsible for its success or failure. Comments indicating such were relatively common during the evaluative research. For example, one project coordinator remarked at the onset of the evaluation: ‘so the plan is for you to come in, take a look at everything we’re doing, and tell us what we need to stop doing and what we need to do better’; a confronting attitude for the researcher determined to avoid delivering verdicts and to instead facilitate processes of reflection. Passivity and apparent hostility towards the research can also be a product of stakeholders’ lack of confidence in their ability to contribute, or of a lack of understanding of research practice or of the ultimate benefits of the research. It might also be occasioned by a perceived connection between evaluative research and oversight processes, where stakeholders are nervous about the implications of research findings on their professional futures. Or it might simply be a product of a lack of time (or a perceived lack of time) and the pressures of competing priorities (Bamberger et al. 2004; Bearman et al. 2008; Harris, Jones & Coutts 2010; Ryan, Chandler & Samuels 2007). The impact of these personal predispositions, of course, becomes greater the more stakeholders are involved; they represent a significant challenge to securing the active participation of stakeholders necessary to maximise the research’s impact and to ensuring that decision-making around the project is genuinely collective. 128 | Gateways | Reed Challenges of governance reflect issues around decision- making, whether related to the strategising, designing or implementation of the research, or to responding to unanticipated events, difficulties or opportunities. On the one hand, these are intimately connected to problems of definition: how do we determine who to include in the governance structures of a community-engaged research project – all stakeholders or only some? If the latter, using what criteria – the size of the stake in the project? How do we respond when the size of that stake changes as the program develops and individual or group numbers decrease or individuals or groups take on more responsibility for the program? Should we include those who are most likely to understand and engage with the research goals, or reach out to those who are not well disposed towards the research (a perhaps more democratic gesture but also one that comes with risks and no little potential for frustration)? And how should staff turnover be handled? The difficulty arises in deciding where and how to balance the trade off between inclusiveness versus the ability to rapidly respond to unforeseen problems. Of course, this challenge, as with those others above, becomes all the greater in the context of limited resources. External evaluations are often limited to short time periods, and because they are not directly linked to core practices, it can also be difficult to make a case for the assignment of significant resources. As became clear during the evaluative research at Macquarie, the limited timeframes and funding of many evaluative research projects do not always work well with the inevitable resource intensity of establishing and maintaining fully consultative and inclusive research practices, particularly where the stakeholders are initially resistant or disinterested. The lack of time or limited opportunities for face-to-face contact can make it particularly difficult to establish trust with stakeholders who are negatively predisposed towards research practice. Staff turnover in stakeholder organisations (an issue exacerbated, in this instance, where programs rely heavily on volunteers) can further tax the resources available to the researcher, particularly if these changes result in the ‘resetting’ of key relationships in the middle of an evaluation. MEETING THE CHALLENGES Notwithstanding the tendency for these challenges to sometimes seem like insurmountable hurdles, the extant community engagement and evaluation literatures offer a number of strategies for overcoming such difficulties. For instance, one tangible response to the challenges of definition is a robust stakeholder analysis process that identifies the interest and power of key stakeholders, their relationships with one another and the program, and their orientation towards the program (e.g. Brugha & Varvasovszky 2000; Gilson et al. 2012). Mapping out these characteristics allows researchers to make ethical but also pragmatic decisions about who the key stakeholders are, and which inclusion/engagement measures to adopt for which stakeholders. 129 | Gateways | Reed Similarly, bridging paradigmatic differences between the research and community worlds, which requires researchers to be ‘scientifically sound in locally appropriate ways’ (CTSAC 2011, p. 124), could involve collaborations with key stakeholders during planning stages to map out and document some of the values and practices that are shared by all the stakeholders, either as an explicit ‘statement of values’ or as part of an Memorandum of Understanding (MOU; e.g. Pasick et al. 2010, p. 16; Ross et al. 2010; SC CTSI 2012, p. 19). Researchers have to be careful to ensure that these values speak to community needs, and thus that the goals and intended impact of the research address these needs. In the case of the Macquarie evaluation, one of the shared values acknowledged by a number of stakeholders was a belief in the power of mutual learning; this could then be put into action by turning research findings into toolkits intended to support professional development activities. An MOU can also help respond to resourcing challenges by ensuring all available resources are identified, mobilised and rationalised as far as possible (Davis et al. 1999; Davis et al. 2003). Inevitably, finding bridges to overcome divides in the way different stakeholders think and engage with the work will go a long way towards addressing problems of participation and engagement. Clear statements of ground rules and expectations, embedded in universally understood principles that reflect local cultures and needs but also address the practical requirements of the research, can also do much to help smooth the integration of new stakeholders as the program and the research develop. Similarly, a good stakeholder analysis will include an assessment of stakeholders’ strategic concerns and interests, and thus provide a strong platform for the development of specific methods for increasing the engagement of sceptical or disinterested stakeholders. The Macquarie evaluation made good use of some of these strategies, and in many cases the experience of the research also reinforced their value. Many of these strategies formed part of the systematic consultations the researchers initiated with the program teams during the research planning phase, guided by the RUFDATA evaluation planning tool developed at the University of Lancaster (Saunders 2000). In accordance with this model, the researchers engaged the program coordinators and key stakeholders in a series of structured conversations designed to establish a number of key baselines for the evaluation: its key objectives (reasons and uses), what activities it will evaluate (foci), the evidence it intends to collect (data), key stakeholders and dissemination plans (audience), appropriate timescales (timing) and who is responsible for which aspects (agency). The conversations were drafted into a series of collectively owned written agreements. This process provided information that allowed the researchers to make informed decisions about inclusion strategies, as well as how to overcome the problem of high-influence stakeholders with more complex orientations towards the programs and the evaluations. It also constituted a 130 | Gateways | Reed form of stakeholder analysis that provided the rationale by which engagement with stakeholders who were most directly impacted by, or had the greater influence over, the program could be prioritised over those with less influence or interest. There are additional strategies researchers might use to boost stakeholder engagement. For instance, researchers might fuse stakeholder analysis with a strengths-based framework to counter the tendency for stakeholders to feel unqualified, thus identifying approaches to the problem of researcher passivity (Harvey 2014). Establishing channels for ongoing consultation throughout the course of the research – not just in the initial planning phases – is also critical. The establishment of a community advisory group with strong, inclusive leadership based on a solid understanding of power dynamics, for instance, can create a space in which stakeholders can air concerns and respond to emergent challenges; and it can also boost the stakeholders’ sense of agency and ownership of the research project. The establishment of subcommittees and inclusive approval or feasibility processes may also help to increase levels of active engagement by increasing the individuals’ sense of agency in the process. Where issues of resources make such structures difficult to construct and maintain, the researchers might instead commit to planned iteration phases that prompt renewed consultations and reflections with key stakeholders. This was the preferred option in the case of the Macquarie research, where the rather complex nature of the research, its various programs, and the wide range and diversity of stakeholders made it extremely difficult and time consuming to establish formal advisory structures. Instead, the formalised periods of ‘iterative program design’ provided prompts and opportunities for re-engagement and also presented critical opportunities for realigning the research where changes in program delivery or personnel over time had left a distance between the program and its evaluation. The scheduled consultations that were part of this iterative process helped to ensure the research stayed relevant to its original goals and abreast of the influence of these changes. Reflection on the evaluative research at Macquarie also exposed additional complexities that suggest we should implement such processes with a degree of care and sophistication. Sometimes, for instance, there remain nuances such processes (or the literature they are based upon) do not always address. In conducting stakeholder analysis, for example, researchers need to find ways to distinguish between the identities and characteristics of stakeholders in the program and stakeholders in the research. Ultimately, ensuring research methods are implementable and conducive to supporting community-engaged research principles (such as capacity strengthening and knowledge exchange) throughout the lifetime of the research means understanding the nuances of the stakeholders’ orientations towards the research 131 | Gateways | Reed itself, not just the program. There is often a good degree of overlap between the two, but at other times there are some important differences that any stakeholder analysis should address. The Macquarie evaluation also proved that the task of conducting a stakeholder analysis can be complicated by a whole series of specific contextual factors. For instance, it became apparent that high levels of support among practitioners for the programs under assessment could translate into a more opposed orientation towards evaluative research – or indeed any research perceived as intrusive or threatening. Most of the staff involved in the research programs at Macquarie were employed for fixed terms, with no guarantee of employment beyond the end of their term, a reality that is true of many social interventions and makes for a particularly fraught and complex context in which to conduct evaluative research. As a consequence, it was not always straightforward to establish stakeholders’ level of interest and influence in the research, or their true orientation, particularly where there was thought to be a connection between being seen to offer outward support for the research and ongoing employment or other professional opportunities. The author certainly experienced occasions during the evaluations where key stakeholders expressed strong outward support for the evaluative process, but failed to follow through on promises of support – or even worked against the research to a degree behind the scenes. Confronting as they may be, such challenges are an inevitable aspect of social and professional environments characterised by intersecting and competing interests, hopes and fears of different stakeholders, requiring some triangulating of sources and a degree of resilience and creativity on the part of the researcher. These added complexities remind us that, while thorough planning and inclusive structures and processes are important tools for enabling stakeholder engagement, there is no solely procedural panacea to the typical challenges of community- engaged research. Reflecting on the limitations with some of these processes as they manifested through this evaluative research helped to reaffirm the critical foundation provided by effective relationship building, enabled through strong communication and interpersonal skills. Solid professional relationships built on mutual trust, respect and recognition, and characterised by transparency and authenticity, can go a long way to overcoming key participation and paradigmatic differences. They can also open up avenues for the building of evaluative capacity, empowering stakeholders to deliver positive impact themselves on later occasions that might not have been possible without the research project. And, by facilitating mutual learning and knowledge exchange, effective professional relationships can do much to help to mobilise additional resources and smooth the processes of decision-making associated with effective research governance. In many cases, the researcher has a key role to play here: overcoming paradigmatic differences, for instance, requires that 132 | Gateways | Reed researchers ‘demonstrate inclusion and respect to the fullest extent possible’ (Gittelsohn et al. 2003) by respecting community customs and practices. Even where the research has clearly constrained goals and methods, ensuring the community are included may well require a genuine ethnographic engagement, where researchers attend events and functions, for example, not directly related to the research but that nevertheless serve an important role in building trust and respect. There are also various practical solutions the researcher might implement to help minimise potential conflicts and misunderstandings where stakeholders are situated in different professional paradigms. Face-to-face meetings, for instance, are inevitably better than conference calls or emails in this regard, opening up a broader range of communicative cues and extending the possibility of finding shared ground. Misunderstandings about research practice might be overcome via a commitment to knowledge exchange realised through training or capacity-strengthening activities on the one hand and the researcher’s participation in community events on the other. Ultimately, however, effective relationship building requires reciprocity; no engaged project can succeed without all sides respecting and recognising the experience and expertise each side brings to the table. Overcoming paradigmatic differences requires that all parties maintain an awareness of the potential for miscommunication and be scrupulous about the assumptions they make. Often the literature puts much of the onus on the researcher to take responsibility for crafting such relationships and to make the critical compromises necessary for ensuring these relationships develop. In community contexts, this is often based on the perception that the researcher occupies a privileged power position. In reality, however, even when engaged with significantly disadvantaged communities, the power relationships are complex – community gateholders, for instance, can wield significant influence over the outcome of the research. Rather than taking full responsibility for the success or failure of these relationships, the researcher might be better served to think of their role as creating the conditions for such relationships to emerge (such as open communication channels). A researcher who makes too many compromises may find him or herself becoming less rigorous in observing their inclusive ideals as frustration mounts; in striving to manage the responses, fears or interests of others, the researcher should not forget to attend to his or her own. In practice, it may be more effective to craft a middle ground where certain ground rules are respected. Often this can be as simple as observing meeting etiquette, a small but deceptively important aspect of conveying mutual respect and recognition. Creating professional structures and interactions characterised by mutual respect and recognition of each stakeholder’s experience and expertise is a critical foundation, and a commitment to regular communication will help build awareness of the limitations and pressures on both sides (Horn et al. 2008). 133 | Gateways | Reed Ultimately, the most effective relationships will always emerge where both sides are committed to this process and have a shared appreciation for the value such relationships add. The effectiveness of this process is inevitably increased where practitioners and managers provide structures and incentives to support it. At the very least, it should be recognised and acknowledged in the planning stages, prior to the research being commissioned, that relationship building will need extra time and, where possible, work collaboratively with the researchers to build support for and understanding of the research process. SIDEWAYS THINKING There are occasions, as the Macquarie research evidenced, when even these responses are insufficient to overcome the challenges of community-engaged research and deliver both tangible research outcomes and the sort of full and inclusive stakeholder engagement to which this model of research aspires. When solutions have been exhausted, or where relationships just do not function no matter how much energy has been invested from either or both sides, community-engaged research can feel like an uncomfortable, uncertain and unmanageable enterprise. This author certainly encountered such moments, and contended regularly with uncertainty as to how to overcome differences in understanding and attitudes associated with individual personalities or long professional or cultural legacies – factors that proved ultimately beyond the power of the researchers to address satisfactorily within the project’s confines. These occasions remind us of an important reality of community-engaged research. ‘No battle plan’, insisted the Prussian military strategist Helmut von Moltke, ‘survives first contact with the enemy’; so too it is impossible to plan away all the potential complications and challenges that arise when conducting community-engaged research. Planning, strategising and doing all one can to build effective collaborative structures and spaces remains an essential part of conducting engaged research, but often the most appropriate response to these challenges is for the researcher to address his or her own attitudes and expectations (and, where relevant, those of the commissioning organisation). Sometimes this involves adjusting (or perhaps resisting, or at least navigating around) some of the traditional ways in which research quality is recognised within the academy. For instance, where questions of data validity are wedded to positivist conventions, there may well be a corresponding desire to strictly control independent variables. But to embark on a community- engaged research project means to an extent learning to become comfortable with a certain amount of fluidity, uncertainty and compromised objectives. Quite apart from being antithetical to the principles of community-engaged research, strict control is rarely possible in any collaborative context. 134 | Gateways | Reed During the Macquarie research, two particular approaches proved particularly useful in countering the temptation to look for ‘perfect’ solutions and outcomes. The first was to give preference to post-positivist qualitative methods that described alternative modes of knowledge creation. Where quantitative methods sometimes struggled to keep abreast of the changing variables that contributed to the evidence of impact, taking a qualitative, narrative-building approach allowed the evaluative research to meaningfully identify and discuss areas of impact without seeing cause and effect as purely a matter of measuring variables against evidence of change. The second response was to explicitly anchor the research in the values that underwrote both the program and the community-engaged research model, so that the research aimed for a broad vision of impact determined not just by the quality of its data but also by the quality of its relationships and the useability of the findings. This wider approach to the question of research ‘goals’, ‘impact’ or ‘success’ provided a strong foundation that also enabled the researchers greater flexibility in responding to changes in program practices or personnel. This last approach was aided by a little lateral thinking. For instance, notwithstanding the depth and breadth of the challenges mapped above, and their potential to seriously impact upon the successful implementation of engaged evaluative research models, during the Macquarie evaluation a number of unanticipated opportunities for the research to maximise its impact emerged when the challenges were reconsidered from another angle. For instance, where evaluators came across negative or misinformed perceptions of research practice that hindered participation, there was a corresponding opportunity, if that research was conducted in a democratic and inclusive spirit, to go some way towards repairing those perceptions. By combating negative pre-perceptions, engaged research might be seen as a means of facilitating future research activity, even if the immediate results have been limited. Perhaps not what those who commission research are always ready to hear, but certainly a tangible outcome with very worthwhile long-term benefits. One final unanticipated opportunity evidenced by the evaluation of the Macquarie programs has been the potential for evaluative practices to directly contribute to program objectives. As discussed above, one of the key aims of the Macquarie outreach programs was to build capacity and confidence within student participants. There is indicative narrative and anecdotal evidence from the Macquarie research that focus groups and interviews requiring participants to reflect on their participation in the program and comment on possible improvements have both helped to develop critical, reflexive thinking and to make the students feel included and heard. A second key objective was the building of capacity within professional staff (in the schools, in this case); the evidence of the formative evaluation processes has been used for professional development purposes and to guide 135 | Gateways | Reed and inform pedagogic practice. Moreover, as attested to by the Macquarie-based facilitators, much of the programs’ success depends on the strength of the relationships between community and school-based stakeholders and the Macquarie staff. Through consultations with the stakeholders, the evaluation process has thus itself helped to deepen and strengthen those connections upon which the programs depend. CONCLUSION The reflections in this article offer a small contribution to our understanding of the complex relationships and processes that characterise the intersection of academic research, educational interventions, and community and school-based pedagogic practices. In a pragmatic and conceptual sense, these frequently manifest as critical challenges for the researcher committed to synthesising the values that inform these different practices via models of community-engaged research. These challenges keep the engaged researcher honest: like any community-engaged research, managing a community-engaged evaluative project involves striving to secure a balance between maximising opportunities and remaining ever mindful and attentive to the attendant risks. This array of challenges also makes one thing particularly clear: good intentions are not enough. The reflections herein disclose the need for researchers to deploy a range of tools and to take on a number of roles, serving as brokers and mediators and being prepared to spend at least as much time building relationships and exchanging knowledge as on more traditional research activities such as data collection and analysis. Here lies perhaps one of the greatest challenges for engaged evaluative research: it places a huge onus on the researchers, not only in terms of time commitments, but also in terms of the array of skills they must bring to the table. In addition to research-specific skills, the evaluator must have knowledge of evaluation methods and content-specific knowledge, and be able to act as a broker, mediator and educator, adapt his or her language to a diverse range of audiences, and constantly translate, mediate and bridge professional discourses. The requirement for such a broad range of skills is why some writers stress the value of using evaluation teams (Worthern & Sanders 2011). Of course, not every challenge is surmountable. Some may need to be accepted or worked around rather than overcome. Many of the challenges researchers face in implementing engaged evaluative practices relate to the difficulty synthesising divergent forms of knowledge, language, professional practice and agendas. The researcher’s success in uniting these often determines the level of impact evaluative research has, but the lack of clear consensus as to the appropriate response to paradigmatic conflict makes this a tricky and indeterminate business. There is also perhaps more outside the researcher’s control than we might ascertain from consulting best practice literature. The reality is that the quality 136 | Gateways | Reed of engagement in evaluative research, like any form of research, may often benefit from the intervention of third parties, such as the line managers of stakeholders, to help motivate or encourage stakeholder participation. Ultimately, this discussion reinforces the argument that there can be no ‘perfectly’ realised model of community-engaged research. Processes and techniques for overcoming key challenges all tend to manifest a common acceptance of the need to prioritise resources and energies; once we begin to prioritise, of course, we do so on the basis of a tacit assumption of the need to strike a compromise between what is ideal and what is practical. These sorts of reflections remind us again that engaged research should be understood as an approach and a process, with engagement taking place on a continuum, rather than as a description of a concrete series of deliverables and outcomes. Instead of looking for operational responses to all these difficulties, it might also help to address the researcher’s expectations and assumptions, and to broaden our ambitions and sense of what constitutes ‘impactful’ or successful engaged evaluative research. Whatever of the ‘pure’ research outcomes a community-engaged project delivers, a disposition towards engagement, collaboration and mutual respect broadens the mind and guards against complacency. More importantly, however, implicit in its commitment to build confidence, capacity and trust is the potential to open up future, as yet unimagined, possibilities. REFERENCES Arnstein, S 1969, ‘A ladder of citizen participation’, Journal of the American Institute of Planners, vol. 35, no. 4, pp. 216–24. Bamberger, M, Rugh, J, Church, M & Fort, L 2004, ‘Shoestring evaluation: Designing impact evaluations under budget, time and data constraints’, American Journal of Evaluation, vol. 12, no. 1, pp. 5–37. Bearman, M, Lawson, M, Lindley, J, Kiegaldie, D, Jolly, B & Kotsanas, G 2008, ‘Evaluation of an inter-institutional granting scheme for collaboration in educational technologies’, Proceedings of the 31st HERDSA Annual Conference, Rotorua, New Zealand, 1–4 July 2008, Higher Education Research and Development Society of Australasia, Milperra, NSW. Bowen, F, Newenham-Kahindi, A & Herremans, I 2010, ‘When suits meet roots: The antecedents and consequences of community engagement strategy’, Journal of Business Ethics, vol. 95, no. 2, pp. 297–318. Brenner, B & Manice, M 2011, ‘Community engagement in children’s environmental health research’, Mount Sinai Journal of Medicine, vol. 78, no. 1, pp. 85–97. Brugha, R & Varvasovszky, Z 2000, ‘Stakeholder analysis: A review’, Health Policy Plan, vol. 15, no. 3, pp. 239–46. Bryson, J 2004, ‘What to do when stakeholders matter’, Public Management Review, vol. 6, no. 1, pp. 21–23. Clinical and Translational Science Awards Consortium (CTSAC) 2011, Principles of community engagement, 2nd edn, National Institutes of Health, Bethesda, MA. 137 | Gateways | Reed Davis, S, Clay, T, Smyth, M, Gittelsohn, J, Arviso, V & Flint- Wagner, H 2003, ‘Pathways curriculum and family interventions to promote healthful eating and physical activity in American Indian schoolchildren’, Preventive Medicine, vol. 37, no. 6, pp. 24–34. Davis, S, Going, S, Helitzer, D, Teufel, N, Snyder, P & Gittelsohn, J 1999, ‘Pathways: A culturally appropriate obesity-prevention program for American Indian schoolchildren’, American Journal of Clinical Nutrition, vol. 69, no. 4, pp. 796–802. Gilson, L, Erasmus, E, Borghi, J, Macha, J, Kamuzora, P & Mtei, G 2012, ‘Using stakeholder analysis to support moves towards universal coverage: Lessons from the SHIELD project’, Health Policy Plan, vol. 27, no. 1, pp. 64–76. Gittelsohn, J, Davis, S, Steckler, A, Ethelbah, B, Clay, T & Metcalfe, L 2003, ‘Pathways: Lessons learned and future directions for school-based interventions among American Indians’, Preventive Medicine, vol. 37, no. 6, pp. 107–12. Harris, L, Jones, M & Coutts, S 2010, ‘Partnerships and learning communities in work‐integrated learning: Designing a community services student placement program’, Higher Education Research & Development, vol. 29, no. 5, pp. 547–59. Harvey, M 2014, ‘Strengths-based theory and practice’, in D Coghlan & M Brydon-Miller (eds), Encyclopedia of Action Research, Sage, Thousand Oaks, CA. Hashagan, S 2002, Models of community engagement, Scottish Community Development Centre, viewed 18 February 2015, www.lcsansw.org.au/ documents/item/463. Hashimoto, K, Pillay, H & Hudson, P 2010, ‘An evaluation framework for sustaining the impact of educational development’, Studies in Educational Evaluation, vol. 36, no. 3, pp. 101–10. Horn, K, McCracken, L, Dino, G & Brayboy, M 2008, ‘Applying community-based participatory research principles to the development of a smoking-cessation program for American Indian teens: “Telling our story”’, Health Education and Behavior, vol. 35, no. 1, pp. 44–69. International Association of Public Participation (IAPP) 2014, Foundations of public participation, IAP2, Wollongong, NSW. Lawrenz, F & Huffman, D 2003, ‘How can multi-site evaluations be participatory?’ American Journal of Evaluation, vol. 24, no. 4, pp. 471–82. Nesman, T, Batsche, C & Hernandez, M 2007, ‘Theory-based evaluation of a comprehensive Latino education initiative: An interactive evaluation approach’, Evaluation and Program Planning, vol. 30, no. 3, pp. 267–81. Oliver, M, MacBean, J, Conole, G & Harvey, J 2002, ‘Using a toolkit to support the evaluation of learning’, Journal of Computer Assisted Learning, vol. 18, no. 2, pp. 199–208. Pasick, R, Oliva, G, Goldstein, E & Ngyuen, E 2010, Community-engaged research with community-based organizations: A resource manual for UCSF researchers, University of California, San Francisco, CA. Preskill, H & Boyle, S 2008, ‘A multidisciplinary model of evaluation capacity building’, American Journal of Evaluation, vol. 29, no. 4, pp. 443–59. Ross, L, Loup, A, Nelson, R, Botkin, J, Kost, R, Smith Jr, G & Gehlert, S 2010, ‘The challenges of collaborations for academic and community 138 | Gateways | Reed partners in a research partnership: Points to consider’, Journal of Empirical Research on Human Research Ethics, vol. 5, no. 1, pp. 19–31. Russell, N, Igras, S, Johri, N, Kuoh, H, Pavin, M & Wickstrom, J 2008, ‘The Active Community Engagement Continuum’, ACQUIRE Project Working Paper, ACQUIRE/USAID, New York. Ryan, K, Chandler, M & Samuels, M 2007, ‘What should school-based evaluation look like?’ Studies in Educational Evaluation, vol. 33, nos. 3–4, pp. 197–212. Sarrami Foroushani, P, Travaglia, J, Eikli, M & Braithwaite, J 2012, Consumer and community engagement: A review of the literature, Centre for Clinical Governance Research, Australian Institute of Health Innovation, Sydney, NSW. Saunders, M 2000, ‘Beginning an evaluation with RUFDATA: Theorizing a practical approach to evaluation planning’, Evaluation, vol. 6, no. 1, pp. 7–21. Southern California Clinical and Translational Science Institute (SC CTSI) 2012, A quick start guide to conducting community-engaged research, Los Angeles, CA. Shah, M, Nair, S & Wilson, M 2011, ‘Quality assurance in Australian higher education: Historical and future development’, Asia Pacific Education Review, vol. 12, no. 3, pp. 475–83. Smeal, G, Southwell, D & Locke, R 2011, ‘Learning and teaching from the edge to centre stage: Critical factors in embedding sustainable university- wide engagement in external awards and grants funding initiatives’, in K Krause, M Buckridge, C Grimmer & S Purbrick-Illek (eds), Research and development in Higher Education: Reshaping Higher Education, Higher Education Research and Development Society of Australasia, Gold Coast, Queensland, pp. 297–307. Stoner, J, Meaden, H, Angell, M & Daczewitz, M 2012, ‘Evaluation of the parent-implemented communication strategies (PiCS) project using the multi-attribute utility (MAU) approach’, Educational Assessment, Evaluation and Accountability, vol. 24, no. 1, pp. 57–73. Worthern, B & Sanders, J 2011, ‘Content specialization and educational evaluation: A necessary marriage’, Journal of MultiDisciplinary Evaluation, vol. 7, no. 15, pp. 272–95. _GoBack