The Journey and Destination Need to Be Intentional Perceptions of success in community-academic research partnerships Community-academic research partnerships involve communities working in collaboration with universities to conduct research on a broad range of health and social issues (Israel et al. 2013; NIH 2011). Participatory frameworks that engage community members in research are on the rise (Ahmed & Palermo 2010; Israel et al. 2010; NIH 2011). Although there may be differences in what the frameworks are named and how they are implemented (Israel et al. 2013; Wallerstein & Duran 2008) – for example, participatory action research, community-based participatory research, and community-partnered participatory research – they are based on principles of equitable power-sharing and decision-making to improve the health and wellbeing of individuals and communities, particularly those that are marginalised (Ahmed & Palermo 2010; Israel et al. 2013; Wallerstein et al. 2008). In addition to improved outcomes for individuals and communities, these partnerships develop the capacity of community and academic members to conduct research that impacts programs, policies and practices (NIH 2011; Wallerstein et al. 2008). Associating partnered research processes with health, capacity and system outcomes has been a challenge for those looking to advance the science of participatory research (Sandoval et al. 2012; Viswanathan et al. 2004; Wallerstein et al. 2008). It is common for partnership processes and outcomes to interact and merge together, thus the field of participatory research is still trying to establish how to best measure contextual and partnership factors that produce change (Hicks et al. 2012). Research has suggested that members value partnership factors, such as leadership development (Cacari-Stone et al. 2014; Chang et al. 2013; Cheezum et al. 2013; Kegler, Norton & Aronson 2008; Rasmus 2014), the building and maintenance of trust (Jagosh et al. 2015) and the transfer of knowledge between community and academic members as much as health, capacity and system improvements (Hacker et al. 2012; Malone, McGruder, Froelicher & Yerger 2013; Nichols, Anucha, Houwer & Wood 2013; Rasmus 2014). Furthermore, tensions may arise when members of Robin Lindquist- Grantz University of Cincinnati Lisa M Vaughn Cincinnati Children’s Hospital Medical Center University of Cincinnati © 2016 by R Lindquist-Grantz & L Vaughn. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 Unported (CC BY 4.0) License (https:// creativecommons.org/licenses/ by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercial, provided the original work is properly cited and states its license. Citation: Lindquist-Grantz, R & Vaughn, L 2016, ‘The journey and destination need to be intentional: Perceptions of success in community-academic research partnerships’, Gateways: International Journal of Community Research and Engagement, vol. 9, no. 1, pp. 1–21. doi: 10.5130/ijcre.v9i1.4872 Corresponding author: Robin Lindquist-Grantz; lindqurn@uc.edu DOI: http://dx.doi.org/10.5130/ ijcre.v9i1.4872 ISSN 1836-3393 Published by UTS ePRESS http://epress.lib.uts.edu.au/ journals/index.php/ijcre/index Gateways: International Journal of Community Research and Engagement Vol 9/No 1 (2016) 2 | Gateways | Lindquist-Grantz & Vaughn community-academic research partnerships are unclear about the project’s intended outcomes and have differing outcome expectations (Nichols et al. 2013). Although partnership processes such as relationship- building, trust, communication, decision-making, capacity- building and knowledge generation are considered key components of participatory research models (Roman Isler & Corbie-Smith 2012; Wallerstein et al. 2008), researchers are still identifying how partnership processes impact long-term population-level outcomes or other unintended outcomes (Jagosh et al. 2015; Lucero et al. 2016; Roman Isler & Corbie-Smith 2012). Some researchers have referred to these processes as ‘intermediate outcomes’ because they are critical influencers of long-term outcomes (Jagosh et al. 2015; Roman Isler & Corbie-Smith 2012; Schulz, Israel & Lantz 2003), whereas others have referred to them as ‘secondary outcomes’ (Malone et al. 2013). Regardless, efforts to continue the advancement of community-academic research approaches necessitate further examination of the interaction between partnership processes and outcomes (Brugge et al. 2010; Hicks et al. 2012; Lucero et al. 2016; Wallerstein et al. 2008). Furthermore, the complexity of outcomes in partnered research warrants the use of qualitative methods in addition to quantitative methods (El Ansari & Weiss 2006; Lucero et al. 2016; Sandoval et al. 2012). The theoretical underpinnings of community-academic research partnerships span a broad continuum from pragmatic problem-solving traditions at one end to critical emancipatory traditions at the other (Wallerstein & Duran 2008). This expansive continuum is often replicated through varying levels of community engagement in community-academic research partnerships, which may affect individual member feelings of empowerment and agency for social change (NIH 2011; International Association for Public Participation as cited in NIH 2011). Moreover, the personal experiences of members can influence how they engage with the partnership, especially in the beginning (Hicks et al. 2012). The purpose of the pilot study discussed in this article was to understand how individuals who live and work within different contexts think about processes within community-academic research partnerships and their relationship to outcomes in order to contribute to the critical examination of these partnerships. We utilised novel participatory methods to understand the relationship between processes and outcomes within community- academic research partnerships to expand on frameworks that promote partnership success. Using concept mapping methodology combined with participant interviews, we explored (a) how members of community-academic research partnerships define success and (b) how these members evaluate the impact of the partnered approach to research. 3 | Gateways | Lindquist-Grantz & Vaughn METHOD Concept mapping methodology and interviews were employed using a mixed methods convergent design, in which quantitative and qualitative data are collected simultaneously, followed by a merging of the results, so that inferences can be drawn (Creswell & Plano Clark 2011). Although there has been increasing support for community-academic research partnerships, traditional methods of research driven by academicians, and widely supported by institutions of higher education and mainstream funding mechanisms, continue to be the norm (Ahmed & Palermo 2010). The innovative convergent design of web-based concept mapping and interviewing in this study offered the opportunity for greater access to and breadth of response from individuals who had been involved with a community-academic research partnership in a variety of settings. It also provided the depth of understanding that can be generated through individual interviews to obtain a more complete picture. Each method further illustrated and elaborated on the results of the other to provide complementarity in this mixed methods study (Greene, Caracelli & Graham 1989). Concept Mapping Concept mapping promotes stakeholder participation in the generation of qualitative data to which multivariate statistical analyses are applied to produce quantitative results that can be represented graphically and analysed by stakeholders (Kane & Trochim 2007; Trochim 1989; Trochim & Linton 1986). Concept mapping’s versatility and ability to generate valid and reliable data that are conceptualised amongst members of a group (Rosas & Kane 2012) offers an alternative to focus group methods and has been considered particularly useful for planning and evaluation (Kane & Trochim 2007). It can even be used as a reliable and valid method for analysing and interpreting open-ended survey data and for informing the development of qualitative interview questions (Jackson & Trochim 2002). The participatory elements of the method have been found useful in studies of community health issues (Burke et al. 2005; Kelly et al. 2007; McFall et al. 2009; Vaughn, Jacquez & McLinden 2013; Vaughn & McLinden 2016) which many community-academic research partnerships address. This may be because concept mapping offers more than a simple method of data collection; it is a powerful visual tool that helps members of the stakeholder group comprehend what they deem to be collectively important so that they can take action. As such, concept mapping aligns well with the pragmatic and problem-solving aspects of participatory research, yet it has rarely been used to study community-academic research partnerships in different contexts. The current study warranted a method that could stand up to the complexity, geographic diversity and contextual specificity of community-academic research partnerships in order to obtain a rich conceptual understanding of how participants think about the definition of partnership success as a means to strengthen participatory research approaches. 4 | Gateways | Lindquist-Grantz & Vaughn To understand perceptions across a variety of partnerships, the concept mapping method was conducted remotely using the web-based survey tool, SurveyMonkey. It included questions about the characteristics of the partnership which the participant was representing and the participant’s role in the partnership. Participants responded to four open-ended prompts aimed at answering the research questions, and were given the opportunity to provide up to five responses for each of the four prompts: 1 In order for a community-academic research partnership to be successful it should achieve goals such as… 2 Based on my experience, unintended but important achievements in community-academic research partnerships are… 3 You know the partnered approach to research is successful when… 4 You know the partnered approach to research is not working when… The online concept mapping was distributed using snowball sampling, starting with researchers within the authors’ academic institution, which is located in a Midwestern city in the United States. This sample included academic researchers who the authors knew had been involved in community-academic research partnerships. Only publicly available email addresses were used in the initial distribution of the web-based concept mapping. At the end of the concept mapping, participants were invited to forward the concept mapping link to other researchers and community members who they knew had also participated in a community- academic research partnership. Responses were recorded using a unique ID assigned to each participant. For the purpose of this study, participants may have been involved in any type of research partnership along the continuum of community engagement, meaning no distinction was made between partnerships where community members were involved in all phases of the research and those where community members were only included in certain phases of the research. There was also no distinction between members who were directly representing residents of a community and those representing community-based organisations. Participants were not given an incentive for completing the open-ended prompts portion of the concept mapping methodology. If a participant had been involved in more than one partnership, they were instructed to respond to demographic questions based on the community-academic research partnership with which they were most recently involved. Of the 27 concept mapping participants, 63 per cent identified as an academic researcher, 33 per cent as a community member and 1 participant did not report an affiliation (4 per cent). Participants reported being involved with partnerships that ranged from 1 to 29 years. Partnership size reported by participants included 1 to 9 members 5 | Gateways | Lindquist-Grantz & Vaughn (n = 8), 10 to 19 members (n = 12) and 20 or more members (n = 7). Nineteen participants (70 per cent) reported working with a partnership that included youth or adult community residents, whereas the rest included community members representing health-care settings, community-based organisations, advocacy groups and the faith-based community. After completing the open-ended prompts and demographic questions in the web-based concept mapping, participants in our study were asked if they would be willing to assist with the response sorting phase. Consistent with concept mapping methodology, the participants sorted the collected qualitative responses by examining patterns of responses and grouping similar ideas into similar categories for each prompt (Kane & Trochim 2007; Rosenburg & Moonja Park 1975; Weller & Romney 1988). Of the six people who volunteered to assist with sorting, three were selected based on (a) their availability within the project time period and (b) diversity amongst their research partnerships, including a community-academic partnership with school personnel; a youth participatory action research collaborative; and a research collaborative comprised of funders, policymakers and community organisations. Participants were offered to conduct the remote sorting electronically using an Excel spreadsheet or paper-based templates created by the authors. All participants chose to use the Excel spreadsheet and were provided with the de-identified responses to the four open-ended prompts via electronic mail. They were also given a detailed instruction sheet that described the steps in the sorting process. All of the sorters were proficient with Excel and reported that the task itself was relatively simple to complete, but that the time for completion was somewhat lengthy. Each sorter completed the task within two weeks of receiving the data. Participants who assisted with sorting were given a gift card incentive. The sorted data were analysed using the SMACOF procedure for multidimensional scaling (MDS) (de Leeuw & Mair 2009) in R statistical software (R Development Core Team 2011). MDS output data were then analysed using hierarchical cluster analysis, which provides a visual map that segregates the data points into clusters of similar ideas (Trochim & Kane 2005). Consistent with concept mapping methodology and cluster analysis, lines were drawn around the data points to display the clusters. The resulting shapes were based on the data point clusters and did not signify anything in and of themselves. However, they did help with analysis of the distance between points within and between clusters. Because the number of clusters could range from one to the total number of items sorted, the researcher chose and labelled the resulting cluster solutions that represented the major concepts and provided sufficient detail without being redundant. The stress index level for each of the four cluster solutions was low (1 = .151; 2 = .223; 3 = .151; 4 = .192). Kane and Trochim (2007) report that most concept mapping projects are between .205 and .365, with lower values indicating a better fit of the data. 6 | Gateways | Lindquist-Grantz & Vaughn Interviews Concept mapping participants were also asked if they would be willing to participate in an individual follow-up interview to provide additional insight into how members of community- academic research partnerships conceptualise success. In-depth qualitative interviewing allows a researcher to explore in detail the thoughts, experiences and opinions of individuals (Rubin & Rubin 2012). Therefore, it was an important supplement to the concept mapping method in further understanding how members of these partnerships distinguish the fine line between partnership processes and outcomes, and methods for determining whether a partnership has been successful. Interview questions were developed when the web-based concept mapping survey launched. This is consistent with a convergent mixed methods design in that the interview questions were not influenced by the concept mapping data, but that integration of the two methods occurred during the analysis and interpretation phases (Creswell & Plano Clark 2011). Interview participants included four academic researchers and one community member. An additional community member scheduled twice for an interview but had to cancel due to schedule conflicts. Similarly to the concept mapping sorting process, these participants were selected for interviews because they were involved in partnerships that spanned the continuum of community engagement in research and had conducted research in various disciplines and fields. This meant only two of the participants who volunteered to do an interview were not selected because they did not meet these criteria. Interview participants were contacted via electronic mail to arrange the interview. One interview was conducted via telephone and the other interviews were conducted in person while the concept mapping data were being sorted. The semi-structured interviews were approximately 30 to 60 minutes in length and were digitally recorded and transcribed. A gift card incentive was provided to participants at the end of the interview. Interviews were analysed using Moustakas’ (1994) methods for phenomenological research to understand the experience of success in a community-academic research partnership. This method includes a series of steps that group expressions around the topic to develop a description of the experience for each individual (Moustakas 1994). Themes are then developed from these individual experiences to provide a group-level description of what it means to be a part of that shared experience (Moustakas 1994; Miles, Huberman & Saldaña 2014). The pilot study was reviewed by the University of Cincinnati Institutional Review Board and determined to be Not Human Subjects Research (#2014-7991). The study provides only an initial exploration of the perceptions of success from members of community-academic research partnerships. Findings are not necessarily generalisable to partnerships or communities. 7 | Gateways | Lindquist-Grantz & Vaughn FINDINGS The following sections synthesise findings from the concept mapping method and individual interviews. Each section is organised by the main study themes that emerged and includes a detailed description of how members of community-academic research partnerships think about outcomes within the context of those partnerships and how they determine whether the partnered approach is successful. Responses to concept mapping prompts and interview questions were similar amongst community and academic participants. Relationships and Partnership Processes are Influential Participants repeatedly identified the role of relationships and partnership processes as critical influencers of success, which supports previous models and studies of the importance of these factors in partnerships. The need for Genuine and Equitable Collaboration was the most cited indicator that a partnership is successful, followed by the Knowledge Generation amongst members of a partnership. These responses support participatory research frameworks which stress that equality among all members is necessary in order to have an impact on the issue being addressed. Other processes such as identifying and adhering to Shared Goals, making sure the partnership is Meeting Deadlines established by members and ensuring that the work of the partnership is Responsive to Community Needs were also considered signs that a community-academic partnership is achieving success. Although members of partnerships expect to achieve measurable Research Objectives/Outcomes on the issue they set out to address, they are more likely to monitor success in the interim based on how the partnership functions throughout the life of a research project and whether there is an equal distribution of power and knowledge amongst all members (Figure 1). These process and relationship factors were also addressed in interviews where many participants noted that the only way to achieve the research outcome goals was to tend to the Figure 1: Goals to be achieved for a community- academic research partnership to be successful 8 | Gateways | Lindquist-Grantz & Vaughn group process. One academic participant connected Research Objectives/Outcomes with process concepts of Genuine and Equitable Collaboration, Shared Goals and Knowledge Generation: I think one of the goals is that you can see what you’re doing … I think a goal I’d like to see, or that I think is important, is having both members in the full range of the processes, the decisions, the use of whatever is generated in the partnership. Conversely, other participants conveyed that the two were separate and important, but they had a harder time determining priority. One academic participant worked through process concepts such as Shared Goals and Genuine and Equitable Collaboration as means to achieve Research Objectives/Outcomes: I don’t know that I can pick process over research outcomes. Because I think it has to be both. I think initially you have to pay more attention to the process, because again trust, relationship-building … I think you’ve got to initially spend more time thinking about the process outcomes. And then I think over time as you work together more you can shift and I think like in the example I gave, we shifted more to focusing on the research outcomes and reaching those goals. Another academic participant also observed the need to focus on Shared Goals in the beginning of the partnership in order to do the work to achieve Research Objectives/Outcomes: Definitely in the beginning there were a lot of differences … not especially with the overall goal … more so with aims for what we can do and where we can focus our efforts on so then we know how to get there. That’s where we differed. And also we had different priorities in terms of our progress. Integration as a Key Partnership Process Tending to the group process was considered crucial in the beginning in order to build trust and rapport, but it was also perceived as a catalyst for developing partnerships where members and research activities could be integrated in ways that transcend typical methods of collaboration. According to participants, this meant exploring innovative methods of communication and participation so the Generation of Knowledge by both community and academic members would result in data that are higher quality and more relevant to the community. Participants noted that this required going beyond structured meeting times and formal communication strategies, such as email, so that academic and community partners could become more embedded in each other’s environments. Participants viewed this expansion of conventional collaboration methods as necessary for partnership sustainability and the production of tangible Research Objectives/ Outcomes. One participant described a long-term partnership that evolved from a community-academic collaboration on a specific project to one where academic researchers were co-located within 9 | Gateways | Lindquist-Grantz & Vaughn the organisation on a part-time basis. This new arrangement facilitated communication and Genuine and Equitable Collaboration between the two parties in order to advance their Shared Goals. ‘I think the longer we work together the more intermingled we’re becoming. The more involved, it’s easier to get more involved. So I think as we’re working toward it, these things naturally happen.’ In another example, a participant described a partnership where Genuine and Equitable Collaboration, Knowledge Generation, Shared Goals and being Responsive to Community Needs were intentional from the start in order to obtain data that was more rigorous and relevant to members of that community. Rather than relying on evolution over time, the academic members initiated the partnership with plans to fully integrate and build the capacity of community members in all phases of the research process. Even with the intentionality behind this partnership, the academic members were surprised by the level of community commitment and engagement, including their ability to drive the research process without years of research training such as that experienced by academics: … we didn’t get into any nuances around sampling … it was in our minds, but we didn’t talk about that, we didn’t train them on any of that … they basically had this discussion of their own accord about sampling and figuring out, without using any of those [academic] words, that ‘oh you know what, we have too many people from this area and we need to get out to these neighborhoods. And how can we do that?’ And then they would come up with their own suggestions … it floored me! Partnership Processes Influence Unintended Achievements As noted in the example provided in the previous theme, intentionality in partnerships can still result in unexpected benefits and achievements. When specifically asked about unintended achievements in the concept mapping method, participants further indicated the influence of partnership processes on stimulating these achievements (Figure 2). Participants noted that Trusting the Process the group had established in regard to communication, decision-making and resolving conflict can be an important achievement in these partnerships. Participants reinforced the importance of Knowledge Generation for both academic researchers and community members, but they also cited how these partnerships can help to Build New Relationships as well as Personal and Professional Fulfillment. One academic interview participant remarked: … this is honestly a feel good thing. It is so much more satisfying personally and professionally. It’s more fun, it’s more fulfilling. I actually feel good about this work. I feel that it’s not some grand rocket science, but in a small way it’s making a huge difference for this small community. 10 | Gateways | Lindquist-Grantz & Vaughn A few participants noted that Expansion beyond the initial research project was unexpected, but was perceived as a sign of achievement. In one example, a community participant said of the academic researchers representing two different institutions, ‘I will say that we engage both [academic partners] in a variety of work because of how well the one specific project went’. An academic participant also noted: ‘It’s expanding in scope; it’s not stagnant. We’re not staying where we started. It’s constantly evolving in its roles … we’re being asked to participate in different things than we would’ve previously been asked to participate in.’ Success Defined by Tangible Products and Outcomes Group processes and functioning were perceived as important factors in partnership success, but when ultimately determining whether a partnership was successful, members primarily based this on improvements in measurable research outcomes and the development of tangible products (Tangible Outcomes/Products; Figure 3). Examples provided by study participants included reports to the community or program and policy plans. As succinctly described by one academic participant: … ultimately it should be about health outcomes. The one that’s hardest to get to and the one that takes the longest. I guess, again, if you see this as a long-term process that would be where we’re headed. Because otherwise, why do it? In speaking about their particular partnership, another academic participant commented: In the end the research had to be written up and presented and published. And also getting grants to do this … there were tangible things that you could list as outcomes and products of the partnership … and we had that so that was, that was our success. Figure 2: Based on my experience, unintended but important achievements in community-academic research partnerships 11 | Gateways | Lindquist-Grantz & Vaughn Tangible Outcomes/Products were considered indicators that the partnership was benefiting the target community and resulting in meaningful change (Community Benefit and Meaningful Change; Figure 3). An academic participant stated: Adding to the success of the partnership, if we’re able to show that not only do we have these cool research outcomes, but that the intervention we did actually made a difference. So an actual health improvement outcome. That would really demonstrate the importance of this type of work. Not only are we able to get the data but we’re able to get higher quality data and more effective interventions out of it. In connecting the relational components of Strong Partnerships and Mutual Engagement in Research with Tangible Outcomes/Products, a community member remarked: I also think there’s this whole relational piece. Are we really benefitting from one another? We are still working for the community, both organizations. Universities and colleges are at some level working for the community. They need to have good community will. And they need the community to succeed to succeed themselves. Partnership Processes as Informal Evaluation Methods In interviews, participants described relying on informal methods of evaluating partnership success until a tangible product or outcome was achieved. Essentially, they used their intuition based on equitable and well-integrated partnership processes until those actionable research goals were achieved. These informal methods were described in a few of the statements offered by participants: But for me it’s just a gut [pause]. Do they trust you? Are you involved in the conversation? When you say something, are your comments thoughtfully regarded, whether or not they’re taken? Are you considered someone worth contemplating, or ideas worth contemplating? (Academic participant). … we always respect our partners. We respect their intelligence. We respect their expertise. And there’s sometimes where you just have to go with a gut instinct (Community participant). Figure 3: Characteristics of a partnered approach to research that is working 12 | Gateways | Lindquist-Grantz & Vaughn … we wouldn’t have the research without the partnership because we wouldn’t have access to the schools that we have access to. We wouldn’t be working with the kids we’re working with. But we’ve worked for years to establish trust and commitment, to establish the relationship that allows us to do this cool research (Academic participant). The outcome in so many ways is the partnership. I mean, it’s that continued collaboration; that continued conversation … so I think maintaining the partnership is an outcome (Academic participant). These findings were further reflected in the concept mapping. When there was a Lack of Outcomes and Change, participants deemed the partnership unsuccessful. The other factors that contributed to a perceived lack of success included Unequal Distribution of Power/Work, Lack of Commitment to Project/Partnership, Lack of Participation amongst members of the partnership and Poor Communication (Figure 4). According to one academic participant, they had to address these issues early in the partnership: And the communication, the fact that we were in touch, that we were resolving any issues as they arose. That was important. And I think that improved the outcomes and the fact that we achieved the goals that we set out to do. In another case where the partnership did not nurture group processes either in the beginning or as conflict arose, there were negative consequences, and the partnership was perceived to be negatively affected. The academic participant described the impact on the reach of the research and relationships within the partnership: I don’t know. I mean, I think it would’ve been different if we all sat down at the table together, including the community coordinators. And a conversation about a long-term goal, or a partnership-y or process-y goal rather than just an explicit research objective. And that probably would’ve been, now looking back at it, maybe that could’ve saved us from where we are now. Figure 4: Characteristics of a partnered approach to research that is not working 13 | Gateways | Lindquist-Grantz & Vaughn DISCUSSION The current study findings are consistent with literature that stresses the importance of partnership processes in community- academic research partnerships (Becker et al. 2013; Duran et al. 2013; Hicks et al. 2012; Israel, Coombe & McGranaghan 2010; Israel et al. 2010; Israel et al. 2013; Jagosh et al. 2015; Lasker, Weiss & Miller 2001; Lucero et al. 2016; Malone et al. 2013; Roman Isler & Corbie-Smith 2012; Sandoval et al. 2012; Schulz, Israel & Lantz 2003; Viswanathan et al. 2004; Wallerstein et al. 2008) and further emphasise the necessity of nurturing these processes and relationships. This study also advances our understanding of the perspectives of community members and academic researchers involved in partnerships regarding the outcomes of their work together and the ways in which they determine whether the partnered approach to research is effective. The main themes from the concept maps and interviews suggest that, although members of these partnerships consider group processes to be the foundation of their work and the primary stimulus for achieving intended research outcomes, a project is only stamped as ‘successful’ if members are able to prove intended research outcomes or produce a tangible item to show the fruits of their labour. Simultaneously, however, participants say that, until those outcomes are achieved or products are developed, they are informally and individually evaluating their progress based on the functioning of the partnership. This may suggest that when partners are silently determining the partnership is not working participation in the research process may be negatively affected. More importantly, the lack of engagement as a result of informal evaluation could ultimately impact the achievement of primary research outcomes as members lose interest or do not push the project beyond the initial stages; therefore, affecting the one determinant of a successful partnership. In an article published after the current study was conducted, Jagosh et al. (2015) describe the interdependence of partnership processes and functioning and changes in long- term population-level outcomes. In their description of a ‘ripple effect’ concept, they posit that factors influencing partnership functioning, such as trust among group members, can be a component within the context of the partnership, a mechanism that propels partnership activities and an outcome of the partnership, even if it has manifested within the context and mechanism of the model. The descriptions of community-academic partnership success, identification of intermediate outcomes and the difficulty many interview respondents in the current study had in differentiating between process outcomes and research outcomes support the context–mechanism–outcome configuration developed by Jagosh et al. (2015). The findings in this study expand on existing evidence about the important role of partnership processes related to outcomes (Becker et al. 2013; Chang et al. 2013; Duran et al. 2013; 14 | Gateways | Lindquist-Grantz & Vaughn Hicks et al. 2012; Israel et al. 1995; Israel et al. 2010; Israel et al. 2013; Lucero et al. 2016; Schulz, Israel & Lantz 2003; Udoh et al. 2013; Wallerstein et al. 2008). This especially means that the unique contexts and needs of individual partnerships should be considered (Chang et al. 2013). Although members of partnerships represented in the current study ultimately wanted to achieve improved individual- and community-level research outcomes, there was strong acknowledgement that in order for this to occur through the partnership there needed to be an emphasis on relationships and partnership processes. These processes, and the functioning of relationships within the partnership, were perceived as critical components of the work, and members used these as intermediaries to gauge whether their efforts were a success until it became possible to evaluate more long-lasting systemic changes. As such, previous references to these factors as ‘intermediate outcomes’ (Jagosh et al. 2015; Roman Isler & Corbie-Smith 2012; Schulz, Israel & Lantz 2003) seem to be the most fitting, as opposed to ‘unintended’ or ‘secondary outcomes’. The latter terms suggest a lack of intentionality in building and maintaining relationships and processes within the partnership, whereas the findings in this study further emphasise the need for members of community-academic partnerships to be quite intentional in their attention to these factors as a means of transforming the community. Furthermore, there are likely to be cases where a partnership has achieved outcomes which it never anticipated; therefore, the term should be reserved for those situations. The partnership covenant developed by the Oakland Late Diagnosis Team (Udoh et al. 2013) is an exemplar of how members of community-academic research partnerships can work together to be more intentional with regard to relationships, partnership processes and core principles of participatory research. Many quantitative instruments for measuring coalition functioning and group dynamics in community-academic research partnerships are available (Granner & Sharpe 2004; Sandoval et al. 2012) and calls have been made for the use of qualitative methods as well (Sandoval et al. 2012). Yet, participants in this study relied on their intuition to determine how the partnership was performing in advance of measurable health and research outcomes. Members of community-academic partnerships should consider ways in which they may monitor, or evaluate, intermediate outcomes as a part of their work in order to sustain partnership momentum for long-term change (Butterfoss & Kegler 2002; Israel, Coombe & McGranaghan 2010; Schulz, Israel & Lantz 2003). This recommendation is consistent with literature on the importance of the implementation and maintenance phase of collaborative groups, including establishing formalised rules and procedures, monitoring member satisfaction and engagement, and evaluating processes that lead to outcomes (Butterfoss, Goodman & Wandersman 1993; Butterfoss & Kegler 2002; Levison-Johnson, Dewey & Wandersman 2009; Mattessich & Monsey 1992; Wandersman, Goodman & Butterfoss 1997). Establishing a process 15 | Gateways | Lindquist-Grantz & Vaughn evaluation at the start of a partnership can empower group members to express and examine concerns as the work progresses (Fetterman 1996; Israel et al. 2010; Schulz, Israel & Lantz 2003) in order to thwart the building of tensions. An agreed upon system for reflecting on the shared work can help to build synergy that contributes to partnership longevity, subsequent research projects and long-lasting benefits to the community (Duran et al. 2013; Jagosh et al. 2015; Udoh et al. 2013), all of which were considered signs of partnership success. Although limited in sample size, the current study provided an initial exploration of how members of community-academic research partnerships are defining partnership success. The findings are not generalisable to every community-academic research partnership and community; however, they may be useful for future studies aiming to connect the operationalisation of community engaged research to collective impact. The findings also provide an initial indication of the extent to which these partnerships are evaluating and discussing their work. Additional prompts may provide further examination of how members of partnerships implement and operationalise the pathway from process to outcomes. Future study would also help to understand and delineate how community-academic research partnerships across the continuum of community engagement define success and assess the impact of their efforts. Leaders in the field of participatory research have called for additional studies and methods to examine the links between partnership processes and outcomes (Sandoval et al. 2012; Viswanathan et al. 2004). In a recent study, Lucero and colleagues (2016) developed a mixed methods ‘iterative integration approach’ to understand how partnership processes connect with health outcomes. Their use of this innovative approach mimicked the cyclical nature of analysis, action and reflection encouraged in community-academic research partnerships to provide new insight into this complex topic. The concept mapping methodology combined with interviews in the current study also offered a novel mixed methods approach for exploring connections between processes and outcomes across partnerships. Future studies might consider using these methods as part of a sequential mixed methods design so that interview questions could specifically explore concept mapping findings. The participatory nature of concept mapping is also useful in the evaluation of complex programs and initiatives that require stakeholder input (Kane & Trochim 2007). As such, concept mapping – alone or in combination with other research methods – could also have utility within evaluations of individual community-academic research partnerships to address the need for iterative monitoring of processes and intermediate outcomes so that primary research outcomes can be attained. The use of an online concept mapping method was appropriate for the scope of this pilot study, but was not without limitations. As cited by Kane and Trochim (2007), the remote 16 | Gateways | Lindquist-Grantz & Vaughn generation of ideas allows the researcher to reach a broad stakeholder group who can record their ideas in their own environment and in their own time; however, in the present study, it seemed to have a negative impact on the number of stakeholders who participated. Future studies would benefit from broader participation of community members and academics as a whole, but especially from community members to ensure their perspectives are equally represented. Increased participant involvement would be especially important in the sorting phase to promote their involvement in determining the most appropriate cluster solutions and naming of clusters in the concept maps. In- person participation may help with these issues. It may also assist with the timing issue as the participatory aspects of the concept mapping could be done within group sessions, alleviating the possibility of tasks lingering as participants complete them in their own time. Lastly, greater participation in concept mapping may be more achievable if used within individual partnership evaluations than in studies across multiple partnerships. In these settings, members of the partnership can better negotiate the timing of concept mapping and interview methods so that they align with other meetings and activities. In addition to the considerations discussed above, direct contact with community members, as opposed to snowball email messaging, may be useful in better explaining the purpose and utility of the study. Computer and internet access, as well as comfort with technology and computer software, should also be considered when deciding whether to do concept mapping remotely or in person. The distribution of incentives – either monetary or in the form of other resources, such as child care – may also promote greater participation, particularly if concept mapping is conducted in person. Use of in-person concept mapping and the strategies noted previously may also have resulted in a greater number of participants available for the individual interviews. The lack of community member input in the individual interviews limited our ability to fully explore whether the perspectives of community and academic members differ. As a result, triangulation between concept mapping and interview responses may be biased toward academic perspectives. Future studies should include a larger sample of community members so that similarities and differences amongst perspectives can be explored in more detail. This study aimed to examine community-academic research partnerships as a whole, but future studies could consider differences in partnerships based on the continuum of community engagement in research (e.g. community-placed vs community-based participatory research). CONCLUSION Relationships, group processes and group functioning influence how members of community-academic research partnerships describe the value of partnerships; however, aside from notable 17 | Gateways | Lindquist-Grantz & Vaughn exceptions (Chang et al. 2013; Duran et al. 2013; Hicks et al. 2012; Israel et al. 1995; Israel et al. 2010; Lucero et al. 2016; Schulz, Israel & Lantz 2003; Udoh et al. 2013; Wallerstein et al. 2008), they may not be explicitly discussed within partnerships. Members report that they rely on intuitive, unintentional and unspoken methods of determining partnership success, particularly in the early phases of research that precede the availability of long-term community- level health and social outcome data. Although partnership processes and functioning are highly regarded by members, when ultimately determining whether the partnership has been successful, members rely on measureable long-term community- level health and social outcomes and the development of tangible products, such as programs, community reports or policy changes. As such, the context of the partnership and processes developed by members should be considered intermediate outcomes and critical influencers of sustained engagement in the partnership so that primary long-term outcomes are achieved. Academics and community members collaborating in research partnerships need to be aware of the power and importance of partnership processes and cultivate them as much as the achievement of research outcomes. The co-creation of transparent process evaluation methods that can be regularly monitored and discussed by all members may assist with fostering open communication about expectations for primary research outcomes and partnership processes as intermediate outcomes. REFERENCES Ahmed, S & Palermo, A 2010, ‘Community engagement in research: Frameworks for education and peer review’, American Journal of Public Health, vol. 100, no. 8, pp. 1380–1387. doi: http://dx.doi.org/10.2105/ AJPH.2009.178137 Becker, A, Israel, B, Gustat, J, Reyes, A & Allen, A 2013, ‘Strategies and techniques for effective group process in CBPR partnerships’, in B Israel, E Eng, A Schulz, E Parker (eds), Methods for community-based participatory research for health, Jossey-Bass, San Francisco, CA, pp. 69–96. Brugge, D, Rivera-Carrasco, E, Zotter, J & Leung, A 2010, ‘Community- based participatory research in Boston’s neighborhoods: A review of asthma case examples’, Archives of Environmental & Occupational Health, vol. 65, no. 1, pp. 38–44. doi: http://dx.doi.org/10.1080/19338240903390214 Burke, J, O’Campo, P, Peak, G, Gielen, A, McDonnell, K & Trochim, W 2005, ‘An introduction to concept mapping as a participatory public health research method’, Qualitative Health Research, vol. 15, no. 10, pp. 1392–1410. doi: http://dx.doi.org/10.1177/1049732305278876 Butterfoss, F, Goodman, R & Wandersman, A 1993, ‘Community coalitions for prevention and health promotion’, Health Education Research: Theory & Practice, vol. 8, no. 3, pp. 315–30. Butterfoss, F & Kegler, M 2002, ‘Toward a comprehensive understanding of community coalitions’, in R DiClemente, R Crosby & M Kegler (eds), Emerging theories in health promotion practice and research, Jossey-Bass, San Francisco, CA, pp. 157–93. Cacari-Stone, L, Wallerstein, N, Garcia, A & Minkler, M 2014, ‘The promise of community-based participatory research for health equity: A 18 | Gateways | Lindquist-Grantz & Vaughn conceptual model for bridging evidence with policy’, American Journal of Public Health, vol. 104, no. 9, pp. 1615–23. doi: http://dx.doi.org/10.2105/ AJPH.2014.301961 Chang, C, Salvatore, A, Lee, P, Liu, S, Tom, A, Morales, A, Baker R & Minkler, M 2013, ‘Adapting to context in community-based participatory research: “Participatory starting points” in a Chinese immigrant worker community’, American Journal of Community Psychology, vol. 51, nos 3–4, pp. 480–91. doi: http://dx.doi.org/10.1007/s10464-012-9565-z Cheezum, R, Coombe, C, Israel, B, McGranaghan, R, Burris, A, Grant- White, S, Weigl, A & Anderson, M 2013, ‘Building community capacity to advocate for policy change: An outcome evaluation of the Neighborhoods Working in Partnership project in Detroit’, Journal of Community Practice, vol. 21, no. 3, pp. 228–47. doi: http://dx.doi.org/10.1080/10705422.2013.8 11624 Creswell, J & Plano Clark, V 2011, Designing and conducting mixed methods research, 2nd edn, Sage, Thousand Oaks, CA. de Leeuw, J & Mair, P 2009, ‘Multidimensional scaling using majorization: SMACOF in R’, Journal of Statistical Software, vol. 31, no. 3, pp. 1–30, viewed 17 August 2015, http://www.jstatsoft.org/v31/i03/ Duran, B, Wallerstein, N, Avila, M, Belone, L, Minkler, M & Foley, K 2013, ‘Developing and maintaining partnerships in communities’, in B Israel, E Eng, A Schulz, E Parker (eds), Methods for community-based participatory research for health, Jossey-Bass, San Francisco, CA, pp. 41–68. El Ansari, W & Weiss, E 2006, ‘Quality of research on community partnerships: Developing the evidence base’, Health Education Research, vol. 21, no. 2, pp. 175–80. doi: http://dx.doi.org/10.1093/her/cyh051 Fetterman, D 1996, ‘Empowerment evaluation: An introduction to theory and practice’, in D Fetterman, S Kaftarian & A Wandersman (eds), Empowerment evaluation: Knowledge and tools for self-assessment and accountability, Sage, Thousand Oaks, CA, pp. 3–46. doi: http://dx.doi. org/10.4135/9781452243573.n1 Granner, M & Sharpe, P 2004, ‘Evaluating community coalition characteristics and functioning: A summary of measurement tools’, Health Education Research, vol. 19, no. 5, pp. 514–32. doi: http://dx.doi. org/10.1093/her/cyg056 Greene, J, Caracelli, V & Graham, W 1989, ‘Toward a conceptual framework for mixed-method evaluation designs’, Educational Evaluation and Policy Analysis, vol. 11, no. 3, pp. 255–74. doi: http://dx.doi. org/10.3102/01623737011003255 and http://dx.doi.org/10.2307/1163620 Hacker, K, Tendulkar, S, Rideout, C, Bhuiya, N, Trinh-Shevrin, C, Savage, C, Grullon, M, Strelnick, H, Leung, C & DiGirolamo, A 2012, ‘Community capacity building and sustainability: Outcomes of community-based participatory research’, Progress in Community Health Partnerships, vol. 6, no. 3, pp. 349–60. doi: http://dx.doi.org/10.1353/cpr.2012.0048 Hicks, S, Duran, B, Wallerstein, N, Avila, M, Belone, L, Lucero, J, Magarati, M, Mainer, E, Martin, D, Muhammad, M, Oetzel, J, Pearson, C, Sahota, P, Simonds, V, Sussman A, Tafoya, G & White Hat, E 2012, ‘Evaluating community-based participatory research to improve community- partnered science and community health’, Progress in Community Health Partnerships, vol. 6, no. 3, pp. 289–299. doi: http://dx.doi.org/10.1353/ cpr.2012.0049 Israel, B, Coombe, C, Cheezum, R, Schulz, A, McGranaghan, R, Lichtenstein, R, Reyes, A, Clement, J & Burris, A 2010, ‘Community-based 19 | Gateways | Lindquist-Grantz & Vaughn participatory research: A capacity-building approach for policy advocacy aimed at eliminating health disparities’, American Journal of Public Health, vol. 100, no. 11, pp. 2094–2102. doi: http://dx.doi.org/10.2105/ AJPH.2009.170506 Israel, B, Coombe, C & McGranaghan, R 2010, ‘Community-based participatory research: A partnership approach for public health’, CD- ROM, http://sitemaker.umich.edu/cbpr/home Israel, B, Cummings, K, Dignan, M, Heaney, C, Perales, D, Simons-Morton, B & Zimmerman, M 1995, ‘Evaluation of health education programs: Current assessment and future directions’, Health Education Quarterly, vol. 22, no. 2, pp. 364–89. doi: http://dx.doi.org/10.1177/109019819402200308 Israel, B, Eng, E, Schulz, A & Parker, E 2013, ‘Introduction to methods for CBPR for health’, in B Israel, E Eng, A Schulz & E Parker (eds), Methods for community-based participatory research for health, Jossey-Bass, San Francisco, CA, pp. 3–37. Jackson, K & Trochim, W 2002, ‘Concept mapping as an alternative approach for the analysis of open-ended survey responses’, Organizational Research Methods, vol. 5, pp. 307–36. doi: http://dx.doi. org/10.1177/109442802237114 Jagosh, J, Bush, P, Salsberg, J, Macaulay, A, Greenhalgh, T, Wong, G, Cargo, M, Green, L, Herbert & C, Pluye, P 2015, ‘A realist evaluation of community-based participatory research: Partnership synergy, trust building and related ripple effects’, BMC Public Health, vol. 15, p. 725. doi: http://dx.doi.org/10.1186/s12889-015-1949-1 Kane, M & Trochim, W 2007, Concept mapping for planning and evaluation, Sage, Thousand Oaks, CA. doi: http://dx.doi.org/10.4135/9781412983730 Kegler, M, Norton, B & Aronson, R 2008, ‘Achieving organizational change: Findings from case studies of 20 California healthy cities and communities coalitions’, Health Promotion International, vol. 23, no. 2, pp. 109–18. doi: http://dx.doi.org/10.1093/heapro/dan009 Kelly, C, Baker, E, Brownson, R & Schootman, M 2007, ‘Translating research into practice: Using concept mapping to determine locally relevant intervention strategies to increase physical activity’, Evaluation and Program Planning, vol. 30, pp. 282–93. doi: http://dx.doi.org/10.1016/j. evalprogplan.2007.05.007 Lasker, R, Weiss, E & Miller, R 2001, ‘Partnership synergy: A practical framework for studying and strengthening the collaborative advantage’, Milbank Quarterly, vol. 79, no. 2, pp. 179–205. doi: http://dx.doi. org/10.1111/1468-0009.00203 Levison-Johnson, J, Dewey, J & Wandersman, A 2009, Getting to Outcomes® in systems of care: 10 steps for achieving results-based accountability, ICF Macro, Atlanta, GA. Lucero, J, Wallerstein, N, Duran, B, Alegria, M, Greene-Moton, E, Israel, B, Kastelic, M, Margarati, M, Oetzel, J, Pearson, C, Schulz, A, Villegas, M & White Hat, E 2016, ‘Development of a mixed methods investigation of process and outcomes of community-based participatory research’, Journal of Mixed Methods Research, pp. 1–20. doi: http://dx.doi. org/10.1177/1558689816633309 Malone, R, McGruder, C, Froelicher, E & Yerger, V 2013, ‘Being part of something: Transformative outcomes of a community-based participatory study’, Health Promotion Practice, vol. 14, no. 2, pp. 205–12. doi: http:// dx.doi.org/10.1177/1524839912443242 20 | Gateways | Lindquist-Grantz & Vaughn Mattessich, P & Monsey, B 1992, Collaboration: What makes it work: A review of literature on factors influencing successful collaboration, Amherst H Wilder Foundation, St Paul, MN. McFall, S, Ureda, J, Byrd, T, Valdes, A, Morales, P, Scott, D & Chan, E 2009, ‘What is needed for informed decisions about prostate cancer screening: Perspectives of African-American and Hispanic men’, Health Education Research, vol. 24, pp. 280–91. doi http://dx.doi.org/10.1093/her/cyn018 Miles, M, Huberman, A & Saldana, J 2014, Qualitative data analysis: A methods sourcebook, Sage, Thousand Oaks, CA. Moustakas, C 1994, Phenomenological research methods, Sage, Thousand Oaks, CA. doi: http://dx.doi.org/10.4135/9781412995658 National Institutes of Health, U.S. Department of Health and Human Services 2011, Principles of Community Engagement, 2nd edn, viewed 22 October 2014, www.atsdr.cdc.gov/communityengagement/pdf/PCE_ Report_508_FINAL.pdf Nichols, N, Anucha, U, Houwer, R & Wood, M 2013, ‘Building equitable community-academic research collaborations: Learning together through tensions and contradictions’, Gateways: International Journal of Community Research and Engagement, vol. 6, pp. 57–76, doi: http://dx.doi.org/10.5130/ ijcre.v6i1.2822 R Development Core Team 2011, R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria. Rasmus, S 2014, ‘Indigenizing CBPR: Evaluation of a community-based and participatory research process implementation of the Elluam Tungiinun (Towards Wellness) program in Alaska’, American Journal of Community Psychology, vol. 54, vols 1–2, pp. 170–79. doi: http://dx.doi. org/10.1007/s10464-014-9653-3 Roman Isler, M & Corbie-Smith, G 2012, ‘Practical steps to community engaged research: From inputs to outcomes’, The Journal of Law, Medicine & Ethics, vol. 40, no. 4, pp. 904–14. doi: http://dx.doi.org/10.1111/j.1748- 720X.2012.00719.x Rosas, S & Kane, M 2012, ‘Quality and rigor of the concept mapping methodology: A pooled study analysis’, Evaluation and Program Planning, vol. 35, no. 2, pp. 236–45. doi: http://dx.doi.org/10.1016/j. evalprogplan.2011.10.003 Rosenberg, S & Moonja Park, K 1975, ‘The method of sorting as a data gathering procedure in multivariate research’, Multivariate Behavioral Research, vol. 10, no. 4, pp. 489–502. doi: http://dx.doi.org/10.1207/ s15327906mbr1004_7 Rubin, H & Rubin, I 2012, Qualitative interviewing: The art of hearing data, 3rd edn, Sage, Los Angeles, CA. Sandoval, J, Lucero, J, Oetzel, J, Avila, M, Belone, L, Mau, M, Pearson, C, Tafoya, G, Duran, B & Iglesias Rios, L, Wallerstein, N 2012, ‘Process and outcome constructs for evaluating community-based participatory research projects: A matrix of existing measures’, Health Education Research, vol. 27, no. 4, pp. 680–90. doi: http://dx.doi.org/10.1093/her/ cyr087 Schulz, A, Israel, B & Lantz, P 2003, ‘Instrument for evaluating dimensions of group dynamics within community-based participatory research partnerships’, Evaluation and Program Planning, vol. 26, no. 3, pp. 249–62. doi: http://dx.doi.org/10.1016/S0149-7189(03)00029-6 21 | Gateways | Lindquist-Grantz & Vaughn Trochim, W 1989, ‘An introduction to concept mapping for planning and evaluation’, Evaluation and Program Planning, vol. 12, no. 1, pp. 1–16. doi: http://dx.doi.org/10.1016/0149-7189(89)90016-5 Trochim W & Kane M 2005, ‘Concept mapping: An introduction to structured conceptualization in health care’, International Journal of Quality in Health Care, vol. 17, no. 3, pp. 187–91. doi: http://dx.doi.org/10.1093/ intqhc/mzi038 Trochim, W & Linton, R 1986, ‘Conceptualization for planning and evaluation’, Evaluation and Program Planning, vol. 9, no. 4, pp. 289–308. doi: http://dx.doi.org/10.1016/0149-7189(86)90044-3 Udoh, I, Chopel, A, Minkler, M, Smith, C, Walton, S, Grijalva, C, Yusuf, A & Dunbar, M 2013, ‘Developing and using a partnership covenant to guide a study of late HIV diagnosis’, Progress in Community Health Partnerships: Research, Education, and Action, vol. 7, no. 4, pp. 403–11. doi: http://dx.doi.org/10.1353/cpr.2013.0053 Vaughn, L, Jacquez, F & McLinden, D 2013, ‘The use of concept mapping to identify community-driven intervention strategies for physical and mental health’, Health Promotion Practice, vol. 14, no. 5, pp. 675–85. doi: http://dx.doi.org/10.1177/1524839912462871 Vaughn, L & McLinden, D 2016, ‘Concept mapping’, in L Jason & D Glenwick (eds), Handbook of methodological approaches to community-based research: Qualitative, quantitative, and mixed methods, Oxford University Press, Oxford, UK, pp. 305–14. Viswanathan, M, Ammerman, A, Eng, E, Gartlehner, G, Lohr, K, Griffith, D, Rhodes, S, Samuel Hodge, C, Maty, S, Lux, L, Web, L, Sutton, S, Swinson, T, Jackman, A & Whitener, L 2004, Community-based participatory research: Assessing the evidence, AHRQ Publication No. 04-E022-2. Wallerstein, N & Duran, B 2008, ‘The theoretical, historical, and practice roots of CBPR’, in M Minkler & N Wallerstein (eds), Community-based participatory research for health: From process to outcomes, Jossey-Bass, San Francisco, CA, pp. 25–46. Wallerstein, N, Oetzel, J, Duran, B, Tafoya, G, Belone, L & Rae, R 2008, ‘What predicts outcomes in CBPR?’, in M Minkler & N Wallerstein (eds), Community-based participatory research for health: From process to outcomes, Jossey-Bass, San Francisco, CA, pp. 371–92. Wandersman, A, Goodman, R & Butterfoss, F 1997, ‘Understanding coalitions and how they operate’, in M Minkler (ed.), Community organizing and community building for health, Rutgers University Press, New Brunswick, NJ, pp. 261–77. Weller, S & Romney, A 1998, Systematic data collection, Sage, Newbury Park, CA.