The Grande Prairie PACT program evaluation: Discrepancy between model evaluation practice and constrained real world evaluation of crime prevention in small communities International Journal of Child, Youth and Family Studies (2013) 1: 136–146 136 THE GRANDE PRAIRIE PACT PROGRAM EVALUATION: DISCREPANCY BETWEEN MODEL EVALUATION PRACTICE AND CONSTRAINED REAL WORLD EVALUATION OF CRIME PREVENTION IN SMALL COMMUNITIES Crystal Hincks, Anne Miller, and Monica Pauls Abstract: This article discusses and demonstrates the discrepancies between ideal, theoretical program evaluation processes and real world evaluation practice, which is constrained by numerous and varying factors. The article describes the real world experience of Mount Royal University’s Centre for Criminology and Justice Research researchers in conducting an evaluation of the Police and Crisis Team (PACT) in Grande Prairie, Alberta, including a Social Return on Investment (SROI) analysis. PACT, which partners a Royal Canadian Mounted Police (RCMP) officer with a mental health professional, represents a blend of secondary and tertiary crime prevention and attempts to diminish crime in the community by addressing the risk factors of individuals with mental health concerns (creating trust with individuals, increasing awareness of resources, and decreasing stigmatization in the community). PACT also specifically targets those individuals with mental health issues who are in contact with the law to try to decrease recidivism and increase community safety. Challenges were present in the evaluation due to the time frame, staff turnover, program start-up issues, and confidentiality and sensitivity of the program focus. Despite the challenges, the CCJR team completed an evaluation including a forecast SROI, identifying several successes, challenges, and recommendations for change. Keywords: evaluation, challenges, mental health, social programs, Alberta Safe Communities Innovation Fund (SCIF), Police and Crisis Team (PACT) Crystal Hincks, is a research associate at the Centre for Criminology and Justice Research at Mount Royal University, T154, 4825 Mount Royal Gate SW, Calgary, AB, T3E 6K6. She is currently a Masters candidate in the Department of Sociology at the University of Calgary. E-mail: crystalhincks@gmail.com Anne Miller is a research associate at the Centre for Criminology and Justice Research at Mount Royal University. She completed her Masters in Public Policy and Public Administration from Concordia University in Montréal and is one of two practitioners in North America accredited by the Social Return On Investment (SROI) Network in London. E-mail: miller.annechristine@gmail.com Monica Pauls, M.A. is a research associate at the Centre for Criminology and Justice Research at Mount Royal University. She also teaches at Mount Royal’s Departments of Justice Studies and Child and Youth Studies. E-mail: mpauls@mtroyal.ca mailto:crystalhincks@gmail.com� International Journal of Child, Youth and Family Studies (2013) 1: 136–146 137 Evaluation is an ongoing process, which starts with the development of an initiative and continues throughout the life of a project and beyond. Utilized as a tool to support and enhance, evaluation should provide a means for continual monitoring, diagnosis, and change (Powell, 2006). A rigorous, systematic evaluation framework should be part of the overall planning process of an organization or program. Powell (2006) further notes that solid evaluation monitors progress towards goals and objectives, measures effectiveness, and is integral to the assessment of services and resources. Evaluation guides strategies for change and is critical for program sustainability and replication. Since its inception in January 2010, the Centre for Criminology and Justice Research (CCJR) at Mount Royal University has undertaken over 20 evaluation projects, working from the foundation of this philosophy. However, the real world is often different from learned theory and the circumstances of program evaluation often limit research activities, data collection and analyses. Such is the challenge of engaging in applied research. Academic research centres, like the CCJR, when undertaking these types of projects, should demonstrate flexibility and creativity in order to provide an accurate picture of program effectiveness. This article discusses the experience of conducting applied research in the face of various restrictions and limitations. The authors focus on the evaluation of one particular program, the Police and Crisis Team (PACT) in Grande Prairie, Alberta, to demonstrate how the research was achieved despite several challenges. Background Police and Crisis Teams (PACT) already operate in the two major cities in Alberta, Calgary and Edmonton. Similarly, the Grande Prairie PACT project was developed to respond to the needs of the rural population in the Peace Country region. The mobile community-based crisis team addresses the mental health needs of individuals who come into contact with the law. The number of people with mental illness involved in the justice system is increasing, and the police service in this area lacks the resources and time to deal effectively with this population. Utilizing PACT to respond to these incidents allows for a more appropriate intervention, leading to a reduction in the number of people with mental illnesses involved in the criminal justice system and to an increase in the use of effective community supports. This represents a blend of secondary and tertiary crime prevention. The program attempts to diminish crime in the community by addressing the risk factors of individuals with mental health concerns through creating trust with individuals, increasing their awareness of resources, and decreasing stigmatization in the community; it also specifically targets those individuals with mental health issues who are in contact with the law to try to decrease recidivism and increase community safety. In 2009, Grande Prairie PACT secured a grant from the Safe Communities Innovation Fund (SCIF) of the Alberta Government to run a three-year pilot from 2009 to 2012 (Grande Prairie PACT, 2009). As part of the grant requirements, the pilot was to be evaluated from both a International Journal of Child, Youth and Family Studies (2013) 1: 136–146 138 formative and summative perspective, as well as to have a Social Return On Investment (SROI) analysis conducted. The Centre for Criminology and Justice Research (CCJR) of Mount Royal University was contracted to conduct the evaluation. Although the project was originally funded as a three-year pilot starting in 2009, the CCJR was contracted in July 2011, giving them a four-month time frame to conduct the evaluation. This was the first of several challenges faced by the evaluation team. The reality of conducting the research over such a relatively short period is that the evaluators were restricted in terms of what could actually be done. For example, the evaluators could not locate and interview former PACT clients, in part due to the nature of mental illness, but also due to time constraints. In addition, because the CCJR team was not involved with the program from the start, minimal evaluation planning was included in PACT’s overall program model. Very little formal or informal documentation existed about the program’s development and implementation. This significantly limited the formative component of the evaluation, since the evaluators could only look back on what had been done, rather than working through the processes as they happened. The team was also restricted in terms of the data that was collected. The evaluators had no input into the type or quality of data, or the thoroughness of data collection; they had to work with what was already there to measure program effectiveness and this was not always consistent with the outcomes identified in the program logic model. In collaboration with the PACT team, the CCJR evaluators analyzed the existing statistical data, reviewed program materials, and conducted key informant interviews in order to document the program description, conduct an outcome analysis, and construct a forecast SROI to determine the potential social return on investment of PACT in Grande Prairie. Successes and challenges of the program were identified and discussed in the final report, as well as recommendations for change. The following sections discuss these processes in more detail. Research Activities The research team collected data for the evaluation via one phone interview with a former PACT team member, one client interview, a review of the available statistical data that the PACT team members were responsible for collecting, a review of program information (including the funding proposal, progress reports, community presentations, etc.), as well as one site visit to Grande Prairie during which 12 key informant interviews were conducted with the PACT coordinator, representatives from the RCMP and Alberta Health Services, and individuals from partner agencies. The interviews conducted during the site visit followed a schedule with questions designed to address both the formative and summative aspects of the evaluation. The interviewers used the schedule as a guide, but also encouraged respondents to speak freely and contribute any information they thought was relevant. It is important to note that the evaluators were not included in the process of selecting interviewees, but rather were offered a list of potential stakeholders to interview by the PACT Director. While it was necessary to have the International Journal of Child, Youth and Family Studies (2013) 1: 136–146 139 PACT Director offer the list as opposed to having the evaluators spend a large amount of time developing it, there is room for bias in the interviews if the Director selected only those stakeholders who would offer positive feedback of the PACT program. With respect to the data set maintained by the PACT team members, the evaluators were able to analyze the data provided in order to determine the location, length of call, length of follow-up, presenting problems, referral source, service provided, where the client was referred, as well as a number of demographic details. There were two major limitations to the data set: First, the evaluators had no input into what data was collected, and therefore some indicators that could have been used to assess different processes and outcomes were missing; second, the data set itself was incomplete for a variety of reasons, which restricted the ability to conduct a comprehensive and accurate analysis. Because of the short evaluation timeline, the evaluators were unable to complete or collect the missing data. Lastly, it is important to note that only one client was willing to be interviewed by the evaluators. It was anticipated that several clients would be interviewed in order to determine client satisfaction; however nearly all of the clients refused to be interviewed, while others agreed to interviews but then changed their minds shortly before the scheduled times. Because a major component of the PACT program is the ability of the team members to build rapport with the clients, the evaluators were unable to assess the success of the program from the view of the clients without adequate interviews. Community Context Population It is important to bring into context the community in which the PACT program operates; Grande Prairie, a small municipality located in Alberta’s Peace Region, has a population of approximately 50,000 inhabitants. The surrounding towns and rural areas for whom Grande Prairie serves as a regional hub for many amenities have a combined population of approximately 25,000. Additionally, because of the proximity of the northern oil fields, Grande Prairie has a high transient population; while the constant population remains somewhere near 50,000 people, fluctuations of the economy may add as many as 20,000 more. This can greatly affect the clientele of the PACT program. The transient population is thought to have a significant impact on drug crimes in Grande Prairie, for example; virtually every stakeholder interviewed indicated that drugs are a major concern in the Grande Prairie area, and that the majority of individuals with mental health issues have a concurrent drug or alcohol addiction as well, which often exacerbates their mental condition. This can affect the nature of the calls to the PACT program, with respect to both connecting the client to appropriate resources (i.e., mental health, addictions, or both), as well as maintaining staff safety around unpredictable clients that may be under the influence of drugs or alcohol (Lambert, 2002). The nature of Grande Prairie’s transient population contributed to the difficulty of evaluating the program. Some clients who came into contact with PACT were responded to during a crisis, but follow-up efforts were unsuccessful when those clients could not be located International Journal of Child, Youth and Family Studies (2013) 1: 136–146 140 afterwards. This not only affected the program’s ability to maintain a complete database, but also reduced the opportunity for the evaluators to conduct client interviews. Partnerships and Community Engagement According to several stakeholders, because Grande Prairie is a relatively small and “isolated” city, it was observed that the collaboration between agencies is often fluid. Most agencies have an awareness of other services offered, thus creating a broad resource network for clients. Information sharing between agencies is ongoing so clients are offered current information about services available. Most of the interviewed stakeholders indicated that while there is some awareness of other agencies, most of the familiarity between programs is based on how often those programs offer overlapping services. This is to be expected, as agencies are more likely to have in-depth knowledge of other agencies when they are working collaboratively. With respect to PACT, several stakeholders experienced a fluid relationship with PACT based on frequent use of each other’s services. Other stakeholders indicated that while they felt they had a “general” idea of what PACT was, they could not comment in-depth because they had minimal interactions with the program. The dynamics of community agency relationships can affect evaluation efforts when there is minimal knowledge about a program or agency. The interviewed stakeholders who indicated they had minimal knowledge of the PACT program expressed an interest in learning more about it once the evaluators presented an overview of the goals and functions of the program. Even if a given agency or program did not deem it beneficial to form a working relationship with PACT, this did allow the evaluators to determine why some agencies are unsuitable for partnership and thus assisted the overall evaluation. Although somewhat different from basic social research, evaluation research still requires the application of methods of research design, including measurement, internal validity, and external validity; the main difference therefore lies in the social context in which the methods are applied (Singleton & Straits, 2010). There are, however, methodological issues in evaluation research, specifically as they relate to external validity. While validity is important in establishing a connection between the program’s inputs, outputs, activities, and outcomes, there is a risk in attempting to generalize the results to other programs. Because each evaluation is specific to the organization, program, or agency, it is nearly impossible to address various threats to external validity. Specifically, evaluation researchers must rely on non-probability samples when conducting field research; selecting participants to interview is usually the result of self-selection, whereby the requirement to include certain staff, partners, and stakeholders replaces the selection of random participants. The selection of a sample that is based on availability, or the likelihood that they will offer desirable feedback, can ultimately threaten the validity of the entire program evaluation. With respect to the PACT program, the participants interviewed by the researchers were pre-selected by the director of the program and because the researchers were somewhat unfamiliar with the International Journal of Child, Youth and Family Studies (2013) 1: 136–146 141 program’s partnerships and community relationships, they were reliant on the program staff to provide the contact information of pertinent stakeholders available or willing to participate in an interview. Potentially, this may have biased the results if the director arranged interviews with participants most likely to give positive feedback about the program. Based on the mixed results of the interviews, this does not appear to have occurred. However, with evaluation research, the possibility is always there. PACT Social Return on Investment SROI Analysis Social Return on Investment (SROI) is about financially valuing the social change brought about by a program or intervention. In other words, as Nicholls, Lawlor, Neizert, and Goodspeed (2009) state: SROI measures change in ways that are relevant to the people or organizations that experience or contribute to it. It tells the story of how change is being created by measuring social, environmental and economic outcomes and uses monetary values to represent them. This enables a ratio of benefits to costs to be calculated. (p. 8) SROI analysis requires a significant understanding of the ways that outcomes are achieved, as well as data on the changes experienced by program participants. The lack of data within the PACT program provided some challenges to the use of SROI analysis (to be discussed below). Further, a proper SROI analysis is supposed to involve the program participants in identifying and valuing outcomes to ensure, first, that the analysis is representative of the actual change experienced, and second, that the analysis captures and appropriately values those changes which are most important to participants. Due to the limited time frame of the evaluation, staff turnover within the program, and freedom of information restrictions, participant involvement did not play a large role in the analysis presented here and thus limited the robustness of the findings from the SROI analysis. Social Value Creation With respect to the interventions made by PACT, social value is created in different ways. First, since PACT staff members receive special training to de-escalate situations involving individuals with mental health concerns, fewer of these individuals are brought into custody, charged, and involved in the criminal justice system. Further, beyond the de-escalation of crisis situations, PACT ensures that individuals with mental health concerns are able to access the community resources that are most appropriate to their needs, thereby reducing the impact of untreated and undiagnosed mental health concerns on society. In this way, systemic financial International Journal of Child, Youth and Family Studies (2013) 1: 136–146 142 value is created through the reduction of violent acts and criminal justice system involvement, as well as through the integration and treatment of individuals with mental health concerns. Beyond criminal justice system involvement, the RCMP in Grande Prairie benefit from the specially trained PACT team in that they spend less time trying to address issues related to mental health, including spending less time waiting in the hospital emergency room with individuals who could otherwise be sent to more appropriate services. The PACT team alleviates stresses on the RCMP and is also able to resolve issues more quickly with their specialized skills. Individuals with mental health concerns who have contact with the law also benefit from the deployment of PACT in Grande Prairie as their situations are treated with greater sensitivity and in a more appropriate manner. These individuals frequently require police intervention due to incidents brought on by their mental state, and as the PACT team responds to their repeated calls, it develops both consistency and a trusting relationship with the RCMP that helps to de-escalate situations and reduce the magnitude of crises. Further, PACT serves to educate the community, victims, and complainants about mental health concerns thereby creating greater understanding and sensitivity towards individuals suffering with mental health issues, as well as helping to repair the fear and damage from crimes committed by these individuals. Finally, with their special training, the PACT team in Grande Prairie is able to effectively intervene in crisis situations, such as attempted suicides, in order to minimize damage, decrease risk, and prevent significant social costs. In this way, a significant amount of social value is created in Grande Prairie through investment in the program. SROI Forecasting, Calculations, and Ratio This social value was measured using SROI analysis. When the evaluation was conducted, the PACT program had only been operational for one year, and much of the needed SROI data was not available. Therefore, a forecast SROI was constructed for the program. A forecast SROI takes into account any actuals gathered from the program (in this case, the first year actuals) as well as research on the results from other, comparable, programs to determine the expected return on investment for a new program. For the PACT program forecast SROI, the team began by mapping the outcomes in an impact map (which is similar to a logic model, and can help feed the development of an effective logic model). These outcomes were then assigned financial proxies that could represent the value of each change either systemically or to the stakeholders (see Table 1). Next estimates were made about the number of target stakeholders experiencing each change. Where actuals were available for things like the number of calls taken by PACT (that would otherwise have been taken by the RCMP), these figures were used in the analysis. Where no actuals were available due to the novelty of the program, and in the absence of evaluation research from other similar programs, only one stakeholder was included in the analysis in order to avoid over-claiming. In other words, in order to present the minimum possible return on investment generated through PACT, where numbers could not be based on actuals or research, only one stakeholder was counted in the analysis. Finally, estimations were made based on evaluations of other, similar, programs. International Journal of Child, Youth and Family Studies (2013) 1: 136–146 143 After estimating the total value of the investment in the PACT program, this value was discounted, outcome-by-outcome, for deadweight (i.e., the amount of change that would have happened anyway), attribution (i.e., the amount of change attributable to the actions of others), and drop-off (i.e., the amount of change that will drop off over time). Overall, a 3.5% discount rate was applied to the entire analysis to account for the volatility of the success of the program, and a 3.7% inflation rate was applied to values extended into the future. Table 1: Financial Proxies used in the PACT Forecast SROI Indicator of Outcome Financial Proxy Fewer emergency calls from clients with mental illness Cost of police call out Cost of EMS call out Less escalation of situation at emergency calls Emergency Room Visit Criminal court case Cost of pain and suffering per assault victim Less escalation of situation at emergency calls and less suicidal ideation Cost per suicide Fewer presentations to hospital ER with mental health complaints Emergency Room Visit Cost of incarceration (adult in Alberta provincial prison) Less time spent in ER by both client and RCMP awaiting psychiatric assessment Cost of additional police stay at ER Less “revolving door syndrome” for stakeholders who present to ER, and get readmitted within two weeks from discharge from hospital. Psychiatric Admission to Hospital Medical costs to treat undiagnosed mental illness Increased employment Short-term work loss due to diagnosed mental illness Increased stable housing Cost per homeless person The final forecasted SROI Ratio for the PACT program was calculated to be 2.04 : 1. This indicates that the program more than doubles its investment through the social value that it International Journal of Child, Youth and Family Studies (2013) 1: 136–146 144 creates in the community. Considering the extension of SROI analysis beyond the confines of regular cost-benefit analysis, this return is on par with similar programs, such as the Integrated Mobile Crisis Response Team (IMCRT) in Victoria, British Columbia (Baess, 2005) that has indicated that: “the average cost per case was $300.00 for situations handled by regular police intervention versus $190.00 per case when IMCRT clinicians were paired with patrol officers”, a ratio of $1.57 : $1.00. Further, while many different specific financial values were used in this calculation, the overall value of the PACT program goes beyond these values to the impact on the lives of stakeholders and community members that are in contact with the program. Challenges and the Usefulness of an SROI Forecast In a similar way to the rest of the evaluation, the SROI analysis was challenging due to a lack of data, and the absence of a data plan associated specifically with the SROI, as well as the difficulty experienced in contacting key stakeholders (namely participants). While one of the most important aspects of an SROI (i.e., forecast or evaluative) is stakeholder involvement, the remoteness of the community, the lack of follow-up possibilities, the short time frame of the evaluation, and the nature of the program were all contributing barriers to proper stakeholder involvement. In the end, this means that the forecasted SROI ratio probably undervalues the program overall. Although these challenges were present, from an evaluation and programming perspective the process of SROI analysis and the forecasted results can be extremely useful for future evaluations. The SROI impact mapping process resulted in a comprehensive model of the expected outcomes of PACT in Grande Prairie, and can feed logic model improvement and program planning. Further, determining what indicators should be tracked in order to value outcomes from the program results in a framework for future data collection and outcomes tracking. In the future, the forecast SROI can act as the basis of a proper evaluation plan leading to an increasingly better understanding of the impact of the PACT program. The forecast SROI can also help describe the program to potential investors, illustrate the value of investment to the community, and provide a clear understanding of the activities and expected outcomes of a program when dealing with high staff turnover (as is the case in the social sector in many northern communities, including Grande Prairie). Finally, it should also be kept in mind that as the program evolves, the forecast SROI can be modified and adjusted in order to continue to capture the value of the program, and actuals can be used in the analysis. Conversely, as activities or outcomes change and are valued through the established SROI framework, program improvements can be immediately valued, and actions that foster increased social value can be built upon and developed. In this way, while the SROI analysis portion of this evaluation was limited in some ways, it also presents a unique opportunity to add a significant social metric tool to the evaluation toolbox of the PACT program in Grande Prairie. International Journal of Child, Youth and Family Studies (2013) 1: 136–146 145 Conclusion Engaging in field research and evaluation is often fraught with challenges that can bring into question the validity and reliability of the outcomes. This is critical when such evaluations may, and often do, have a direct bearing on the future funding and sustainability of such programs. The focus of this study involved the evaluation of the Grande Prairie PACT program. Despite the challenges, the CCJR team completed the evaluation for the PACT program in Grande Prairie, identifying several successes, challenges, and recommendations for change. The challenges presented should not be seen as failures, but rather as areas for growth. All new programs have weaknesses and face obstacles. The ability of the program to address these issues and make necessary changes is what will determine success and sustainability. Evaluators need to adapt to all contexts and address the various inadequacies of all programs. While implementing an evaluation strategy from the start of a program and continuing to be involved throughout an initiative’s life is ideal, the reality of evaluation is not always perfect. Flexibility, creativity, and the ability to work with the program will ultimately lead to a successful evaluation. International Journal of Child, Youth and Family Studies (2013) 1: 136–146 146 References Baess, E. (2005). Integrated Mobile Crisis Response Team (IMCRT): Review of pairing police with mental health outreach services. Prepared for the Victoria City Police Department. Retrieved from http://www.pmhl.ca/webpages/reports/Pairing-report.pdf Grande Prairie PACT. (2009). Application to the Safe Communities Innovation Fund (SCIF). Grande Prairie, AB: Author. Lambert, M. T. (2002). Linking mental health and addiction services: A continuity-of-care team model. Journal of Behavioral Health Services & Research, 29(4), 433–444. Nicholls, J., Lawlor, E., Neizert, E., & Goodspeed, T. (2009). A guide to social return on investment. London: Office of the Third Sector, the Cabinet Office. Powell, R. R. (2006). Evaluation research: An overview. Library Trends, 55(1), 102–120. Singleton, R. A., & Straits, B. C. (2010). Approaches to social research (5th ed.). New York/Oxford: Oxford University Press. http://www.pmhl.ca/webpages/reports/Pairing-report.pdf�