International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 DOI: http://dx.doi.org/10.18357/ijcyfs83/4201718001 THE USE OF ADMINISTRATIVE DATA TO STUDY THE TRIPLE P – POSITIVE PARENTING PROGRAM Rubab G. Arim, Anne Guèvremont, V. Susan Dahinten, and Dafna E. Kohen Abstract: This study is among the first Canadian population-based evaluations designed to examine associations of the Triple P – Positive Parenting Program (Triple P) for mother and child outcomes at the community level. Uniquely, this study was conducted independently of program implementation, using data collected for other purposes. Three anonymized British Columbia provincial administrative data sources were used, in addition to program data collected by administrators (Island Health). The study employed a quasi-experimental design to examine benefits of Triple P at the community level, and used sociodemographic community characteristics to match 11 target communities where Triple P was implemented with comparison communities where Triple P was not implemented. The study’s design and analyses took into account the phased-in implementation of Triple P across Vancouver Island (2004–2008), drawing on pre- and post- implementation data for all of the studied communities. Hierarchical linear modeling results showed that children living in communities where the program had been administered were more likely to have been diagnosed with conduct disorders and to have used counseling services. Program intensity was not associated with any of the child health outcomes. For mothers, higher program intensity was associated with lower odds of being diagnosed with mental health conditions. Future research should continue to demonstrate the feasibility of a population-based approach and the use of secondary data along with program data to examine community-based intervention programs. Keywords: intervention, child, maternal, mental health, services, conduct disorder, anxiety, secondary data, hierarchical linear modeling Rubab G. Arim, Ph.D. (corresponding author) is a Social Science Researcher in the Health Analysis Division, Statistics Canada, 100 Tunney’s Pasture Driveway, Ottawa, ON K1A 0T6. Email: rubab.arim@canada.ca Anne Guèvremont, M.Ed. is a Research Analyst in the Health Analysis Division, Statistics Canada, 100 Tunney’s Pasture Driveway, Ottawa, ON K1A 0T6. Email: anne.guevremont@canada.ca V. Susan Dahinten, Ph.D., RN, MBA is an Associate Professor in the School of Nursing, University of British Columbia, T201-2211 Wesbrook Mall, Vancouver, BC V6T 2B5. Email: susan.dahinten@nursing.ubc.ca Dafna E. Kohen, Ph.D. is a Principal Analyst in the Health Analysis Division, Statistics Canada, 100 Tunney’s Pasture Driveway, Ottawa, ON K1A 0T6. Email: dafna.kohen@canada.ca http://dx.doi.org/10.18357/ijcyfs83/4201718001 mailto:rubab.arim@canada.ca mailto:anne.guevremont@canada.ca mailto:dafna.kohen@canada.ca International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 60 Providing parenting support is an important first step in a public health approach to fostering children’s mental health (Hiscock et al., 2008). The Triple P – Positive Parenting Program (Triple P) is a population-based, community level parenting program developed in Australia for parents of children from birth to age 16 (Sanders, 1999; Sanders, Cann, & Markie- Dadds, 2003; Sanders, Turner, & Markie-Dadds, 2002). Triple P is a population-based parenting program implemented at the community level and offered to all parents (Sanders et al., 2002). Rather than being a single program, Triple P includes five levels of intervention on a continuum of increasing intensity (see Sanders et al., 2002; Sanders et al., 2003 for details) and aims to enhance the knowledge, skills, and confidence of parents and to prevent negative parenting practices, thereby reducing behavioral, emotional, and developmental problems in children (Prinz, Sanders, Shapiro, Whitaker, & Lutzker, 2009). To date, many international evaluation studies have been conducted on Triple P, including five meta-analyses of the effectiveness of the program on parent and child outcomes at the individual level (de Graaf, Speetjens, Smit, de Wolff, & Tavecchio, 2008; Nowak & Heinrichs, 2008; Sanders, Kirby, Tellegen, & Day, 2014; Thomas & Zimmer-Gembeck, 2007; Wilson et al., 2012). Most evaluation studies have described decreases in parental levels of stress, anxiety, and depression (e.g., Calam, Sanders, Miller, Sadhnani, & Carmont, 2008; Matsumoto, Sofronoff, & Sanders, 2007; Stallman & Ralph, 2007) as well as a decrease in children’s behavior problems, such as hyperactivity and emotional problems (see Arkan, Üstün, & Güvenir, 2013; Schmidt, 2012), following program implementation. However, a recent study by Sampaio, Sarkadi, Salari, Zethraeus, and Feldman (2015) showed no significant improvements in parental mental health or child behavior problems at a 6-, 12- and 18-month follow-up. Similarly, McConnell, Breitkeuz, and Savage (2011) found no significant differences on any outcome measure including parenting stress, positive interaction, and child behavior problems. These mixed findings may be due to methodological or programmatic differences such as differences in child ages or intensity of program implementation. For example, Sampaio et al. (2015) focused on preschoolers whereas Stallman and Ralph (2007) focused on adolescents; both Sampaio et al. (2015) and McConnell et al. (2011) examined only Levels 2 and 3 of Triple P whereas other studies that found significant improvements addressed Levels 4 and 5, which are more intensive. Previously cited evaluations have several limitations. First, most studies used small samples focusing on individual level data as opposed to population-based community data even though Triple P is a community-based intervention strategy (see Fives, Pursell, Heary, Nic Gabhainn, & Canavan, 2014; Prinz et al., 2009 for exceptions). Second, few studies (see Averdijk, Zirk-Sadowski, Ribeaud, & Eisner, 2016; Heinrichs, Kliem, & Hahlweg, 2014) examined outcomes beyond 3 years post-program implementation. Third, the greater number of significant outcomes and the larger effects were found for parent-reported outcomes. Parental evaluations may be less objective than other more independent sources of evaluation, particularly if parents feel strongly invested in the program. Fourth, program intensity has not been previously considered. The present study addressed these limitations by focusing on associations International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 61 at the community level; considering both maternal and child health outcomes measured up to 6 years post-implementation; using population-based administrative data collected independently of the intervention and research study (i.e., prior to and without knowledge of Triple P using other provincial administrative data, such as data from the Ministry of Health not collected for research purposes); and including a measure of program intensity. Administrative data refer to information collected by organizations and departments for their own non-research purposes. In Canada, health care coverage is provided by the provinces. As such, each provincial ministry is responsible for the collection of population-based data such as hospitalizations, diagnostic information provided by physicians, and medical mental health service use (see Prinz et al., 2009 for an example from the United States). There are various advantages of using administrative data for research (see Dahinten, Arim, Guevremont, & Kohen, 2014). In particular, this is a data source well-suited to assess a population-based intervention. Thus, the present study compared target communities, where Triple P had been implemented, with comparison communities, where Triple P had not been implemented, to address the following three research questions:  What is the association between Triple P and selected mental health outcomes for mothers when comparing target communities with comparison communities?  What is the association between Triple P and selected health outcomes for children aged 6 to 12, when comparing target communities with comparison communities?  Is program intensity associated with outcomes for mothers and children? The choice of health outcomes was purposeful given the aims of Triple P to reduce parenting stress as well as children’s behavioral, emotional, and developmental problems. As such, we looked at mental health diagnoses and use of counseling services in mothers, as well as mental health diagnoses, conduct disorder diagnoses, use of counseling services, and injuries in children. Examining health services use is important not only for understanding health outcomes, but also for better understanding policy-relevant issues such as patterns of use and costs. In the present study, we anticipated that an examination of health services use would provide a population health comparison at a community level. To date, few studies have examined health services use together with mental health outcomes as outcomes of parenting programs (Charles, Bywater, Edwards, Hutchings, & Zou, 2013; Edwards, Céilleachair, Bywater, Hughes, & Hutchings, 2007; Prinz et al., 2009). For example, studies evaluating the Incredible Years parenting program found a decrease in children’s conduct problems (Edwards et al., 2007; Hutchings et al., 2007) and parental depression as assessed by standardized instruments (Charles et al., 2013; Hutchings et al., 2007). However, similar changes were not reflected in service use as measured by the parent-reported Client Service Receipt Inventory (Charles et al., 2013; International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 62 Edwards et al., 2007). Charles et al. (2013) indicated no difference in parental service use (i.e., primary services, social services, and hospital services) for participating families at a 6-, 12-, and 18-month follow-up, and Edwards et al. (2007) found no differences in child health (primary care and hospital services), social care, and special education service use at a 6-month follow-up for participating families. These findings highlight the importance of exploring health service use in addition to health outcomes, as well as looking past the implementation period. Method Study Design A non-randomized quasi-experimental design was used to investigate the associations of Triple P at the community level, drawing on administrative data collected in 2002 and 2010. More specifically, comparison communities (where Triple P was not implemented) and target communities (where Triple P was implemented) were matched on sociodemographic community characteristics and then compared. The design and analysis took into account the phased-in implementation of Triple P across the area served by the Vancouver Island Health Authority (Island Health) over the period from 2004 to 2008, drawing on a minimum of 2-year pre- and post-implementation data for all communities, with up to 6-year post-implementation data for some communities. Triple P was implemented by Island Health in collaboration with the Ministry of Child and Family Development. Over the five- year period from 2004 to 2008, 142 training sessions were held for all levels of the Triple P program and various intervention strategies such as Level 5 Pathways and Level 4 Group Triple P; 812 practitioners were accredited within six months after the training (Arim, Knott, Dahinten, Kohen, & Pace, 2013). Source of Data The study drew on three anonymized administrative data sources: (a) the British Columbia (BC) Ministry of Health (2012a, 2012b) data holdings, accessed through Population Data BC; (b) the publicly available 2006 Census of Population data from Statistics Canada (2006); and (c) program administration data from Island Health (http://www.viha.ca/). Population Data BC is a central data repository that facilitates interdisciplinary research on determinants of human development and health. Three separate administrative data holdings from the BC Ministry of Health were used for this study: (a) the Medical Services Plan (MSP) Payment Information File (British Columbia Ministry of Health, 2012a), which includes data from 1985 onwards regarding medical services provided by fee-for-service practitioners to all individuals covered by BC’s universal health insurance program; (b) the Consolidation File: MSP & Premium Billing (British Columbia Ministry of Health, 2012b), which includes population demographic data prepared for research use by Population Data BC; and (c) the Discharge Abstract Database: Hospital Separations (Canadian Institute for Health Information, 2012) that includes data from 1985 on, including all discharges, transfers, and deaths of in-patients and day surgery patients from acute care hospitals in BC. All data files accessed through Population Data http://www.viha.ca/ International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 63 BC are linkable at the individual level, but de-identified to maintain anonymity and confidentiality. For this study, data were extracted from 11 communities where Triple P had been implemented and from 11 matched communities elsewhere in BC. Individual anonymized data on BC residents in the 22 communities were used for the calendar years 2002 and 2010. These years provided a minimum of 2 years of pre- and post-implementation data for all communities, with up to 6-year post-implementation data for some communities. The population under study included mothers aged 15 to 65 years with children aged 0 to 18 years, as identified in the child’s first health care registration1, as well as children aged 6 to 12 years. Our focus was on school-age children (6–12 years). At school age, mental health conditions are more likely to be diagnosed than at younger ages (Centers for Disease Control and Prevention, 2013, Table 7). In order to match communities, publicly available data from the 2006 Census of Population2 were custom tabulated (i.e., aggregated to the Local Health Area3; LHA). Finally, Triple P program data were compiled and made available through partners at Island Health. Island Health provided community-specific data on program implementation and program characteristics. These data were sourced from computerized and paper information systems of Island Health and from local providers; each record included the date of implementation, the levels of programming implemented (e.g., Level 4 Group Triple P), and the intensity of program delivery (e.g., number of practitioners trained). The final population under study included mothers aged 15 to 65 in 2010 (N=242,766) and children aged 6 to 12 in 2010 (N=82,591) living in 22 LHAs, who were registered with MSP in both 2002 and 2010 and who were living in the same LHA in both years. Of children and mothers who were registered with MSP in 2002, approximately 80% were also registered with MSP in 2010 and about 70% were living in the same local health area. Data Access The encrypted data files were made accessible to the research team in an online secure research environment through Population Data BC; information on community characteristics was from publicly available 2006 Census of Population data; and program intensity information came from Island Health. Measures Individual characteristics: Mother’s age, and child’s gender and age, were used as covariates but also considered as possible moderators of the program’s effects because they are 1 Few children (0.2%) had more than one mother as identified in the first health care registration of the children. These mothers were not excluded from the analyses. 2 2006 Census of Population was the only available census data given the time frame of the study. 3 Local Health Areas are the smallest geographic areas defined by the Ministry of Health. In this study, LHAs are referred as communities. International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 64 associated with mental health and service use outcomes (Arim et al., 2015; Centers for Disease Control and Prevention, 2013). These variables were obtained from the Consolidation File (British Columbia Ministry of Health, 2012b). Community characteristics: Eleven communities that received Triple P between the years of 2004 and 2010 were designated as the “target” communities. Each target community had a “comparison” community that did not receive Triple P but that was matched on seven aggregated sociodemographic community characteristics at the LHA level: (a) population size, (b) proportion of young families (Census families with children aged less than 15), (c) proportion of individuals with an Aboriginal identity, (d) unemployment rate for population aged 15 or more years, (e) proportion of individuals aged 25–54 years who had not completed high school, (f) proportion of individuals with a mother tongue other than English or French, and (g) proportion of individuals who had not moved during the previous 5-year period. A list of matched target and comparison communities along with their community characteristics has been previously published elsewhere (Dahinten et al., 2014). Although there was a 100% agreement between the two researchers on the matched communities, relatively few or small differences were apparent. Because of the close rater agreement, and the fact that these sociodemographic community characteristics are known to be associated with maternal and child outcomes as well as parenting behaviors (Kohen, Leventhal, Dahinten, & McIntosh, 2008; Kohen, Brooks-Gunn, Leventhal, & Hertzman, 2002), the seven community characteristics were also used as covariates in the analyses. Aggregated baseline community prevalence rates in 2002 for each health outcome — mental health diagnoses and use of counseling services for mothers aged 15 to 65 years, and mental health diagnoses, conduct disorder diagnoses, use of counseling services, and injuries for children aged 6 to 12 years — were also included as covariates based on data from MSP and Hospital Separations for the calendar year of 2002. Health outcomes: Two binary health outcomes (mental health diagnoses and use of counseling services) for mothers aged 15 to 65 years and four binary health outcomes (mental health diagnoses, conduct disorder diagnoses, use of counseling services, and injuries) for children aged 6 to 12 years were considered based on data from MSP and Hospital Separations for the calendar year of 2010. These outcomes were selected in the context of their relevance to the aims of Triple P, which include preventing behavioral, emotional, and developmental problems in children and adolescents. Maternal and child mental health diagnoses were identified by the International Classification of Diseases, Ninth version, Clinical Modification (ICD-9-CM) diagnostic codes in the MSP data or the International Classification of Diseases, Tenth version, Canada (ICD-10-CA) diagnostic codes in the Hospital Separations data for depression and anxiety, including diagnoses for affective psychoses, neurotic depression, adjustment reaction, and depressive disorder. Specifically, four ICD-9-CM diagnostic codes (296, 300, 309, 311) in the MSP data and 23 ICD- 10-CA diagnostic codes (F31, F32, F33, F32.0, F34.1, F38.0, F38.1, F40, F41, F42, F41.2, F43.1, F43.2, F43.8, F44, F45.0, F45.1, F45.2, F48, F53.0, F68.0, F93.0, F99) in the Hospital International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 65 Separations data were used. Presence of any of these diagnostic codes in the calendar year of 2010 indicated maternal or child mental health diagnoses. These definitions have been previously used in other provincial administrative health data studies (e.g., Brownell et al., 2012; Martens et al., 2010). Maternal and child use of counseling services was based on MSP data that captures billed services covered by MSP but does not capture other “out of pocket” services such as counseling provided privately. Both service and specialty codes were used to capture services such as counseling and counseling psychotherapy. Specialty codes included services provided by psychiatrists, counselor/psychiatrists, and educators. This definition has been used in previous research using provincial administrative health data (e.g., Arim et al., 2015). Child conduct disorder diagnoses were identified using a similar strategy as that used for mental health diagnoses. The presence of a single ICD-9-CM diagnostic code for disturbance of conduct (312) in the MSP data or a single ICD-10-CA diagnostic code for conduct disorder, childhood onset type (F91) in the Hospital Separations data for the calendar year of 2010 indicated child conduct disorder diagnoses. This definition has been used by others (Brownell et al., 2015). Child injuries were identified by a single ICD-10-CA diagnostic code that identifies the presence of any injuries within all recorded diagnostic codes for each individual in the Hospital Separations data for the calendar year of 2010. Program characteristics: Typically, program intensity can be captured by the levels of Triple P, but since all five levels had been implemented on Vancouver Island, rather than using levels as an indicator of intensity, this study focused on “practitioner density” with the idea that having longer access and a higher number of trained practitioners may be associated with program benefits. Program intensity was derived from the program administration data from Island Health. It was calculated based on the number of years of training offered in a given community multiplied by the number of practitioners accredited in the community, adjusted for population size and multiplied by 1000 in order to have the same scale as the other variables in the analyses. As an example, a community with a population size of 14,000, which had 5 years of training and 90 practitioners accredited had a program intensity score of 32.14. Data Analysis First, descriptive statistics were examined separately for mothers and children. Due to the large sample size in this study, very small differences emerged as statistically significant. Given this, Cohen’s d effect sizes (Cohen, 1988; Rosenthal, 1996) were computed to examine the differences between target and comparison communities. Next, logistic hierarchical linear modeling (HLM) was used, with mothers or children nested within communities, to assess program effects, including the intensity of Triple P implementation, after controlling for the selected community-level baseline (i.e., pre-program implementation) health conditions and International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 66 sociodemographic characteristics. Thus, the HLM analyses were informed by a two-level model with both individual (e.g., age) and community level (e.g., unemployment rate) variables. Separate analyses were conducted for each of the six outcomes for mothers or children. The Laplace estimation method was selected given that the level of measurement of the outcome variables was binary (Raudenbush & Bryk, 2002). The focus of the data analysis was to examine between-community differences for mother and child mental health and service use outcomes that may be associated with the intensity of the implementation of Triple P, adjusted for individual characteristics, community characteristics, and program delivery. For this reason, although an unconditional two-level model (i.e., no predictor variables) was tested first, with subsequent models adding individual and community variables, only two models are presented in the results: Model 1, which examines the association between the program delivery and health outcomes, and which includes all the individual and community covariates and the target community variable; and Model 2, which adds the program intensity variable after controlling for the effects of individual and community covariates and the target community variable (other models are available upon request). The purpose of examining these models separately was to isolate the effects of (a) the program delivery and (b) the program intensity, as they were moderately associated (r = .73). For both models, all variables were added as fixed effects (i.e., their effects were assumed to be constant across communities). All continuous variables were centered on the mean so the intercept value could represent the “average” child (Raudenbush & Bryk, 2002). Interactions of individual characteristics (i.e., age and gender) with program delivery (i.e., target community) and intensity variables were also examined. Results Descriptive statistics (means and standard deviations or percentages) of all study variables by community type (i.e., target vs. comparison) are presented as are comparison results, including effect sizes for mother and child outcomes (see Table 1). Maternal descriptive statistics: The average age of mothers at about 41 years was similar in the target and comparison communities. Regarding community characteristics, overall population size was larger in the target communities compared with the comparison communities; there were fewer families with children younger than 15 in the target communities; more individuals were unemployed in the target communities; and fewer individuals had a mother tongue other than English or French. International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 67 Table 1 Descriptive Statistics of all Study Variables Mothers (N = 242,766) Children (N = 82,591) Communities Target (n = 117,745) Comparison (n = 125,021) Target (n = 39,498) Comparison (n = 43,093) Individual Characteristics Female, % 100.00 100.00 49.30 49.08 Age, M (SD) 40.99 (7.84) 40.99 (7.60) 9.08 (2.01) 9.07 (2.01) Community Characteristics Population size, M(SD) 23586.02 (15216.45)M 17279.00 (7052.60) 23566.43 (15259.97)M 17316.12 (7046.24) Families with children < 15 years, M(SD) 30.55 (3.02)L 34.37 (3.54) 30.59 (3.01)L 34.43 (3.53) Individuals with Aboriginal identity, M(SD) 6.38 (4.58) 6.04 (4.71) 6.41 (4.62) 6.05 (4.72) Unemployment rate, M(SD) 6.16 (1.68)S 5.82 (1.18) 6.17 (1.69)S 5.82 (1.19) Individuals without high school, M(SD) 11.97 (4.17) 11.33 (5.45) 11.99 (4.19) 11.32 (5.48) Mother tongue not English or French, M(SD) 10.78 (2.79)L 15.04 (6.01) 10.77 (2.81)L 15.09 (6.03) Not moved during 5-year period, M(SD) 53.85 (4.25) 54.26 (3.79) 53.85 (4.23) 54.26 (3.77) Mental health diagnoses in 2002, % 11.01 9.63 1.93 1.53 Use of counseling services in 2002, % 17.93 16.14 4.80 4.83 Conduct disorders diagnoses in 2002, % N/A N/A 1.92 1.91 Injuries in 2002, % N/A N/A 0.68 0.60 Program characteristics Program intensity, M(SD) 21.55 (14.55) N/A 21.58 (14.57) N/A Health outcomes Mental health diagnoses in 2010, % 14.92 14.31 2.02 2.03 Use of counseling services in 2010, % 18.93 17.07 3.55 3.27 Conduct disorders diagnoses in 2010, % N/A N/A 2.28S 1.62 Injuries in 2010, % N/A N/A 0.45 0.43 Note. Slight differences in the community sociodemographic characteristics are due to children sharing a mother in the data. Established criteria were used for small (about 0.2), medium (about 0.5), large (about 0.8), and very large (about 1.3) effect sizes for differences in means, and small (7 points), medium (18 points), large (30 points), and very large (45 points or greater) for differences in percentages with odds ratios being used for perce ntages outside the 15% to 85% range. N/A= Not available. Means are calculated per 100,000 in the community using individual scores aggregated up to the community level. SSmall effect size vs. “comparison” communities. MMedium effect size vs. “comparison” communities. LLarge effect size vs. “comparison” communities. XLVery large effect size vs. “comparison” communities. International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 68 In 2002, prior to Triple P, mothers’ mental health diagnoses (11.1% vs. 9.6%) and use of counseling services (17.9% vs. 16.1%) did not differ between the target and comparison communities given the effect sizes. In 2010, mothers’ mental health diagnoses (14.9% vs. 14.3%) and use of counseling services (18.9% vs. 17.1%) again did not differ between the target and comparison communities based on effect sizes. Child descriptive statistics: Approximately half of the children were female in both the target and the comparison communities. The average age of children was also similar at about 9 years. As expected based on the selection criteria, the pattern of results for community sociodemographic characteristics was similar between communities. Regarding baseline prevalence of health outcomes, there were no differences in children’s mental health diagnoses (1.9% vs. 1.5%), use of counseling services (4.8% vs. 4.8%), conduct disorder diagnoses (1.9% vs. 1.9%), or injuries (0.7% vs. 0.6%) between the target and the comparison communities in 2002 given the effect sizes. In 2010, there were no differences in the mental health diagnoses (2% vs. 2%), use of counseling (3.6% vs. 3.3%), or injuries (0.5% vs. 0.4%) between the target and the comparison communities based on effect sizes. In contrast, children were more likely to be diagnosed with conduct disorders (2.3% vs. 1.6%) in the target communities in 2010, with a small effect size. Maternal mental health outcomes: HLM analyses results for mothers’ health outcomes (see Table 2) indicated no differences in mental health diagnoses between mothers living in the target communities compared with those living in the comparison communities (Model 1). However, higher program intensity was associated with lower odds of being diagnosed with mental health conditions (Model 2). This pattern of results was similar for mothers’ use of counseling services, albeit not statistically significant. Overall, the model fit statistics suggested that the inclusion of program intensity improved the model for mental health diagnoses but not for use of counseling services, after accounting for the effects of the target community variable. Table 2 also identifies individual and community covariates that were found to be associated (albeit weakly) with maternal health outcomes. A series of interaction terms between maternal age and target community and program intensity variables were investigated but not found to be statistically significant for either of the health outcome variables and are thus not reported. Child health outcomes: HLM analyses results for children’s health outcomes (see Table 3) indicated that children living in target communities where Triple P was implemented were more likely to be diagnosed with conduct disorders and to use counseling services (Model 1). Living in the target communities (Model 1) was not associated with child mental health diagnoses or injuries. Program intensity was also not associated with child health outcomes. Overall, given the model statistics, results suggest that the inclusion of program intensity does not significantly improve the models, after taking the target community variable into account. As for child outcomes, none of the interaction terms that we investigated between individual level variables and target community or program intensity variables were statistically significant. International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 69 Table 2 Summary of Results from HLM Analyses for Mothers’ Mental Health Outcomes in 2010 Mental Health Diagnoses Use of Counseling Services Model 1 Model 2 Model 1 Model 2 Fixed Effects Individual Level Variables Age 0.99 (0.99-0.99)*** 0.99 (0.99-0.99)*** 0.99 (0.99- 0.99)*** 0.99 (0.99-0.99)*** Community Level Variables Prevalence in 2002 1.09 (1.06-1.12)*** 1.08 (1.04-1.11)*** 1.06 (1.01-1.10)* 1.05 (1.01-1.10)* Population size 1.01 (1.00-1.02)* 1.01 (1.00-1.01) 1.00 (0.99-1.02) 1.00 (0.99-1.01) Families with children aged < 15 years 1.00 (0.98-1.02) 1.00 (0.98-1.02) 1.01 (0.98-1.03) 1.01 (0.98-1.04) Individuals with an Aboriginal identity 0.97 (0.95-0.99)** 0.98 (0.95-1.00)* 0.98 (0.95-1.01) 0.99 (0.96-1.02) Unemployment rate 1.09 (1.03-1.15)** 1.09 (1.03-1.15)** 1.06 (0.96-1.17) 1.06 (0.96-1.16) Individuals without high school 1.01 (0.98-1.04) 1.01 (0.98-1.04) 1.00 (0.95-1.05) 0.99 (0.95-1.04) Mother tongue other than English or French 1.01 (0.99-1.02) 1.01 (0.99-1.02) 0.98 (0.95-1.01) 0.98 (0.95-1.01) Not moved during 5-year period 1.02 (1.00-1.04)* 1.02 (1.00-1.04) 1.03 (1.01-1.06)** 1.03 (1.01-1.06)* Target community 0.88 (0.78-1.01) 1.04 (0.87-1.24) 0.88 (0.70-1.12) 1.01 (0.74-1.37) Program intensity 0.95 (0.91-0.99)* 0.96 (0.90-1.02) Random Effects Intercept 0.01*** 0.01*** 0.03*** 0.03*** Model Fit Statistics -2 Log Likelihood 201161.80 201156.40 226606.90 226605.70 Akaike Information Criterion (AIC) 201185.80 201182.40 226630.90 226631.70 Bayesian Information Criterion (BIC) 201198.90 201196.60 226644.00 226645.90 Note. The values for the fixed effects are odds ratio estimates with 95% confidence limits in brackets. Effects of continuous variables are assessed as one unit offsets from the mean. * p < .05. ** p < .01. *** p < .001. International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 70 Table 3 Summary of Results from HLM Analyses for Children’s Health Outcomes in 2010 Mental Health Diagnoses Conduct Disorder Diagnoses Use of Counseling Services Injuries Model 1 Model 2 Model 1 Model 2 Model 1 Model 2 Model 1 Model 2 Fixed Effects Individual Level Variables Female 0.74 (0.67-0.81)*** 0.74 (0.67-0.81)*** 0.37 (0.33-0.42)*** 0.37 (0.33-0.42)*** 0.61 (0.57-0.66)*** 0.61 (0.57-0.66)*** 0.73 (0.60-0.90)** 0.73 (0.60-0.90)** Age 1.16 (1.14-1.19)*** 1.16 (1.14-1.19)*** 1.00 (0.98-1.03) 1.00 (0.98-1.03) 1.13 (1.10-1.15)*** 1.13 (1.10-1.15)*** 1.00 (0.95-1.05) 1.00 (0.95-1.05) Community Level Variables Prevalence in 2002 1.41 (1.14-1.73)** 1.30 (1.03-1.63)* 1.34 (1.18-1.51)*** 1.35 (1.18-1.54)*** 1.08 (1.00-1.15)* 1.07 (0.98-1.16) 1.52 (0.58-3.96) 1.67 (0.61-4.58) Population size 1.00 (0.99-1.01) 1.00 (0.99-1.01) 0.99 (0.97-1.00)* 0.98 (0.97-1.00)* 1.00 (0.99-1.02) 1.00 (0.99-1.02) 1.01 (0.99-1.03) 1.01 (0.99-1.03) Families with children aged < 15 years 1.04 (1.01-1.06)** 1.03 (1.01-1.06)* 1.10 (1.06-1.14)*** 1.10 (1.06-1.14)*** 1.01 (0.97-1.05) 1.01 (0.97-1.05) 1.02 (0.97-1.06) 1.02 (0.97-1.06) Individuals with an Aboriginal identity 0.97 (0.94-1.01) 0.98 (0.94-1.02) 1.01 (0.96-1.05) 1.01 (0.96-1.05) 0.97 (0.93-1.02) 0.97 (0.93-1.02) 0.97 (0.90-1.05) 0.98 (0.91-1.06) Unemployment rate 1.13 (1.05-1.21)** 1.14 (1.06-1.23)*** 1.46 (1.31-1.63)*** 1.46 (1.31-1.63)*** 1.12 (1.00-1.25)* 1.11 (0.98-1.25) 1.02 (0.89-1.18) 1.03 (0.89-1.19) Individuals without high school 0.95 (0.91-0.99)* 0.95 (0.91-0.99)* 0.90 (0.85-0.95)*** 0.90 (0.85-0.95)*** 0.98 (0.92-1.04) 0.98 (0.92-1.04) 1.02 (0.94-1.12) 1.02 (0.94-1.11) Mother tongue other than English or French 0.98 (0.95-1.00)* 0.98 (0.96-1.01) 1.00 (0.97-1.04) 1.00 (0.97-1.04) 1.02 (0.98-1.05) 1.02 (0.98-1.05) 0.98 (0.94-1.02) 0.98 (0.94-1.02) Not moved during 5- year period 1.01 (0.99-1.03) 1.01 (0.98-1.03) 0.96 (0.93-0.99)** 0.96 (0.93-0.99)** 1.03 (1.00-1.06)* 1.03 (1.00-1.06)* 1.05 (1.01-1.10)* 1.05 (1.00-1.10)* Target community 0.89 (0.76-1.05) 1.10 (0.83-1.45) 1.88 (1.46-2.43)*** 1.99 (1.34-2.95)*** 1.30 (1.03-1.64)* 1.24 (0.84-1.82) 0.92 (0.68-1.25) 1.01 (0.65-1.57) Program intensity 0.93 (0.85-1.01) 0.98 (0.90-1.08) 1.02 (0.92-1.12) 0.96 (0.84-1.10) Random Effects Intercept N/A N/A 0.02 0.02 0.03 0.03 N/A N/A Model Fit Statistics -2 Log Likelihood 16130.75 16127.70 15064.35 15064.21 24003.47 24003.37 4673.88 4673.53 Akaike Information Criterion (AIC) 16154.75 16153.70 15090.35 15092.21 24029.47 24031.37 4697.88 4699.53 Bayesian Information Criterion (BIC) 16266.61 16274.88 15104.54 15107.49 24043.66 24046.65 4809.74 4820.71 Note. The values for the fixed effects are odds ratio estimates with 95% confidence limits in brackets. Effects of continuous variables are assessed as one unit offsets from the mean. N/A= Not available. * p < .05. ** p < .01. *** p < .001. International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 71 Discussion The purpose of this study was to examine population level outcomes of a community- based parenting program for mothers and children using archived, population-based, administrative data sources. A unique strength of this study was the use of administrative data that were not collected for the research or program evaluation purposes of the present study and were therefore free from several sources of potential response bias. In addition, sophisticated analyses examining several mother and child outcomes and controlling for correlated factors were incorporated into the research design. Regarding associations with the Triple P program, there were no differences in maternal health outcomes between mothers living in the target communities compared with those living in the comparison communities. These results are contradictory to previous small sample studies (e.g., Matsumoto et al., 2007; Stallman & Ralph, 2007) and suggest that positive influences of Triple P reported at the individual level may not be maintained over an extended period of time at the population level. In contrast, children living in the target communities were more likely to be diagnosed with conduct disorders and more likely to use counseling services. Although these results do not support our expectations or the results of previous studies (e.g., Edwards et al., 2007; Hutchings et al., 2007), they may suggest other positive aspects of the program, such as an increased awareness of children’s behavior problems on the part of mothers. Indeed, the increased likelihood of using counseling services for children may be a result of an increased awareness of what services are available and needed, resulting from the identification of children’s behavior problems after participation in Triple P program in the target communities. Additional research is needed to further understand the associations between Triple P and service use outcomes in other studies before firm conclusions can be drawn. Taken together, our findings suggest that parent outcomes (fewer mental health diagnoses in treatment communities) may be more sensitive than child outcomes in measuring positive program effectiveness when population level administrative data are used (Bloomfield & Kendall, 2012), although the significant effects may not be maintained over time at the population level. It is also possible that benefits for child outcomes would be observed in a longer follow-up period since identification and service use would be the first steps to reduce and prevent behavioral and emotional problems. Regarding associations with program intensity, our findings partially supported our hypotheses. For children, program intensity was not associated with health outcomes. However, for mothers, it seemed that the longer the Triple P program was available in the community, the more practitioners who were trained and providing services, fewer mental health diagnoses were made. This finding highlights the importance of health care service as an important outcome measure, one which is often omitted (see Prinz et al., 2009). Additional research could contribute to our understanding of the associations between parenting programs and health care service use, an important public health outcome. International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 72 In 2010, about 15% of mothers in the target communities had mental health diagnoses and 19% used counseling services compared with 14% and 17%, respectively, of those in the comparison communities. These findings are consistent with Canadian statistics reporting that in any given year, one in five people in Canada report a mental health problem (Smetanin et al., 2011). For children, higher percentages of conduct disorders were observed in the target communities than in the comparison communities. While these findings warrant further research, it is possible that the mothers in the target communities were more knowledgeable and aware of children’s behavior problems due to the Triple P program, which has a component that aims to improve parents’ knowledge, skills, and confidence about parenting (Sanders, 2008; Sanders et al., 2002). This could have led to increased service use for children and to diagnoses of conduct disorders in children. As would be expected, there was a low prevalence of children’s diagnoses of mental health conditions and use of counseling services compared to mothers’. This difference may be due to the fact that MSP data capture the services provided by physicians such as psychiatrists, but not other services such as those provided by community and school-based professionals (e.g., specialized educators and counselors; Arim et al., 2015). In addition, services that are provided through alternative payment plans, such as salaried, sessional, and service agreement contracts, are not included in the MSP payment files. Future research may include other administrative data sources to supplement information on mental health problems, such as information on prescription drugs (e.g., PharmaNet) or reporting systems specific to mental health (e.g., the Ontario Mental Health Reporting System Metadata). Finally, associations with child injury are likely underestimated since we only included injuries that required overnight hospitalization, while other sources of data, such as emergency room data, were not included in the study data file. Several limitations of the study should be acknowledged. First, our study did not directly assess parenting behaviors; however, indicators of maladaptive parenting have been included (e.g., injuries) as in other studies (Prinz et al., 2009). Second, our study included 22 groups, whereas previous research on sufficient sample sizes for multilevel modeling showed that estimates of the standard errors of the second-level variances are too small when the number of groups is less than 100 (about 15% too small with 30 groups; Maas & Hox, 2005). Replication studies may be warranted as the number of target communities increases. In addition, the findings of this study are correlational; they do not claim cause and effect. Since analyses and results are reported at the community level (because Triple P was implemented, and the program data were compiled, at the community level by Island Health), our findings do not provide evidence about outcomes for individual participants of Triple P and we cannot assume complete participation of all community members. In a similar vein, although we attempted to statistically control many sociodemographic differences between the target and comparison communities, many differences could not be empirically controlled. However, members of our collaborative team living in BC confirmed the similarities between target and comparison communities prior International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 73 to final selection and prior to the analytic phase of the project. Third, although program intensity was captured, other program data were limited. Although data were available on practitioner’s training, other program features such as program attendance and “dose” of the intervention were not — for example, the information about how many parents were reached by each practitioner was only available for 2008 to 2010. Additional exploratory analyses suggested a moderate correlation (r = 0.51) between program intensity and practitioner’s reach score, suggesting that reach may be an important program feature to assess and include in future studies. In addition, some practitioners may have provided services in multiple communities or in communities other than those where they were trained; thus, a potential spillover effect may have occurred. This may also explain why intended effects were not observed. Finally, our findings cover a broader age group for mothers (15 to 65 years) but are limited to the school-age population of children (6 to 12 years)).Our results are also restricted by the outcomes available to us in the administrative data. Additional measures such as parenting behaviors, parenting competence, and parenting stress that would have also been important to include as links with our reported outcomes cannot be verified in the present study, but can only be assumed. Despite these limitations, there are several strengths of the study. First, the researchers were not involved in the planning, design, or implementation of the intervention or the collection of program or outcome data. Second, this study is the first to use HLM to examine the impact of Triple P on mothers’ and children’s health outcomes. The use of HLM allowed us to control for a number of community differences, including baseline scores of the outcomes, sociodemographic community characteristics, and Triple P programming. Finally, our multidisciplinary project group (including administrative data analysts, program coordinators, researchers, and policy stakeholders) made possible the sharing of program implementation information such as the intensity of program delivery. Indeed, program intensity was an important aspect to consider, demonstrating an association with lower maternal mental health diagnoses, and highlighting the importance of program evaluators and administrators collaborating to identify features that should be collected from the start of program implementation. This study aimed to demonstrate associations of a community-level parenting program with health outcomes for mothers and children through the use of provincial administrative data, and also examined program intensity as measured by practitioner density. Our findings suggest that the positive influences of Triple P on maternal health may not be maintained at the population level over an extended period of time. This finding highlights the importance of continued services and programs for mothers. However, program intensity plays an important role in contributing to positive maternal mental health, which suggests the importance of sustaining program intensity over time. In contrast, children were more likely to be diagnosed with conduct disorders and more likely to use counseling services in communities where Triple P had been implemented, and program intensity did not play an important role in child health outcomes. This finding raises several questions, such as: Is more time needed to observe benefits of Triple P for children (given that the program is designed to support mothers)?; and, Should International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 74 interventions directly targeting children be used in conjunction with parent support programs in order to have positive associations with child outcomes? Furthermore, there is a need to better understand the patterns of service use and whether they are indicators for positive or negative program effects. Our study also yields recommendations for program data that is important to collect during program implementation such as details about program reach and intensity (e.g., number of days or hours) as well as relevant follow-up information for participants. Future research should continue to explore the feasibility of a population-based approach and the use of secondary data along with program data to examine community-based intervention programs. Administrative data can fill some data gaps but cannot necessarily address all research questions. This study has demonstrated an example of an innovative use of administrative data for a program evaluation study that relies on both a close collaborative relationship with program administrators, and the support of other forms of data collection. Acknowledgments All data analysis was conducted at the Ottawa Hospital Research Institute, the former affiliation of the first author. All inferences, opinions, and conclusions drawn in this article are those of the authors, and do not reflect the opinions or policies of the Data Stewards at the BC Ministry of Health, or Statistics Canada. The authors wish to acknowledge Evelyne Bougie’s assistance with the compilation of Census data and to thank Esther Pace and Cindy Knott of Island Health for their support and assistance with the compilation of Triple P data. Compliance with Ethical Standards Funding: This work was supported by the Canadian Institutes of Health Research [funding reference number 109906]. Disclosure of potential conflicts of interest: The authors declare that they have no conflicts of interest. Ethical approval: This study was approved by the Research Liaison staff of Population Data BC, the Data Stewards at the BC Ministry of Health, the University of British Columbia Office of Research Ethics, the Ottawa Health Science Network Research Ethics Board, and the Island Health Research Ethics Board. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent: For this type of study, formal consent is not required. International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 75 References Arim, R. G., Knott, C., Dahinten, V. S., Kohen, D. E., & Pace, E. (2013). A summary report on the implementation of the Positive Parenting Program (Triple P) on Vancouver Island between the years of 2004-2010 (pp.17-18). Unpublished report. Arim, R. G., Kohen, D. E., Brehaut, J. C., Guevremont, A., Garner, R. E., Miller, A. R., … Rosenbaum, P. L. (2015). Developing a non-categorical measure of child health using administrative data. Health Rep, 26(2), 9–16. Arkan, B., Üstün, B., & Güvenir, T. (2013). An analysis of two evidence-based parent training programmes and determination of the characteristics for a new programme model. Journal of Psychiatric and Mental Health Nursing, 20, 176–185. doi:10.1111/j.1365- 2850.2012.01876.x Averdijk, M., Zirk-Sadowski, J., Ribeaud, D., & Eisner, M. (2016). Long-term effects of two childhood psychosocial interventions on adolescent delinquency, substance use, and antisocial behavior: A cluster randomized controlled trial. Journal of Experimental Criminology, 12(1), 21–47. doi:10.1007/s11292-015-9249-4 Bloomfield, L., & Kendall, S. (2012). Parenting self-efficacy, parenting stress and child behaviour before and after a parenting programme. Primary Health Care Research & Development, 13, 364–372. doi:10.1017/S1463423612000060 British Columbia Ministry of Health. (2012a). Medical Services Plan (MSP) payment information file P [Data extract]. Victoria, BC: Population Data BC. Retrieved from http://www.popdata.bc.ca/data British Columbia Ministry of Health. (2012b). Consolidation file: MSP & premium billing [Data extract]. Victoria, BC: Population Data BC. Retrieved from http://www.popdata.bc.ca/data Brownell, M., Chartier, M., Au, W., MacWilliam, L., Schultz, J., Guenette, W., & Valdivia, J. (2015). The educational outcomes of children in care in Manitoba. Winnipeg, MB: Manitoba Centre for Health Policy. Brownell, M., Chartier, M., Santos, R., Ekuma, O., Au, W., Sarkar, J., … Guenette, W. (2012). How are Manitoba’s Children Doing? Winnipeg, MB: Manitoba Centre for Health Policy. Calam, R., Sanders, M. R., Miller, C., Sadhnani, V., & Carmont, S. A. (2008). Can technology and the media help reduce dysfunctional parenting and increase engagement with preventative parenting interventions? Child Maltreatment, 13(4), 347–361. doi:10.1177/1077559508321272 http://dx.doi.org/10.1111/j.1365-2850.2012.01876.x http://dx.doi.org/10.1111/j.1365-2850.2012.01876.x http://dx.doi.org/10.1007/s11292-015-9249-4 http://dx.doi.org/10.1017/S1463423612000060 http://www.popdata.bc.ca/data http://www.popdata.bc.ca/data http://dx.doi.org/10.1177/1077559508321272 International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 76 Canadian Institute for Health Information. (2012). Discharge abstract database: Hospital separations [Data extract]. Victoria, BC: Population Data BC. Retrieved from http://www.popdata.bc.ca/data Centers for Disease Control and Prevention. (2013). Mental health surveillance among children: United States, 2005–2011 [Supplement]. Morbidity and Mortality Weekly Report, 62(2), 1– 35. Charles, J. M., Bywater, T. J., Edwards, R. T., Hutchings, J., & Zou, L. (2013). Parental depression and child conduct problems: Evaluation of parental service use and associated costs after attending the Incredible Years basic parenting programme. BMC Health Services Research, 13, 523. doi:10.1186/1472-6963-13-523 Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Earlbaum Associates. Dahinten, V. S., Arim, R. G., Guevremont, A., & Kohen, D. E. (2014). The case for using administrative data to examine a population-based parenting intervention. International Journal of Child Health and Human Development, 7(2), 115–124. de Graff, I., Speetjens, P., Smit, F., de Wolff, M., & Tavecchio, L. (2008). Effectiveness of the Triple P Positive Parenting Program on behavioral problems in children: A meta-analysis. Behavior Modification, 32(5), 714–735. doi:10.1177/0145445508317134 Edwards, R. T., Céilleachair, A., Bywater, T., Hughes, D. A., & Hutchings, J. (2007). Parenting programme for parents of children at risk of developing conduct disorder: Cost effectiveness analysis. BMJ, 334, 682. doi:10.1136/bmj.39126.699421.55 Fives, A., Pursell, L., Heary, C., Nic Gabhainn, S., & Canavan, J. (2014). Parenting support for every parent: A population-level evaluation of Triple P in Longford–Westmeath [Final report]. Athlone, Ireland: Longford Westmeath Parenting Partnership. Heinrichs, N., Kliem, S., & Hahlweg, K. (2014). Four-year follow-up of a randomized controlled trial of Triple P Group for parent and child outcomes. Prevention Science, 15(2), 233–245. doi:10.1007/s11121-012-0358-2 Hiscock, H., Bayer, J. K., Price, A., Ukoumunne, O. C., Rogers, S., & Wake, M. (2008). Universal parenting programme to prevent early childhood behavioural problems: Cluster randomised trial. BMJ, 336, 318–321. doi:10.1136/bmj.39451.609676.AE Hutchings, J., Gardner, F., Bywater, T., Daley, D., Whitaker, C., Jones, K., Eames, C., & Edwards, R. T. (2007). Parenting intervention in Sure Start services for children at risk of developing conduct disorder: pragmatic randomised controlled trial. BMJ, 334, 678. doi:10.1136/bmj.39126.620799.55 http://www.popdata.bc.ca/data http://dx.doi.org/10.1186/1472-6963-13-523 http://dx.doi.org/10.1177/0145445508317134 http://dx.doi.org/10.1136/bmj.39126.699421.55 http://dx.doi.org/10.1007/s11121-012-0358-2 http://dx.doi.org/10.1136/bmj.39451.609676.AE http://dx.doi.org/10.1136/bmj.39126.620799.55 International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 77 Kohen, D. E., Brooks-Gunn, J., Leventhal, T., & Hertzman, C. (2002). Neighborhood income and physical and social disorder in Canada: Associations with young children’s competencies. Child Development, 73(6), 1844–1860. doi:10.1111/1467-8624.t01-1-00510 Kohen, D. E., Leventhal, T., Dahinten, V. S., & McIntosh, C. N. (2008). Neighborhood disadvantage: Pathways of effects for young children. Child Development, 79(1), 156–169. doi:10.1111/j.1467-8624.2007.01117.x Maas, C. J., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1(3), 86–92. doi:10.1027/1614-2241.1.3.86 Martens, P. J., Bartlett, J., Burland, E., Prior, H., Burchill, C., Huq, S., Romphf, L., … Bailly, A. (2010). Profile of Metis health status and healthcare utilization in Manitoba. Winnipeg, MB: Manitoba Centre for Health Policy. Matsumoto, Y., Sofronoff, K., & Sanders, M. R. (2007). The efficacy and acceptability of the Triple P-Positive Parenting Program with Japanese parents. Behaviour Change, 24(4), 205– 218. doi:10.1375/bech.24.4.205 McConnell, D., Breitkeuz, R., & Savage, A. (2011). Independent evaluation of the Triple P Positive Parenting Program in family support services settings. Child & Family Social Work, 17(1), 43–54. doi:10.1111/j.1365-2206.2011.00771.x Nowak, C., & Heinrichs, N. (2008). A comprehensive meta-analysis of Triple P-Positive Parenting Program using hierarchical linear modeling: Effectiveness and moderating variables. Clinical Child and Family Psychology Review, 11, 114–144. doi:10.1007/s10567- 008-0033-0 Prinz, R. J., Sanders, M. R., Shapiro, C. J., Whitaker, D. J., & Lutzker, J. R. (2009). Population- based prevention of child maltreatment: The U.S. Triple P system population trial. Prevention Science, 10(1), 1–12. doi:10.1007/s11121-009-0123-3 Raudenbush, S., & Bryk, A. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage. Rosenthal, J. A. (1996). Qualitative descriptors of strength of association and effect size. Journal of Social Service Research, 21(4), 37–59. doi:10.1300/J079v21n04_02 Sampaio, F., Sarkadi, A., Salari, R., Zethraeus, N., & Feldman, I. (2015). Cost and effects of a universal parenting programme delivered to parents of preschoolers. European Journal of Public Health, 25(6), 1035–1042. doi:10.1093/eurpub/ckv106 http://dx.doi.org/10.1111/1467-8624.t01-1-00510 http://dx.doi.org/10.1111/j.1467-8624.2007.01117.x http://dx.doi.org/10.1027/1614-2241.1.3.86 http://dx.doi.org/10.1375/bech.24.4.205 http://dx.doi.org/10.1111/j.1365-2206.2011.00771.x http://dx.doi.org/10.1007/s10567-008-0033-0 http://dx.doi.org/10.1007/s10567-008-0033-0 http://dx.doi.org/10.1007/s11121-009-0123-3 http://dx.doi.org/10.1300/J079v21n04_02 http://dx.doi.org/10.1093/eurpub/ckv106 International Journal of Child, Youth and Family Studies (2017) 8(3–4): 59–78 78 Sanders, M. R. (1999). Triple P-Positive Parenting Program: Towards an empirically validated multilevel parenting and family support strategy for the prevention of behavior and emotional problems in children. Clinical Child and Family Psychology Review, 2(2), 71–90. doi:10.1023/A:1021843613840 Sanders, M. R. (2008). Triple P-Positive Parenting Program as a public health approach to strengthening parenting. Journal of Family Psychology, 22(4), 506–517. doi:10.1037/0893- 3200.22.3.506 Sanders, M. R., Cann, W., & Markie-Dadds, C. (2003). The Triple P-Positive Parenting Programme: A universal population-level approach to the prevention of child abuse. Child Abuse Review, 12(3), 155–171. doi:10.1002/car.798 Sanders, M. R., Kirby, J. N., Tellegen, C. L., & Day, J. J. (2014). The Triple P-Positive Parenting Program: A systematic review and meta-analysis of a multi-level system of parenting support. Clinical Psychology Review, 34(4), 337–357. doi: Sanders, M. R., Turner, K. M., & Markie-Dadds, C. (2002). The development and dissemination of the Triple P—Positive Parenting Program: A multilevel, evidence-based system of parenting and family support. Prevention Science, 3(3), 173-189. doi:10.1023/A:1019942516231 Schmidt, F. (2012). Effectiveness of Triple P Services at the Children’s Centre Thunder Bay: Final Report for Years 2007 to 2011. Thunder Bay, ON: Children’s Centre Thunder Bay. Smetanin, P., Stiff, D., Briante, C., Adair, C., Ahmad, S., & Khan, M. (2011). The life and economic impact of major mental illness in Canada: December 2011. North York, ON: RiskAnalytica, on behalf of the Mental Health Commission of Canada. Stallman, H. M., & Ralph, A. (2007). Reducing risk factors for adolescent behavioral and emotional problems: A pilot randomised controlled trial of a self-administered parenting intervention. Australian e-Journal for the Advancement of Mental Health, 6(2), 1–13. doi:10.5172/jamh.6.2.125 Statistics Canada. (2006). 2006 Census of Population. Ottawa, ON. Retrieved from http://www12.statcan.ca/census-recensement/2006/index-eng.cfm Thomas, R., & Zimmer-Gembeck, M. J. (2007). Behavioral outcomes of Parent-Child Interaction Therapy and Triple P—Positive Parenting Program: A review and meta-analysis. Journal of Abnormal Child Psychology, 35(3), 475–495. doi:10.1007/s10802-007-9104-9 Wilson, P., Rush, R., Hussey, S., Puckering, C., Sim, F., Allely, C. S., Doku, P., … Gillberg, C. (2012). How evidence-based is an ‘evidence-based parenting program’? A PRISMA systematic review and meta-analysis of Triple P. BMC Medicine, 10, 130. doi:10.1186/1741-7015-10-130 http://dx.doi.org/10.1023/A:1021843613840 http://dx.doi.org/10.1037/0893-3200.22.3.506 http://dx.doi.org/10.1037/0893-3200.22.3.506 http://dx.doi.org/10.1002/car.798 http://dx.doi.org/10.1023/A:1019942516231 http://www.tandfonline.com/doi/abs/10.5172/jamh.6.2.125 http://www12.statcan.ca/census-recensement/2006/index-eng.cfm http://dx.doi.org/10.1007/s10802-007-9104-9 http://dx.doi.org/10.1186/1741-7015-10-130