International Journal of Child, Youth and Family Studies (2018) 9(2): 28–37 DOI: 10.18357/ijcyfs92201818211 SUCCESS AND FAILURE IN RESIDENTIAL CARE: RESULTS AND EXPERIENCES FROM THE EVALUATION OF CHILD AND YOUTH CARE (EVAS) Michael Macsenaere and Corinna Pummer-Pilaj Abstract: EVAS is the largest study in Germany that evaluates the structures, processes, and results of residential care. The results show a success rate of over 60% and a cost-benefit ratio of 3:1. The study provides a number of indications as to what works; for example, the willingness to cooperate and the participation of the clients, a well-founded social-educational diagnosis, and a sufficient duration of help. Keywords: residential care, evaluation, effectiveness, efficiency, what works, cooperation, social-educational diagnosis, duration of help Michael Macsenaere PhD (the corresponding author) is the Managing Director of the IKJ Institut für Kinder- und Jugendhilfe gGmbH, Saarstraße 1, 55122 Mainz, Germany.Email: Macsenaere@ikj-mainz.de Corinna Pummer-Pilaj PhD is the Scientific Director of the IKJ Austria, Vienna. Email: Pummer-pilaj@ikj-austria.at mailto:Macsenaeresenaere@ikj-mainz.de mailto:Pummer-pilaj@ikj-austria.at International Journal of Child, Youth and Family Studies (2018) 9(2): 28–37 29 Approximately half a million young persons per year make use of educational assistance in Germany. They are entitled to such assistance whenever their current environment cannot guarantee an upbringing according to their developmental needs and only if the assistance service is both appropriate and necessary to fulfil these needs. “Educational assistance” as provided in Germany covers a broad range of pedagogical services and some therapeutical ones related to them, starting with child guidance counselling (with a comparatively low threshold for entry) to day-care and outpatient support, and also including fully inpatient services. In total, the range of services offered in accordance with German legislation covers eight different means of intervention. This paper focuses on an inpatient intervention: residential child care. German social law (§ 34 SGB VIII) describes this intervention as follows: All-day educational assistance offered by institutional providers (residential child care) supports children and adolescents in their development by complementing their natural everyday experiences with pedagogical and therapeutical services in order to help them overcome deficits in education and upbringing in their current environment. In accordance with age, state of development, and potential for improvement in the family of origin, its main purpose is to  make a return to the family of origin possible, or  prepare for education and upbringing in another family, or  offer a stable environment in which the young person can prepare for an eventually independent lifestyle. Young persons are to be advised and supported in matters of vocational training, employment, and all issues with regards to living a well-structured life. Pressure on providers of residential support for education and upbringing (see Macsenaere, Esser, Knab, & Hiller, 2014) to provide legitimation of its benefits is currently increasing due to the high costs of residential care. Residential child care seems to be particularly affected by this, as shown by various practices relating to cost. For example, the duration of the aid is limited to 18 months, residential child care is used only as a last resort, additional interventions are thinly spread, and the decision to obtain help for a child’s upbringing is often put off until it is too late. Given these pressures, it is especially important to examine the possibilities and limitations of this type of assistance. Some insights have been provided by qualitative studies (e.g., Finkel, 2004; Lambers, 1996; Rätz-Heinisch, 2005); however, in order to be able to make generalizable statements it would be helpful to supplement their findings with up-to-date nationwide results from quantitative research. To this end, the results of a systematic evaluation of youth welfare interventions (the “Evaluation of Educational Aids” [EVAS]; Macsenaere & Knab, 2004) are presented, and the practical application of EVAS is described by presenting an example of a residential institution. International Journal of Child, Youth and Family Studies (2018) 9(2): 28–37 30 The Database: EVAS In 1995, an interdisciplinary working group was set up with the aim of creating a set of tools to enable child and youth welfare institutions as well as youth protective services to promote quality development based on a uniform and systematic evaluation. After analysing the results of studies on youth welfare with a focus on the quality of the results (including the Jugendhilfe- Effekte-Studie [Youth Welfare Effects Study] (JES; Macsenaere, 2007; Schmidt et al., 2003), a set of tools was developed that contained the most suitable items from both the content and test- theoretical points of view. After several pre-tests, the tools were condensed into a four-page instrument that could be used by practitioners with reasonable effort. EVAS started in Bavaria in 1999 and subsequently spread to the entire Federal Republic of Germany (Macsenaere & Knab, 2004). EVAS has now developed into the largest evaluation of youth welfare interventions in the German-speaking region: Distributed over all 16 federal states, data on more than 50,000 youth welfare interventions are currently available, coming from around 250 institutions in Germany, Austria, and Luxembourg. The following statements refer to 25,757 cases of residential child education and upbringing according to § 34 SGB VIII, of which 18,136 have been completed. Who Utilises Residential Education? The average age of the residents at the beginning of an intervention is roughly 13 years, which is considerably higher than that for other forms of care intervention. The 14- to 17-year- olds constitute the largest group. This group has grown gradually over the past years and currently accounts for more than 50% of all youth in residential care. This may indicate a trend towards granting residential care at later ages. Roughly 60% of the residents are male and 40% female, which mirrors the figures for residential care in Federal Germany as a whole. Only 29% of the residents’ parents have joint custody. In almost half of the cases one parent alone has custody (mothers 39%, fathers 5%), and in at least 15% the child has a guardian. In 95% of cases residential care follows on from some prior form of youth assistance. This finding is alarming, since the probability of failure of a youth welfare intervention increases with the number and intensity of previous interventions (Macsenaere & Herrmann, 2004). It is similarly worrying that since 2001 the trend has been to provide residential education after supportive interventions that are unusually numerous or intense. The main form of previous placement is residential education (33%), followed by inpatient psychiatry (26%), social pedagogical family help (25%), emergency removal (19%), child guidance counselling (11%), and day groups (15%) (Macsenaere & Herrmann, 2004). The main reasons for admission relating to children’s immediate environments are domestic conflicts (68%, with an upward trend in recent years) and parental failure in education or upbringing (50%). The main reasons for admission that pertain directly to the child are performance problems (44%), dissociative disorders (35%), and developmental deficits (26%). International Journal of Child, Youth and Family Studies (2018) 9(2): 28–37 31 Compared to the clientele in outpatient care, those using residential services generally have less robust resources, especially with regard to the resources of self-concept, self-assurance, coping strategies, family function, and social-communicative competences. Resources insufficiently expressed are characterised by a strongly externalizing disturbance pattern with aggressive (41%), dissocial (e.g., lying, truancy; 37%), and delinquent behaviour (19%). Other important problems are social insecurity (44%), attention deficits and agitation (36%), and deficient or undifferentiated attachment behaviour (31%). It is noticeable that the externalising symptoms are more likely to be observable in boys while internalising abnormalities are more frequently expressed by girls, which serves as a possible explanation for the greater average age of the female clientele at the onset of support interventions (Macsenaere & Herrmann, 2004). The residents tend to have entered residential care as teenagers rather than as small children. Often, they have had many prior admissions (or in some cases particularly intensive ones), correlating with a combination of low resources and pronounced externalising problems. Considering the help factors for youth welfare, this constellation of features is an unfavourable starting situation, one that considerably reduces the probability of success and increases the risk of failure. What are the Results? Any intervention — including residential care — may produce not only its intended effects, but usually also has side effects that can have a significant impact on the outcome. In order to determine the results of an intervention, therefore, both desired effects and side effects must be recorded. Sociopedagogical and social work diagnostics furnish instruments that allow the providers of residential support to record the resources and deficits of young people and their families at multiple points in time from the beginning of the intervention to its end (Hermsen & Macsenaere, 2007). Impact measurement conducted with these tools covers more than just the extent of goal achievement, thus avoiding two major weaknesses: the degree of attainment is often too low and thus not reliable, and attainment levels only provide indications for intended effects and cannot capture side effects (Macsenaere, 2007). In view of the great importance of planned interventions, it is recommended to additionally take the attainment levels into account in any evaluation; however, they should be given less weight with regard to determining the effectiveness of an aid. On this basis, a so-called effect index is calculated by EVAS, which provides a first overview of the result of the examined intervention. This index takes into account all relevant developments during assistance as well as the attainment levels. Despite the extremely unfavourable initial situation, approximately 60% of the evaluated interventions have a positive effect index and thus indicate positive change. Significantly large effect sizes (Cohen’s d) were achieved in more than half of these cases. Every educator working in residential education knows that success cannot be attained in every instance. This is confirmed by the results obtained from EVAS, which indicate negative change in approximately 35% of all sample cases (see Figure 1). International Journal of Child, Youth and Family Studies (2018) 9(2): 28–37 32 Figure 1. Effect strengths of residential care. The statistical results are reflected in the personal statements of the clients. From the EVAS follow-up questionnaires, three examples of the success of the intervention are given here: I feel better because I and my mother now get along with each other again and I don’t have that much trouble with her any more. (Young person) Thank you for helping me. On the whole I felt very well and you did everything for me: Thank you sooooooo much. (Young man) It is now easier to deal with him. He no longer reacts aggressively right away. (Mother) A number of central factors, as shown by Macsenaere and Esser (2015), can be empirically determined to factor into the success or failure of residential education. With regard to the clients, a high willingness to cooperate, a low age at the beginning of the intervention, and as few prior experiences with youth welfare as possible facilitate an intervention’s success. It is also important to conduct a sociopedagogical survey (Bayerisches Landesjugendamt, 2001; Macsenaere, Paries, & Arnold, 2009) taking into account the resources of the young person and their family and involving them in the planning process for the intervention. For institutions — but also equally for youth protective services — the most essential step towards success is helping clients make use of How strong are the effects of educational aids? 0 5 10 15 20 25 30 35 negative large negative medium negative small no change positive small positive medium positive large 16,3 7,7 10,7 4,0 14,8 13,4 33,1 S tr e n g th o f e ff e c ti v e n e s s in % positive neutral negative International Journal of Child, Youth and Family Studies (2018) 9(2): 28–37 33 any self-help resources they have (see also Peters, 2006). Failure to consider this markedly decreases the probability of the intervention succeeding as a whole. In order to promote the necessary cooperation between clients and service providers, the educational approach must be built around the resources of the participants, while avoiding a deficiency-based understanding of residential education as a “repair workshop”. The duration of the intervention also directly correlates with the effects achieved: supportive interventions with a duration of less than 18 months, on average, do not result in significant positive change. On the other hand, interventions that last for a longer time will, on average, have an appreciable effect. The client-specific and procedural effects mentioned here must be supplemented by the structural ones, as elaborated by the Youth Aid Effects Study (Macsenaere, 2003) and INTEGRA (Internationale Gesellschaft für erzieherische Hilfen, 2003). Since the effectiveness of the intervention depends to a great extent on its duration, with the majority of the effects only achieved after the first year, a general capping of the duration of support after, for example, 18 or 24 months, is not useful — not even from an economic point of view. Not only would the full potential of residential education be left untapped and the clients not helped according to their full needs, this “cost-saving model” would have a low effect strength and would not change the need for support, leading to future expenditures for subsequent high-level interventions. Cost-Benefit Relation of Residential Education Current studies in efficiency research address the question of what economic benefits can reasonably be expected given the considerable costs of residential care. Investigations by Roos (2005) and Hermsen, Roos, and Zinkl (2007) predict strong long-term benefits for the categories of education, employment, health, and delinquency based on data from JuLe-Study (Bundesministerium für Familie, Senioren, Frauen und Jugend, 1998), JES, and EVAS. Using these estimates, one euro of expenses would result in approximately three euros of economic benefit in the long run. At that rate of return, residential education would be economically worthwhile with loan financing at an interest rate of between 3.7% and 10.5%. Interestingly, the economic benefits are particularly pronounced in the case of girls. Accordingly, residential education is not only effective overall, but is also economically sensible and in the long term worthwhile for the taxpayer, justifying the classification of the associated expenses as an investment rather than mere costs. How is EVAS Used in Institutions? The above-mentioned results for residential education are made possible by the inter- institutional aggregation of individual cases. In addition, the participating institutions have an interest in using the evaluation procedure to inform the concrete pedagogical work within the International Journal of Child, Youth and Family Studies (2018) 9(2): 28–37 34 institution. An institutional manager of the Schottener Sozialdienste [Schotten social services] gives an example from the point of view of an institution providing residential services: EVAS can be applied in the daily pedagogical work. The online version allows the immediate evaluation of all data entered, both for the entire institution as well as for an individual case. This means that up-to-date results can be obtained at any time, which can be used as a basis for discussion concerning intervention planning or central control processes. The educators have a corresponding and timely access to the most important information and can select them specifically from the entire package. Here are some examples:  EVAS individual evaluation can be used directly in official meetings. This makes a higher degree of objectivity and a better professional positioning possible. The regular use of the EVAS results helps in the structuring of realistic educational interventions and medium- or long-term strategies (what should we react to, what measures do we derive from it?). EVAS contributes to greater transparency of pedagogical actions and to the improvement of the inter-institutional practice.  Use for educational (assistance) planning: The EVAS evaluation leads to the replacement of the “soft” documentation practice with fact-based evaluation that provides comprehensible results based on empirically collected data. Concrete comparisons, a competent argumentation in everyday life as well as in situations involving immediate crises and a more efficient planning and use of the facility resources are thereby possible.  Preparation and implementation of the intervention planning discussions: EVAS provides an institution with arguments based on empirical data when it comes to the consideration of the development of the clients versus the economic needs of the employees of the institution, the provider, and the cost bearer. The development of the child, the child- and family-related goal-fulfilment as well as prognosis relating to these are systematically presented. An exact development curve can be calculated and the specific course of action can be documented accordingly. In practice, we have found that the use of EVAS data is particularly relevant in more complex cases. Focusing on the deficits is avoided since the EVAS items allow an analysis of the resources, e.g. in terms of social integration, social-communicative competences, special skills and achievements, interests, self-reliance, function in the group and the family, and physical health. Based on this information, a more precise formulation of the objectives is possible in the working group tasked with planning interventions. International Journal of Child, Youth and Family Studies (2018) 9(2): 28–37 35 The manager continued: From our point of view, EVAS has developed into a user-friendly evaluation system in the last few years: it can be used by every educator after minimal training, the well-designed questionnaires can be completed quickly, and evaluations are available immediately after data entry. It is the basis for control processes and quality development at several levels. Outlook The present EVAS results on residential education are positive despite the difficult initial situations of the clients. On the other hand, the results reveal potential for optimisation that could lead to a gradual improvement of the intervention:  It is important to avoid a “stringing together” of youth welfare interventions. The goal must be to choose the appropriate intervention even if it is associated with higher costs. Primary EVAS results indicate that in many cases an unsuitable type of intervention is assigned.  The intervention must focus on the resources of the young people and families involved. To this end, it is necessary for institutions as well as for young people as coproducers of interventions to sharpen their view of these resources. This may be achieved, for example, by using a sociopedagogical survey giving insight into these resources.  The change in the client’s development is to be reviewed and evaluated at least every six months during assistance planning. The results of the intervention can be predicted after just 6 to 12 months. This knowledge should be integrated into the support planning process and used for optimisation.  The averages across positive outcomes show broad variance from successful to failed interventions. This diversity represents an opportunity for future improvement that should not be underestimated. It becomes all the more important when we see youth welfare in an international context. It would therefore be very interesting to compare the available data with international partners based on a common evaluation procedure, and to use the results for quality development. International Journal of Child, Youth and Family Studies (2018) 9(2): 28–37 36 References Bayerisches Landesjugendamt [Bavarian State Youth Office] (Ed.). (2001). Sozialpädagogische Diagnose – Arbeitshilfe zur Feststellung des erzieherischen Bedarfs [Sociopedagogical diagnostics – Guideline for determining educational needs]. München, Germany: Author. Bundesministerium für Familie, Senioren, Frauen und Jugend (Ed.). (1998). Leistungen und Grenzen der Heimerziehung. Ergebnisse einer Evaluationsstudie stationärer und teilstationärer Erziehungshilfen [Benefits and limitations of home education: Results of an evaluation study of home schooling and daycare facilities] (Vol. 170). Stuttgart, Germany: Kohlhammer. Finkel, M. (2004). Selbständigkeit und etwas Glück: Einflüsse öffentlicher Erziehung auf die biographischen Perspektiven junger Frauen [Independence and a little luck: Influences of public education on the biographical perspectives of young women]. Weinheim, Germany: Juventa. Hermsen, T., & Macsenaere, M. (Eds.). (2007). Wirkungsforschung in der Kinder- und Jugendhilfe [Impact research in child and youth welfare] (Schriftenreihe der KFH Mainz; Vol. 2). Mainz, Germany: KFH Mainz. Hermsen, T., Roos, K., & Zinkl, K. (2007). Effizienz-Benchmarking in der Kinder- und Jugendhilfe [Efficiency benchmarking in child and youth welfare]. In T. Hermsen & M. Macsenaere (Eds.), Wirkungsforschung in der Kinder- und Jugendhilfe (pp. 243–260). Mainz, Germany: KFH Mainz. Internationale Gesellschaft für erzieherische Hilfen (Eds.). (2003). Abschlussbericht zum Modellprojekt „INTEGRA – Implementierung und Qualifizierung integrierter, regionalisierter Angebotsstrukturen in der Jugendhilfe am Beispiel von fünf Regionen“ [Final report on the model project "INTEGRA - Implementation and qualification of integrated, regionalized supply structures in youth welfare using the example of five regions"]. Frankfurt, Germany: Author. Lambers, H. (1996). Heimerziehung als kritisches Lebensereignis [Home education as a critical life event]. Münster, Germany: Votum. Macsenaere, M. (2007). Verfahren zur Wirkungsmessung in den erzieherischen Hilfen: Jugendhilfe-Effekte-Studie [Method for measuring the impact of educational aids: Youth welfare effects study]. In Institut für soziale Arbeit (Ed.), Wirkungsorientierte Jugendhilfe [Activity-oriented youth welfare] (Vol. 1, pp. 25–31). Münster, Germany: Author. Macsenaere, M., & Esser, K. (2015). Was wirkt in der Erziehungshilfe? [What works in educational aids?]. München, Germany: Reinhardt. International Journal of Child, Youth and Family Studies (2018) 9(2): 28–37 37 Macsenaere, M., Esser, K., Knab, E., & Hiller, S. (Eds.). (2014). Handbuch der Hilfen zur Erziehung [Handbook of educational aids]. Freiburg, Germany: Lambertus. Macsenaere, M., & Herrmann, T. (2004). Klientel, Ausgangslage und Wirkungen in den Hilfen zur Erziehung [Clientele, starting position and effects in aids for education]. unsere jugend, 56(1), 32–42. Macsenaere, M., Klein, W., & Scheiwe, N. (2003). Jugendhilfe-Effekte-Studie: Was leistet Jugendhilfe? [Youth welfare effects study: What is youth welfare?]. unsere jugend, 55(2), 484–491. Macsenaere, M., & Knab, E. (2004). Evaluationsstudie erzieherischer Hilfen (EVAS): Eine Einführung [Evaluation of educational aids (EVAS): An introduction]. Freiburg, Germany: Lambertus. Macsenaere, M., Paries, G., & Arnold, J. (2009). EST! Evaluation der Sozialpädagogischen Diagnose-Tabellen – Abschlussbericht [EST! Evaluation of social pedagogical diagnosis tables - Final report]. München, Germany: Bayerisches Staatsministerium für Arbeit und Sozialordnung, Familie und Frauen. Peters, F. (2006). Zum Stichwort: Wirkungsorientierung/wirkungsorientierte Steuerung. Forum Erziehungshilfen, 12(5), 260–261. Rätz-Heinisch, R. (2005). Gelingende Jugendhilfe bei „aussichtslosen Fällen“!: Biographische Rekonstruktionen von Lebensgeschichten junger Menschen [Successful youth welfare in "hopeless cases"!: Biographical reconstructions of life stories of young people]. Würzburg, Germany: Ergon. Roos, K. (2005). Kosten-Nutzen-Analyse von Jugendhilfemaßnahmen [Cost-benefit analysis of youth welfare measures]. In F. Petermann (Ed.), Studien zur Jugend- und Familienforschung [Studies on youth and family research] (Vol. 23). Frankfurt, Germany: Lang. Schmidt, M., Schneider, K., Hohm, E., Pickartz, A., Macsenaere, M., Petermann, F., … Knab, E. (2003). Effekte erzieherischer Hilfen und ihre Hintergründe [Effects of educational aids and their backgrounds] (Schriftenreihe des BMFSFJ; Vol. 219). Stuttgart, Germany: Kohlhammer.