DOI: 10.28934/ea.22.55.1.pp76-89 ORIGINAL SCIENTIFIC PAPER An Evaluation of the Short-term Skills Trainings Targeting the Roma Ethnic Minority Lara Lebedinski110 F* 1 Institute of Economic Sciences, Belgrade, Serbia ABSTRACT This paper studies the impact of short-term skills trainings targeting youth in the Roma ethnic minority. In order to better understand the impact of the program, we contrast outcomes of Roma and non-Roma. Participants of these short-term skills trainings were predominantly males in their twenties. Six months after the training end the impact estimates of skills trainings for Roma are 15.55 percentage points while for non-Roma we find an impact of 28.60 percentage points. In international comparisons these impacts are very large suggesting that the program has an exceptionally high impact on participants. It is difficult to pinpoint the reasons why Roma and non-Roma have different impact estimates, but potential explanations are differences in background characteristics (education and working experience) and differences in the content of the trainings that these two groups attended. Key words: Active labour market policies, short-term skills trainings, Roma ethnic minority JEL Classification: C22, J0, J68 INTRODUCTION The goal of this paper is to examine how Roma participants of short-term skills trainings performed on the labor market after attendance of skills trainings implemented by the German Development Cooperation and German Agency for International Cooperation (GIZ). In the recent years, the German Development Cooperation and GIZ as implementing agency have supported several projects aiming to improve the position on the labor market of the most vulnerable groups in Serbia through the provision of short-term skills trainings. The trainings lasted from a few days up to 8 months. This paper includes data on Roma participants from two different programs.11F1 The Youth Employment Program (YEP) project was designed with the goal to support young people in Serbia in improving their labour market situation. While the project included a number of different employment initiatives targeting disadvantaged groups, in this paper we examine only participants of short-term skills trainings. In particular, we examine only vocational trainings institution (VTI) based trainings– i.e. skills trainings for labor market.12F2 VTI based trainings are * E-mail: lara.lebedinski@ien.bg.ac.rs 1 "Youth Employment Promotion" (YEP) and "Inclusion of Roma and other marginalised groups in Serbia" (InR). 2 The YEP program included also employer based-trainings – i.e. skills trainings for known employers. Employer based-trainings were mostly public-private partnerships (PPP) between private sector companies and GIZ. The PPP agreements specified that employers should employ at least 70% of the Lara Lebedinski 77 trainings taking place exclusively at a training institution and are not firm based. It should be noted that some of the YEP skills-trainings targeting exclusively Roma were shaped according to the needs and interests of this specific group.13F3 Another group of participants included in this paper are non-Roma participating in one of the two projects, we include them to be able to benchmark the results of the Roma population.14F4 The goal of the InR was to support the government of Serbia in implementing the Strategy for the Social Inclusion of Roma, and in this paper we focus on one element of this support, i.e. skills trainings. Young people in Serbia experience difficulties when entering the labour market (Marjanović, 2016). Being both Roma and young poses additional challenges. Roma are considered to be a vulnerable group both in Serbia and in the region. On average, they have a very low educational attainment and are overrepresented in the informal labour market where jobs are insecure and wages are low. It is very difficult to break out of this vicious cycle of poverty, but one way to support them is to include them in skills training which can potentially help them find better and more secure jobs. This paper studies the impact of short-term skills trainings on labor market trajectories of Roma and non-Roma participants. Using propensity score matching, we find that both Roma and non-Roma participants who attended short-term skills trainings have a higher likelihood to be employed six months after completing the training. The estimated impact of the program is large in international comparison. The paper proceeds as follows. In the second section we provide a literature review, in the third section of the paper we discuss the descriptive statistics with the aim to describe the participants of the skills trainings and their labor market outcomes before and after the skills trainings. The main part of the paper is in the fourth section, the impact analysis which relies on matching methodology and selects a control group of individuals who are similar to the participants of short-term skills trainings. We contrast the findings of non-Roma and Roma participants. Section four provides impact estimates on the employment of training participants. Section five summarizes the findings and concludes. LITERATURE REVIEW Young people have a lower employment rate than the general population (O'Higgins, 1996). There are two main reasons for this stylized fact: first, young people face barriers when entering the labour market and second, they face difficulties to stay attached to the labour market, i.e. keeping a stable and well-paid job (Eichhorst and Rinne, 2018). With no or little work experience, young people face difficulties finding their first job and getting a foothold in the labour market. A well documented feature of labour markets in transition and in particular Serbia is that the education systems do not provide much practical experience and are not well aligned with the labour market needs (Aleksić et al., 2021; Rokicka, 2019). As a result, firms need to make significant additional investments in human capital to help young workers become independent in their work and for this reason companies try to avoid employing labour market entrants. Additionally, a number of youth graduate in fields and professions which are not demanded on the labour market. In times of economic downturn and lower demand, young people are the first ones to get fired due to tenure based mandatory severance pay (Verick, 2019) and this makes them vulnerable in uncertain economic periods. We turn now to the characteristics and specificities of the ethnic group that is in the focus of this study. Roma experience a disadvantage on the labour market in terms of lower employment trainees under any form of formal contract. We include only VTI based trainings, because no Roma were reported to be participants of employer-based trainings. 3 At some point, YEP had a task to base its programs on the needs of the Roma people in order to motivate more Roma to participate in YEP measures. 4 InR project 78 Economic Analysis (2022, Vol. 55, No. 1, 76-89) and lower wages and this is particularly reflected in the outcomes of women (O'Higgins, 2012). Roma are overrepresented in the informal labour market where wages and job security are lower than in the formal labour market (Lebedinski, 2020). In order to help the young generation to improve their life conditions, it would be important to invest in the education of Roma. However, the educational attainment of Roma children is far below the attainment of non-Roma. In Serbia, the school attendance among Roma in the age group 7-9 years is 86% and it falls to 66% in the age group 13-15 years (Brüggemann, 2012). In order to improve the attendance of Roma children in compulsory education, a number of countries have introduced Roma teaching assistants into schools. These people help and support Roma children at school. Serbia also has assistants in schools since 2008, but their effectiveness is limited as they are expected to work with a large number of Roma pupils (Battaglia & Lebedinski, 2015; 2017). Roma are disadvantage not only on the labour market and at school, but they also experience worse health outcomes than the majority of the population (Doyle, 2004). In addition to this, informal employment implies that in some countries they do not have access to health services (Mihailov, 2012). Active labour market policies can be grouped into four different categories: (i) classroom or on the job trainings, (ii) job search assistance, (iii) subsidized private sector employment and (iv) subsidized public sector employment. In general, ALMPs have small effects (Card et al., 2010; 2018). Skills trainings fall into the first category, i.e. classroom or on the job trainings, and it has been shown that they have smaller short-term effects, but the impacts increase in the medium to long-term. However, skills programs targeting youth have been less successful than skills programs with other groups. A meta-study looking only at youth programs confirms the finding that the result of youth employment programs are unsatisfactory (Kluve et al., 2019). With regards to job search assistance programs, these are considered low-cost interventions and previous meta studies have shown that they have a small, but positive impact on the employment rate (Card et al., 2010; 2018). Subsidized private sector employment are less effective in the short- term, while some positive impacts have been found in the medium- to long-term. Past meta- analysis (Card et al., 2010; 2018) have shown that both short- and long-term effects of interventions should be evaluated. Some intervention impacts can be only short-lived or other positive impacts (or even negative) can be revealed only after some time. Moreover, different types of ALMPs can be more beneficial for some groups than for others and heterogenous impacts should be considered when deciding on which ALMP type to implement to target a specific group. This is not the first study to examine the causal impact of active labour market policies in Serbia. An extensive past study by Bachmann et al. (2019) looked at the impact of skills trainings in Serbia implemented as part of the same YEP program funded by the GIZ. Bachman et al.'s (2019) study examined the impact of skills trainings on the overall youth population, while this paper focuses on the Roma youth population. The previous study found that 8 months after the training end the participants of vocational trainings institution based trainings in the program had a 16 percentage points higher likelihood to be employed than the matched control group. Further results indicate that the medium-term effect size is large than the short-term impact. SAMPLE OVERVIEW AND DESCRIPTIVE STATISTICS Sample overview The initial sample had 594 individuals from two different programs (YEP and InR), out of which 519 participated in short-term skills trainings of the YEP program and another 75 in the short- term skills trainings in the InR program. The YEP program target young people and therefore the participants of this program can be both non-Roma and Roma. The analysis includes both non- Roma and Roma for comparison purposes. Lara Lebedinski 79 For this paper, we retrieved data from the National Employment Service (NES)15 F5 on the periods of registered employment and registered unemployment. We do not have administrative NES data for all short-term skills participants for three main reasons. First, in order to get access to the NES administrative data, participants had to give their unique identifier (JMBG). Some participants did not give their JMBG when registering for the training and we could not retrieve their data from NES. Second, we can only access data for participants who ever registered with NES and some individuals never did so and are therefore not present in the NES administrative dataset. Third, we had to exclude some participants from the analysis because their administrative data was inconsistent. For instance, for few participants we had overlapping employment and unemployment spells and such individuals were excluded. Starting from a sample of 594 participants of YEP and InR, among the 347 non-Roma, 306 were identified in the NES administrative dataset and from both programs there are 203 Roma in the NES dataset. In total, our NES dataset contains 509 participants (306 non-Roma and 203 Roma) i.e. beneficiaries of YEP and InR that could be identified in the NES data. Table 1. Sample size NES data and survey data Total NES data YEP Non-Roma 347 306 Roma 172 169 InR (only Roma) 75 34 Total 594 509 Note: Only vocational trainings institution based trainings. Source: Author's calculations based on NES and survey data Descriptive statistics We start by discussing descriptive statistics of the socio-economic characteristics and labour market status before and after the participation in the training. Table 2 shows the socio- demographic characteristics according to the administrative data for the whole sample (column (1)), separately for YEP non-Roma (column (2)), YEP Roma (column (3)) and participants of InR (column (4)). In columns (5) and (6), we compare non-Roma and Roma participants of YEP (column (5)) and Roma from YEP and from InR project. The reason to compare non-Roma and Roma is to understand whether the Roma are more disadvantaged than the non-Roma among YEP participants. Second, we compare Roma from the YEP and the InR project (column (6)) with the aim to understand whether these two groups of participants differ. The available time span from the administrative data does not allow to examine employment six months after for participants of the InR project and this outcome is not reported for them. First, we summarise the socio-demographic characteristics: • Participants of skills-trainings are predominantly males in their twenties who finished secondary education. Among Roma, there are more females than among non-Roma, most likely due to the content of the training that they attended (e.g. there were a number of trainings attracting mainly females, such as trainings for manicure, pedicure, hairdressers, etc.) • Roma participants of skills trainings are less educated than non-Roma participants, most Roma finished only primary school. InR participants are less educated than YEP Roma. Almost three fourths of InR participants have finished at most primary school. 5 National employment service is the public agency with the mandate to provide job-brokering services in Serbia. 80 Economic Analysis (2022, Vol. 55, No. 1, 76-89) • Roughly half of the participants belong to a NES target group (these are individuals who belong to some vulnerable group and are considered difficult to employ). A much larger share among Roma than among non-Roma belongs to the NES target groups.16F6 This is true for both programs. Note that not all Roma are registered as such in the NES dataset. The reason is that NES has self-reported ethnicity information and some Roma prefer not to report their ethnicity to NES. Second, with respect to the formal employment before the trainings, we note the following: • On average, the individuals in our sample were either unemployed or unregistered in the week prior to starting the training. There are no differences in terms of labor market status between non-Roma and Roma in the week prior to the beginning of the training with the exception of participation in ALMPs. Roma were somewhat more likely to be included in ALMPs than non-Roma, but the participation in ALMPs is minor. Third, we find the following for formal employment after the training: • Most interestingly, with respect to the formal employment status after 6 months, we find that 20% to 21% of Roma were employed (depending on the program), while this share stood at 55% for non-Roma. Table 2. Socio-economic characteristics and labor market outcomes of participants according to NES data, by type of training Total N=509 Non-Roma YEP N=306 Roma YEP N=169 InR N=34 p-value YEP p-value Roma (1) (2) (3) (4) (5) [(2)-(3)] (6) [(3)-(4)] Personal-level variables Female 23.2% 10.3% 42.2% 47.1% <0.001 0.61 Age on 06mar2020 27 (23-32) 29 (25-33) 24 (21-30) 24 (20-31) <0.001 0.90 Highest level of education <0.001 0.007 Primary school or less 27.8% 12.7% 46.1% 73.5% Three-years VET school 28.6% 31.3% 29.2% 2.9% Four years secondary school (VET or general) 33.6% 41.6% 22.1% 17.6% University/college or higher 10.0% 14.4% 2.5% 5.9% NES target group*: Belongs to NES target group 56.2% 37.5% 83.8% 91.2% <0.001 0.27 Surplus of employees 4.2% 6.5% 0.6% 0.0% 0.004 0.64 Single parents 2.7% 0.7% 5.2% 8.8% 0.002 0.41 Unemployed parents 14.8% 10.7% 20.1% 26.5% 0.006 0.41 Internally displaced people 2.5% 1.4% 3.9% 5.9% 0.088 0.60 Recipient of social assistance 18.8% 5.5% 40.9% 32.4% <0.001 0.36 Roma 24.2% 0.0% 61.0% 64.7% <0.001 0.69 Other vulnerable 0.8% 1.4% 0.0% 0.0% 0.14 Status in 7 days before training start Employed 18.0% 23.4% 8.3% 11.8% <0.001 0.51 Unemployed 47.9% 46.2% 52.1% 44.1% 0.21 0.40 6 We rely on the previous paper to define NES target groups. NES classifies individuals into target groups in order to be able to target vulnerable groups with their programs. Lara Lebedinski 81 Total N=509 Non-Roma YEP N=306 Roma YEP N=169 InR N=34 p-value YEP p-value Roma (1) (2) (3) (4) (5) [(2)-(3)] (6) [(3)-(4)] ALMP 1.8% 0.9% 3.6% 2.9% 0.029 0.86 Out of labor-force 1.5% 1.2% 1.8% 2.9% 0.57 0.66 Unregistered 45.4% 44.2% 47.9% 44.1% 0.43 0.68 Number of days in 360 days before training start Employed 67 (±110) 88 (±121) 31 (±73) 37 (±86) <0.001 0.68 Unemployed 126 (±137) 117 (±135) 138 (±140) 156 (±143) 0.10 0.49 ALMP 3 (±24) 1 (±12) 7 (±36) 7 (±40) 0.010 0.99 Out of labor-force 7 (±45) 5 (±37) 9 (±53) 20 (±65) 0.31 0.29 Unregistered 154 (±142) 146 (±141) 173 (±142) 138 (±140) 0.043 0.19 Status in 7 days after training end Employed 19.9% 20.2% 19.5% 17.6% 0.85 0.80 Unemployed 36.4% 36.4% 39.1% 23.5% 0.56 0.086 ALMP 0.7% 0.6% 1.2% 0.0% 0.46 0.52 Out of labor-force 0.9% 0.3% 1.8% 2.9% 0.071 0.66 Unregistered 53.0% 50.0% 55.0% 73.5% 0.28 0.046 Status within 7 days after 6 months from training end Employed 42.6% 55.2% 21.3% n.a. <0.001 n.a. Unemployed 23.9% 18.8% 37.9% n.a. <0.001 n.a. ALMP 0.7% 0.3% 1.8% n.a. 0.071 n.a. Out of labor-force 0.7% 0.6% 1.2% n.a. 0.46 n.a. Unregistered 30.6% 26.3% 38.5% n.a. 0.005 n.a. Number of days in 180 days after 6 months training end Employed 62 (±69) 79 (±70) 29 (±53) n.a. <0.001 n.a. Unemployed 53 (±67) 43 (±61) 72 (±75) n.a. <0.001 n.a. ALMP 1 (±10) 1 (±8) 2 (±15) n.a. 0.29 n.a. Out of labor-force 2 (±14) 1 (±6) 3 (±21) n.a. 0.077 n.a. Unregistered 60 (±68) 54 (±65) 72 (±74) n.a. 0.007 n.a. Notes: *Multiple answers possible. Continuous measures are summarized by the median, followed by the interquartile range in brackets (p25-p50). Or by the mean followed by ± and the Standard Deviation in brackets. Out-of-labor force are individuals who registered with NES, but are not actively searching for work for some justifiable reason (e.g. they are sick, on maternity leave, etc.). A p-value of 0.05 or smaller implies that a difference is statistically significant at 5%. n.a. - not available. Source: Author’s calculations based on NES dataset. Finally, in Figure 1 we turn to the graphical presentation of the trajectories of labor market outcomes for the non-Roma and Roma trainees (data shown in Table 2). Each graph shows the percentage of individuals by registration status in each week where 0 denotes the training end. We denote with a vertical line the maximum and minimum duration of the training relative to the training end. Before entering the training, both non-Roma and Roma had a low likelihood to hold a formal job. Among the non-Roma the formal employment rate was somewhat above 20%, while the share of employed Roma was approximately 10%. Somewhat less than 40% of non-Roma were registered unemployed before the training start and the rest, approximately 40% were neither 82 Economic Analysis (2022, Vol. 55, No. 1, 76-89) registered employed with CROSO (Social Insurance Central Register) nor registered unemployed with NES. Following the training start, the employment rate increases somewhat for non-Roma and it increases sharply for the Roma by approximately 20 percentage points until the end of the training. The reason for this finding is that Roma were given employment contracts by YEP during the period of trainings which made it possible to pay them a pocket money. After the trainings ended, there is a positive effect on employment for both non-Roma and Roma, but this effect is more strongly pronounced for non-Roma. The employment rate rises above 50% over the observed period, while the employment rate of Roma increases above 30% (first row of graphs). On the other hand, both the share of unemployed and the share of those not registered is falling. Figure 1. Percentage of individuals by labor market status (0 denotes the training end) Source: Author’s calculations based on NES dataset IMPACT ANALYSIS The descriptive analysis provides an indication that the short-term skills training improved the employment rate of participants. In this section we aim to identify the causal impact of the training: we want to compare the actual labour market outcomes of participants with the counterfactual outcomes, i.e. the potential outcomes had they not attended the skills training. The main evaluation problem and challenge in this type of exercise is that participants were not randomly selected for the training. First, the participants belong to a disadvantaged group (both non-Roma and Roma) and this is a consequence of the selection criteria of the YEP project. Presumably, the participants have worse employment opportunities than the average person of their age. Second, participants were likely actively searching for work and this is how they found out about this opportunity to attend a short-term skills training. This implies that the participants Lara Lebedinski 83 were very motivated to find options to improve their labor market situation. Conversely, a high level of motivation implies that the participants would possibly have a higher chance of employment even in the absence of the training program.17F7 The main challenge is to identify a credible control group and we follow the methodology which has been used in a previous study (Bachman et al., 2019) to estimate the causal impact of labour market programs in Serbia. The identification strategy employed is a statistical matching methodology based on the seminal paper of Sianesi (2004). The idea is to select from the the potential control group, i.e. all people from the NES database, a smaller control group ("matched control group") which has similar socio-demographic characteristics and a similar labour market history to the treatment group. The estimated impact of a specific skills training program is the comparison of labour market outcomes between the treated group and the "matched control group". We estimate in this way the impact for each skills training separately.18F8 Assessment of matching quality In Table 3 we compare the GIZ YEP trainees with the matched sample. The comparison sample is constructed for non-Roma and Roma jointly. The control sample counts 8,093 individuals and the intervention sample is reduced to 239 participants among which 61 are Roma. The reason that the number of training participants falls is because we select only trainings with at least 18 participants. Remember that we had to exclude Roma from the InR project because we did not have their labor market outcomes for the period 6 months after the trainings. With the exception of the Roma variable there were no significant differences before the training took place. Six months after the training ended, there is a significant improvement in employment for the participants of GIZ trainings. These findings suggest that matching i.e. the selection of the control group was very successful and that the program improved the labor market outcomes of participants. Table 3. Comparison of GIZ trainees with matched control group Total N=8,332 Control N=8,093 Treatment N=239 p-value Personal-level variables Age on 01apr2019 28 (24-33) 28 (24-33) 28 (24-32) 0.45 1=female 16.1% 16.1% 15.5% 0.79 Highest level of education 0.96 Primary school or less 21.8% 21.8% 21.8% Three-years VET school 29.0% 29.0% 28.5% Four years secondary school (VET or general) 38.0% 37.9% 39.7% University/college or higher 11.2% 11.2% 10.0% NES target group*: Belongs to NES target group 48.9% 48.8% 53.1% 0.18 Surplus of employees 5.3% 5.3% 5.0% 0.83 Single parents 1.2% 1.2% 1.7% 0.48 Unemployed parents 14.0% 14.0% 15.1% 0.63 Internally displaced people 1.4% 1.4% 2.1% 0.36 7 Note that Roma in YEP and in InR were reached mostly through Roma associations, who went directly to the settlements, informed them and introduced them to the trainings; otherwise they would not be in a position to look for such programs or employment by themselves, especially women. We still believe that their choice to enter this program demonstrates that they had a higher level of motivation to find employment than other Roma with similar characteristics. 8 Details about the methodology can be found in Bachmann et al. (2019). 84 Economic Analysis (2022, Vol. 55, No. 1, 76-89) Total N=8,332 Control N=8,093 Treatment N=239 p-value Recipient of social assistance 10.9% 10.8% 13.8% 0.14 Roma 17.8% 17.6% 25.5% 0.002 Other vulnerable groups 0.0% 0.0% 0.0% 0.12 Status in 360 days before training start Employed 22.8% 22,8% 22,1% 0.74 Unemployed 37% 37% 38,5% 0.58 ALMP 1.1% 1,1% 1,4% 0.44 Out of labor-force 1.9% 1,9% 2,2% 0.80 Unregistered 37.0% 37% 35,8% 0.60 Number of days in 360 days before training start Employed 82 (±124) 82 (±124) 79 (±117) 0.74 Unemployed 133 (±141) 133 (±141) 138 (±135) 0.58 ALMP 4 (±27) 4 (±27) 5 (±31) 0.44 Out of labor-force 7 (±44) 7 (±44) 8 (±47) 0.80 Unregistered 133 (±141) 133 (±141) 128 (±131) 0.60 Status in 7 days after training end Employed 25.7% 25.8% 23.4% 0.42 Unemployed 39.0% 39.0% 38.9% 0.97 ALMP 1.6% 1.6% 0.8% 0.35 Out of labor-force 2.0% 2.1% 1.3% 0.38 Unregistered 33.3% 32.8% 35.8% 0.88 Status in 7 days 6 months after training end Employed 33.0% 32.5% 49.4% <0.001 Unemployed 30.9% 31.1% 22.6% 0.005 ALMP 1.4% 1.4% 0.4% 0.19 Out of labor-force 1.8% 1.9% 0.4% 0.099 Unregistered 33.6% 33.7% 28.9% 0.12 Number of days in 180 days 6 months after training end Employed 52 (±75) 52 (±75) 67 (±69) 0.001 Unemployed 61 (±77) 61 (±77) 51 (±65) 0.056 ALMP 2 (±18) 2 (±19) 1 (±6) 0.12 Out of labor-force 3 (±23) 4 (±23) 1 (±13) 0.12 Unregistered 60 (±77) 60 (±77) 57 (±66) 0.50 Notes: *Multiple answers possible. Continuous measures are summarized by the median, followed by the interquartile range in brackets (p25-p50). Or by the mean followed by ± and the Standard Deviation in brackets. A p-value of 0.05 or smaller implies that a difference is statistically significant at 5%. Source: Author’s calculations based on the NES dataset. Impact estimates Figures 2 and 3 display the evolution of labor market outcomes of participants sample and respective matched group sample for non-Roma and Roma. The respective impact estimates and their statistical significance19F9 at selected month (3, 6, 9 months) are reported below each graph. The matching quality can be visually confirmed by examining the labor market evolution of the outcomes for treated and control group prior to entering the training. Remember that, starting from the left in each graph, the first vertical solid line marks the beginning of the longest training. The second vertical solid line marks the median start of the training and the third solid line marks the end of the training. 9 Roughly, an absolute t-value of |1.96| indicates statistical significance on the 5 percent confidence level. Lara Lebedinski 85 The positive intervention impact on employment for non-Roma rises gradually and it reaches 28.60 percentage points after 6 months (26 weeks) as shown in Figure 2. Conversely, the share of registered unemployed falls over the observed period suggesting that those who were registered unemployed are the ones who found employment. The share of individuals who are neither registered with NES nor employed remains unchanged over the time frame. Some of these individuals are certainly informally employed, but we do not have a means to prove this claim. Turning now to the intervention impact for Roma, we note that there is a similar positive trend for employment over time as observed for the non-Roma. The impact estimates for employment are 15.55 percentage points after six months for Roma. Also, for Roma, the graphs suggest that the registered unemployed are the ones who found employment as their share decreases gradually over time. Again, the share of individuals who are neither registered with NES nor employed is constant. When comparing the impact on non-Roma and Roma, we observe a much larger effect for non-Roma. It is difficult to pinpoint the exact cause, but possible reasons include the following. First, Roma are less educated and younger (so they presumably have less experience) than non- Roma and both of these factors affect the employability of individuals. Second, Roma attended different trainings20F10 than non-Roma which might not necessarily offer the same labor market opportunities. It is possible to compare the estimated impact size to an international review of vocational training program impacts from 12 evaluations from 8 countries. It should be noted that such comparisons have several caveats. First, the content, intensity and duration of trainings in each setting is different and similarly, the background of the participants is different. Second, this international review uses survey data while this paper relies on administrative data. Third, the impact estimates in this paper are 6 months after training end while the international review provides estimates after 12 to 18 months. It is not clear in which direction these differences can bias the results, but having these caveats in mind, we can compare the key outcome formal employment. The impact size of comparable programs in other countries ranged from no impact up to an impact of the size of 8 percentage points on formal employment. This is much lower than the impact estimates of the size 15.55 percentage points that we found for YEP Roma participants or equivalently 28.60 percentage points for YEP non-Roma participants and we conclude that the program is very successful also in international comparison. Finally, based on the findings in this paper, it is possible to recommend the continuation of the program. The program targets a disadvantaged group and additional job search support after the training end would be very valuable. Furthermore, to ensure that such a high and positive impact persists, it is recommended to continue monitoring the implementation and conduct impact evaluations on a regular basis after different time periods (e.g. after six months, 12 months, 18 months, two years and longer). 10 The difference between the trainings for non-Roma and Roma derives from the fact that some trainings were shaped specifically having in mind the interests of the Roma. 86 Economic Analysis (2022, Vol. 55, No. 1, 76-89) Figure 2. Share of treatment and matched control group in each labor market status by week relative to training end: Non-Roma Source: Author's calculation based on NES dataset Lara Lebedinski 87 Figure 3. Share of treatment and matched control group in each labor market status by week relative to training end: Roma Source: Author's calculation based on NES dataset 88 Economic Analysis (2022, Vol. 55, No. 1, 76-89) CONCLUSION In this paper we have examined the labour market outcomes of Roma after their attendance of short-term skills trainings. We provide additionally data for non-Roma as a benchmark. The descriptive statistics reveals that participants of the skills trainings are predominantly males in their twenties who finished secondary education. Among Roma, there are more females than among non-Roma, most likely due to the content of the training that they attended (e.g. there were a number of trainings attracting mainly females, such as trainings for manicure, pedicure, hairdressers, etc.). A comparison between non-Roma and Roma shows that Roma participants are, on average, less educated. According to administrative data, the individuals in our sample were mostly either unemployed or unregistered in the week prior to starting the training. The impact analysis suggests a positive causal impact of the skills trainings of the size of 28.60 percentage points after 6 months on employment for non-Roma. We conjecture that the individuals who find employment after the trainings are those who were registered unemployed prior to the training start (and not unregistered). Compared to the control group, for non-Roma the share of registered unemployed falls by 18 percentage points and the share of unregistered falls by 5.8 percentage points. For Roma, the impact analysis suggests a positive causal impact of 15.55 percentage points on employment after 6 months. We believe that individuals who were formally registered unemployed before attending the skills trainings are the ones who found employment as a result of the program. In international comparison, these estimates are very high suggesting that the program has an exceptionally high impact on program participants (both non-Roma and Roma). We note that the impact estimates on employment are different between non-Roma and Roma. It is difficult to pinpoint the exact cause, but possible reasons include the following. First, Roma are less educated and younger (so they presumably have less experience) than non-Roma and both of these factors affect the employability of individuals. Second, as some trainings for Roma were shaped in line with their interests, as a result, some Roma attended different trainings than non-Roma in terms of content which ultimately might not offer the same labor market opportunities. ACKNOWLEDGEMENTS We would like to thank data analysis.code for the Stata for sharing the Jonathan Stöterau REFERENCES Aleksic, D., Anic, A., Arandarenko, M., Krstic, G., Ognjanov, G., Vuksanovic, N., and Žarkovic, J. (2021). Youth Situation in Serbia: Employment, Skills and Social Inclusion. European Training Foundation. Bachmann, R., Kluve, J., Martinez Flores, F. and Stöterau, J. (2019). Employment impacts of German development cooperation interventions: A collaborative study in three pilot countries, RWI Projektberichte, RWI - Leibniz-Institut für Wirtschaftsforschung, Essen. Battaglia, M., and Lebedinski, L. (2015). Equal access to education: An evaluation of the Roma teaching assistant program in Serbia. World Development, 76, 62-81. Battaglia, M., and Lebedinski, L. (2017). The curse of low expectations: remedial education and perceived returns to education of Roma people. Economics of transition, 25(4), 681-721. Brüggemann, C. (2012). Roma Education in Comparative Perspective. Analysis of the UNDP/World Bank/EC Regional Roma Survey 2011. Roma Inclusion Working Papers. Bratislava: United Nations Development Programme Card, D., Kluve, J., & Weber, A. (2010). Active Labour Market Policy Evaluations: A Meta- Analysis. The Economic jJournal, 120(548), 452-477. Lara Lebedinski 89 Card, D., Kluve, J., & Weber, A. (2018). What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations. Journal of the European Economic Association, 16(3), 894–931. Doyle H., “Improving Access of Roma to Health Care through the Decade of Roma Inclusion”, OSI, (2004) Eichhorst, W., and Rinne, U. (2018). Promoting youth employment in Europe: Evidence-based policy lessons. In European youth labour markets, 189-204. Springer. Kluve, J., Puerto, S., Robalino, D., Romero, J. M., Rother, F., Stöterau, J., Weidenkaff, F., & Witte, M. (2019). Do youth employment programs improve labor market outcomes? A quantitative review. World Development, 114, 237–253. Lebedinski, L. (2020). The Effect of Residential Segregation on Formal and Informal Employment of Roma in Serbia. Eastern European Economics, 58(2), 108-136. Marjanovic, D. (2016). Labour market transitions of young women and men in the Republic of Serbia (Work4Youth Publication Series No. 36). International Labour Office. Mihailov, D. (2012). The health situation of Roma communities: Analysis of the data from the UNDP/World Bank/EC Regional Roma Survey 2011. Roma Inclusion Working Papers. Bratislava: United Nations Development Programme O'Higgins, N. (2012). Roma and non-Roma in the Labour Market in Central and South Eastern Europe. Roma Inclusion Working Papers. Bratislava: United Nations Development Programme. O’Higgins, N. (1997). The challenge of youth unemployment. International Social Security Review, 50(4), 63–93. Rokicka, M., Unt, M., Täht, K., & Nizalova, O. (2018). Youth labour market in Central and Eastern Europe. In European youth labour markets, 61-78. Springer. Sianesi B. (2004). "An Evaluation of the Swedish System of Active Labor Market Programs in the 1990s," The Review of Economics and Statistics, 86(1), pp. 133-155. Verick S. (2009). Who is Hit Hardest During a Financial Crisis? The Vulnerability of Young Men and Women to Unemployment in an Economic Downturn. IZA Discussion Paper No. 4359. Article history: Received: January 12, 2022 Revised: June 9, 2022 Accepted: June 13, 2022