Microsoft Word - 23920 July 2020_RS_CL02 (PD ACCEPTED CHANGES 8.6)_CL_RS.docx Original Research © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 3 Confirmatory Factor Analysis for the Service-Learning Outcomes Measurement Scale (S-LOMS) Ka Hing Lau1 and Robin Stanley Snell2 1 Office of Service-Learning, Lingnan University, 2 Department of Management, Lingnan University Cite as: Lau, K.H. & Snell, R.S., (2020). Confirmatory Factor Analysis for the Service-Learning Outcomes Measurement Scale (S-LOMS). Metropolitan Universities, 32(1), 3-34. DOI: 10.18060/23920 This is an open access article distributed under the terms of the Creative Commons Attribution License. Editor: Valerie L. Holton, Ph.D. Abstract Service-learning was introduced into Hong Kong over a decade ago, yet there is a research gap about the self-perceived developmental outcomes for students, partly due to the lack of a reliable measurement instrument across course disciplines and types of service-learning. This study validated a recently created Service-Learning Outcomes Measurement Scale (S-LOMS) through confirmatory factor analysis (CFA) with data from 629 students. S-LOMS measures self- perceived student development through 56 items, which cover outcome domains under four overarching categories: knowledge application, personal and professional skills, civic orientation and engagement, and self-awareness. Alternative measurement models were compared in this validation exercise, with the results indicating that although a model with 11 domains and without overarching categories was preferred, there was also support for a model with 10 domains subsumed under the four abovementioned overarching categories. Multi- sample analyses indicated that both models were stable across gender. The practical implication of our findings is that for the purpose of measuring the developmental impacts on students of engaging in service-learning, S-LOMS offers investigators a number of options besides using the entire 56-item scale. Some administrative options are described at the end of the paper. Keywords: Service-learning, Hong Kong, student developmental impacts, measurement instruments © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 4 Introduction Service-learning is “a form of experiential education in which students engage in activities that address human and community needs together with structured opportunities intentionally designed to promote student learning and development” (Jacoby, 1996, p. 5). This transformative pedagogy has spread around the globe (Shumer, Stanton, & Giles, Jr., 2017). Hong Kong is no exception, as over the past decade service-learning has been introduced to most universities there (Ma, 2018; Ma & Chan, 2013). Developmental outcomes for students arising from service- learning have been well illustrated in the west (e.g. Astin & Sax, 1998; Astin, Vogelgesang, Ikeda, & Yee, 2000; Celio, Durlak, & Dymnicki, 2011; Conway, Amel, & Gerwien, 2009; Driscoll, Holland, Gelmon, & Kerrigan, 1996; Felten & Clayton, 2011; Einfeld & Collins, 2008; Eyler & Giles, 1999; Elyer, Giles, Stenson, & Gray, 2001; Lundy, 2007; Novak, Markey, & Allen, 2007; Prentice, 2007; Richard, Keen, Hatcher, & Pease, 2017; Rama, 1998; Simon & Cleary, 2006; Warren, 2012; Yorio & Ye, 2012). Evaluating developmental outcomes for students is considered to be one of the five critical elements of successful service-learning programs, according to Jacoby (1996). This is because evaluation sheds light on directions for improvement and growth, and it encourages sustainable collaboration between schools and the community. However, the body of scholarly publications about student development arising from service-learning in Asia remains limited (Xing & Ma, 2010; Snell & Lau, 2020). This scarcity can be attributed to the absence of a standardized and reliable measurement instrument for assessing those student developmental outcomes after service-learning experience that are targeted by universities in Hong Kong. There have only been a small number of similar measurement instruments used in the past for this purpose, and these have had various inadequacies. Problems have included: (a) importing scales which have been developed in the west into Asian contexts, and which are subject to standardization concerns (e.g. Lo, Kwan, Chan, & Ngai, 2016; Ngai, 2009; Siu, Tang, & Lai, 2013); (b) covering a limited set of developmental domains (e.g. Eyler, Giles, & Braxton, 1997); and (c) having insufficient target respondents for scale validation (e.g. Ma, Chan, & Tse, 2019). Addressing this gap, Snell and Lau (2020) developed an instrument named the “Service-Learning Outcomes Measurement Scale (S-LOMS).” S-LOMS measures learning outcomes perceived by students under four overarching categories covering 15 domains. The first category is knowledge application and has one cognominal domain. The second category is personal and professional skills, which comprises the six domains of creativity, problem solving skills, relationship skills, team skills, self- reflection skills, and critical thinking skills. The third category is civic orientation and engagement, which comprises the five domains of commitment to social betterment, understanding community, empathy and caring for others, respecting diversity, and sense of social responsibility. The fourth category is self-awareness, which comprises the three domains of self-efficacy, self-understanding, and commitment to self-improvement. S-LOMS consists of 56 descriptive items regarding the above domains, to which participants respond with a 10-point Likert scale, ranged from 1 as “strongly disagree” to 10 as “strongly agree.” © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 5 In comparison with similar measurement instruments that have been adopted in the past, S- LOMS carries several merits. First, it has been designed for the context of Hong Kong, reflecting the local culture and recent developments within the higher education sector there (Snell & Lau, 2020). Second, the set of domains included in S-LOMS comprehensively covers the desired developmental outcomes of Hong Kong based service-learning programs. Third, the administration of S-LOM is both standardized and flexible, such that practitioners can elect to measure developmental categories or domains, according to their needs. Fourth, S-LOM is expected to undergo rigorous validation before its practical implementation. In a previous validation study (Snell & Lau, 2020), S-LOMS was tested with 400 Hong Kong university students, and the current study involves a further 600-plus respondents. It is intended that there will be subsequent studies of test-retest reliability and criterion validity, which will engage additional respondents. We anticipate that the conceptual relevance and scale validity of S- LOMS will attract its usage by service-learning practitioners as a tool for assessing progress on the enhancement of developmental outcomes for students. The starting point for the development of S-LOMS as a measurement instrument was a review of the common student developmental domains arising from service-learning, as documented in past literature. This was followed by considering the special educational and social context for service-learning in Hong Kong. For example, within the overarching category of civic orientation and engagement, the instrument was oriented more toward moral development than participatory democracy. To further match the emerging instrument to the local context, the authors also invited local service-learning practitioners to examine the developmental domains and proposed items in the development process. As a result, 15 developmental domains under the four aforementioned overarching categories were identified. An initial study (Snell & Lau, 2020) was then conducted to validate S-LOMS based on its administration with a sample of 400 university students. S-LOMS was found to have satisfactory internal consistency with the underlying dimensionality uncovered through exploratory factor analysis (EFA) by using the method of Principle Components with oblimin rotation. In that study, regarding reliability, S-LOMS achieved the Cronbach’s alpha value above .70 for its four categories, while the 15 original domains collapsed into 11, as follows. Creativity and problem solving skills combined into the higher-order domain of creative problem solving skills. Another higher-order domain comprised relationship and team skills. A third higher-order domain, community commitment and understanding, combined commitment to social betterment with understanding community. A fourth higher-order domain, caring and respect, combined empathy and caring for others with respecting diversity. The other domains remained discrete. The current study continues the measurement instrument validation journey. This paper reports the validation results of testing S-LOMS with a new sample through confirmatory factor analysis (CFA) against the above factor structure that emerged in the previous EFA study (Snell & Lau, 2020). It is intended that subsequent research not reported here will test for other types of © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 6 validity regarding S-LOMS, such as test-retest reliability, and will then use S-LOMS to measure developmental outcomes for students through before and after administration around service- learning experience. The above practice is a typical step in the scale development process (e.g., Brown, 2015; Hurley, Scandura, Schriesheim, Brannick, Seers, Vandenberg, & Williams, 1997; Tay & Jebb, 2017; Worthington & Whittaker, 2006). While EFA is used to identify the dimensionality for a set of variables, it does not force variables to be loaded on certain factors in advance. By contrast, CFA tests whether data fits a pre-specified factor structure (Stevens, 2009). The current study tested a series of alternative models with various factor structures. Since the 11-domain factor structure discussed above had received empirical support from only one prior EFA study, the current study adopted a prudent approach in testing that structure together with the originally theoretical 15-domain factor structure proposed by Snell and Lau (2020), together with other possible structures, so as to compare which one would provide a better fit with a new set of data. Methods Participants The current study recruited 629 university students from four Hong Kong government universities, namely Lingnan University, The Hong Kong Polytechnic University, Hong Kong Baptist University, and The Education University of Hong Kong. Female respondents constituted a larger part of the sample (59.5%) and the average age was 20.5 (s.d. = 2.21). Broken down by major disciplines, the sample comprised engineering and science (40.9%), business (19.9%), social sciences (14.3%), arts (12.7%), and healthcare (12.2%). Among the respondents, 65.8% had previous service-learning experience or were in the process of taking service-learning programs or courses. Instrument The original structure with four overarching categories and 15 domains described above was employed in the construction of S-LOMS. In the 56-item instrument administered to the students (see Appendix 1), there were three to four items for each of the 15 domains, in the form of self- descriptive statements. Respondents were asked to indicate the extent of their agreement with the items on a 10-point Likert scale (from 1, “strongly disagree” to 10, “strongly agree”). Procedures The respondents were invited to answer S-LOMS on a voluntarily basis in a classroom setting, with consent from the instructors of the respective courses, which did not necessarily involve service-learning. Besides S-LOMS, the students completed some demographic items about © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 7 gender, age, academic background, and prior service-learning experience. Upon completing the questionnaire, the students received a HK$50 supermarket voucher. Statistical Analysis CFA was employed in the analysis, using EQS version 6.4 for Windows, and the extent of missingness of the sample data and the assumption of multivariate normality were checked, in order to decide the estimation methods. Regarding data missingness, 520 of the 629 participants (82.7%) provided no missing responses. The mean percentage of missing responses in an item was 0.4%. 63 missing patterns were identified among the 109 respondents with missing responses. Moreover, the sample was tested with multivariate normality. The related indices provided by the EQS indicated violation of the assumption. Specifically, both the Yuan, Lambert, and Fouladi’s coefficient (1,332.76) and its normalized estimate (208.24) showed values over 5.00, inferring the nonnormally distributed pattern of the sample data (Bentler, 2006). As the data showed incomplete and nonnormal patterns, the full information maximum likelihood (FIML) method with robust correction was employed in EQS for the CFA execution, recommended by Bentler (2006). The scaled chi-square (Yuan-Bentler, i.e. Y-B c2) and other indices under Yuan-Bentler’s correction of the results were adopted for deciding goodness of fit for the models. This approach is regarded as an effective adjustment procedure when the model violates multivariate normality and is applied to incomplete data (Blunch, 2016; Byrne, 2008; Savalei & Bentler, 2005). In executing the analysis on the models specified below, a typical CFA parameterization was adopted, as described in steps (a)-(d). In step (a), the first path between each designated factor (whether a learning domain or an overarching category) and its first variable (whether an assigned item or a learning domain) was set as 1.0, for the sake of model identification and latent variable scaling. In step (b), all other parameters and factor variances were freely estimated. In step (c), a constant variable V999 with no variance and a mean value of 1.0 was created for each variable equation. In step (d), covariances were freely estimated between each designated factor. For the sake of comparison, no modification such as error covariances were made to the models. As the model chi-square test, although commonly used, is subject to a number of limitations (Hooper, Coughlan, & Mullen, 2008) and tends to be rejected as not fitting (Thompson, 2004), other goodness of fit indices, including CFI, NNFI, and RMSEA were also used for assessing the models (Tabachnick & Fidell, 2013). Since the robust correction was implemented, the values of the above indices under the results of Yuen-Bentler correction was adopted. Acceptable model fit was defined as follows: CFI (³ .90); NNFI (³ .90); RMSEA (≤ .08) (Bentler, 1990; Brown, 2015; Browne & Cudeck, 1992). Since a series of models (see the next section) with different factor structures were tested, model AIC indices were employed in comparing the competing © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 8 models. These are among the most commonly adopted indices for the comparison of non-nested models by using chi-square values (Brown, 2015). The smallest AIC value indicates the best fitting model, under the condition that the models are non-nested. Models Specification Since S-LOMS is a newly established measurement instrument with only one prior EFA validation to support its internal factor structure, the current study tested, through a series of CFAs, whether the data fitted other possible factor structures for the instrument, besides the one already reported by Snell and Lau (2020). The seven models that were tested are represented in Table 1 and are explained next. Model 1 serves as a baseline model, within which all items are loaded onto a single factor. Model 2 is theoretically grounded to the extent that the items are assumed to load directly onto their respective overarching categories identified in prior literature, of which there are four: knowledge application, personal and professional skills, civic orientation and engagement, and self-awareness. The developmental outcome domains such as relationship skills were omitted from this model. Models 3 and 4 tested whether items loaded onto their corresponding developmental outcome domains irrespective of the overarching categories (developmental domains directly to corresponding items). Model 3 was theoretically based, to the extent that it comprised the original 15 domains that S-LOMS had originally been designed to measure (Snell & Lau, 2020). For example, creativity and problem-solving skills were retained as two separate domains instead of being merged into the single domain of creative problem solving skills. However, the four overarching categories were not included in this model. By contrast, Model 4 was empirically based, to the extent that it combined some pairs among the original 15 outcomes to match the 11 domains that had been discovered in the previous EFA study (Snell & Lau, 2020). Model 5 and Model 6 involved two layers of factors, and constituted hybrids of Model 2 with either Model 3 or Model 4. Both Model 5 and Model 6 were theoretically based, to the extent that they included the four overarching categories. In addition, Model 5 included the 15 theoretically based original outcome domains, whereas Model 6 included the 11 domains from the previous EFA study. An additional model, Model 7, was a modification of Model 6 that was created by combining the domains of sense of social responsibility with the domain of community commitment and understanding under the overarching category of civic orientation and engagement, as is explained in the next section. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 9 Table 1. Model Specifications for the Alternative Models of S-LOMS Model Description Graphic Illustration 1 56 items loaded on one factor 2 56 items loaded on four overarching categories (factors) Factor 1 Item 1 Item 2 Item 3 Item 54 Item 55 Item 56 ‧ ‧ ‧ Item 1 Item 4 Item 5 Item 45 Item 46 Item 56 ‧ ‧ ‧ Item 27 Item 28 Factor 1 Factor 2 Factor 3 Factor 4 ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 10 3 56 items loaded on 15 developmental outcome domains (factors) 4 56 items loaded on 11 developmental outcome domains (factors) Item 1 Item 4 Item 5 Item 54 Item 56 ‧ ‧ ‧ Item 8 Domain 1 Domain 2 Domain 15 ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ 15 F ac to rs c or re la te w ith e ac h ot he r Item 1 Item 4 Item 5 Item 54 Item 56 ‧ ‧ ‧ Item 8 Domain 1 Domain 2 Domain 11 ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ ‧ 11 F ac to rs c or re la te w ith e ac h ot he r © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 11 5 56 items loaded on 15 developmental outcome domains then four overarching categories 6 56 items loaded on 11 developmental outcome domains then four overarching categories Item 1 Item 4 Item 5 ‧ ‧ Item 8 Domain 1 Domain 2 ‧ ‧ Category 1 Category 2 Category 3 Category 4 Item 25 Item 27 Domain 7 ‧‧ ‧ ‧ ‧ ‧ Item 28 Item 31 Domain 8 ‧ ‧ Item 43 Item 45 Domain 12 ‧‧ ‧ ‧ ‧ ‧ Item 46 Item 50 Domain 13 ‧ ‧ Item 54 Item 56 Domain 15 ‧‧ ‧ ‧ ‧ ‧ Item 1 Item 4 Item 5 ‧ ‧ Item 12 Domain 1 Domain 2 ‧ ‧ Category 1 Category 2 Category 3 Category 4 Item 25 Item 27 Domain 5 ‧‧ ‧ ‧ ‧ ‧ Item 28 Item 35 Domain 6 ‧ ‧ Item 43 Item 45 Domain 8 ‧‧ ‧ ‧ ‧ ‧ Item 46 Item 50 Domain 9 ‧ ‧ Item 54 Item 56 Domain 11 ‧‧ ‧ ‧ ‧ ‧ © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 12 7 A modification of Model 6 with the combined developmental outcome domains of community commitment and understanding and the domain of sense of social responsibility Item 1 Item 4 Item 5 ‧ ‧ Item 12 Domain 1 Domain 2 ‧ ‧ Category 1 Category 2 Category 3 Category 4 Item 25 Item 27 Domain 5 ‧‧ ‧ ‧ ‧ ‧ Item 28 Item 38 Domain 6 + 8 ‧ ‧ Item 39 Item 45 Domain 7 ‧‧ Item 46 Item 50 Domain 9 ‧ ‧ Item 54 Item 56 Domain 11 ‧‧ ‧ ‧ ‧ ‧ © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920| August 14, 2020 13 Results Model Comparison Table 2 reports the CFA results in terms of the chi-square test, goodness of fit indices, and AIC indices. The chi-square values for all these models were statistically significant, reflecting that large sample size increased the power of the test and thus the likelihood of rejection as not an exact fit. Accordingly, the goodness of the fit indices were taken into consideration (Bentler, 1990), and Models 3, 4, 5, and 6 demonstrated acceptable model fit, with both NNFI and CFI at marginally 0.9 or above, and RMSEA and its 90% confidence interval at or lower than .05. Moreover, all absolute values of standardized residual were small, indicating that those models fit the data well enough. Table 2. Fit Indices & Standardized Residuals for the Alternative Models of S-LOMS Item Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 Model 7 Y-B c2 * 4,245.33 3,811.88 3,340.77 3,450.80 3,635.83 3,607.38 3,631.76 df 1,484 1,478 1,379 1,429 1,465 1,469 1,470 p .00 .00 .00 .00 .00 .00 .00 NNFI .870 .890 .901 .902 .897 .899 .898 CFI .875 .894 .911 .909 .902 .903 .902 RMSEA .054 .050 .047 .047 .048 .048 .048 90% CI .052, .056 .048, .052 .045, .049 .045, .049 .046, .050 .046, .050 .046, .050 AIC 1,277.33 855.88 582.77 592.8 705.83 669.38 691.76 Average Absolute Standard -ized Residual .084 .035 .034 .034 .041 .040 .040 Note: * Y-B c2 denotes the Yuan-Bentler scaled chi-square values with robust correction applied. The fit indices in the table are also adopting the version of robust correction. Comparing the AIC indices for the above four models indicates that Model 3 is the best fit, followed by Model 4, 6 and 5, in preference order. Despite being the best fit, the results for Model 3 nonetheless indicate two issues. Specifically, the factor correlations between two pairs of learning domains, namely 1) creativity and problem solving skills; and 2) commitment to social betterment and understanding community, are 1.0, and correspond to Snell and Lau’s (2020) results in the earlier EFA, which led to the creation of higher-order domains, such as “creative problem solving skills”. Factor correlations approaching 1.0 constitute strong grounds © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 14 for combining multiple factors into a single factor, given the poor discriminant validity that is implied (Brown, 2015). A similar issue was found with Model 5, which put the 15 domains under four overarching categories, in that the factor coefficient between understanding community and its overarching category of civic orientation and engagement was found to be 1.0. Because of these issues, Model 3 and Model 5 were dropped and only those models with a structure involving 11 domains were considered. Among the remaining models, Model 4 was preferred, given its low AIC value, acceptable goodness of fit (Y-B c2 = 3,450.80; df = 1,429; p = .00; NNFI = .902; CFI = .909; RMSEA = .047, CI = .045, .049), and good factor loadings and factor correlations. Model 6, with 11 domains under four overarching categories, was also found to have an issue, in that the path of the domain of sense of social responsibility obtained 1.0 of factor loading from its parent category, indicating the need for further structure simplification under civic orientation and engagement. Accordingly, Model 7 was created as a modification of Model 6 by combining the two conceptually related domains of sense of social responsibility and community commitment and understanding. Model 7 obtained acceptable overall goodness of fit (Y-B c2 = 3,631.76; df = 1,470; p = .00; NNFI = .898; CFI = .902; RMSEA = .048, CI = .046, .050), and a relatively low AIC value (691.76). The 56 items and the 10 domains loaded with statistical significance on their respective domains and categories, nearly all with scores over .60 (except two items with loadings close to .60), while the four categories were significantly yet not perfectly correlated. Although usually more parsimonious models (i.e. Model 4) would be preferred, a more complex model may also be considered if it is based on a theory that can “substantially improve understanding of the phenomenon or can substantially broaden the types of phenomena understood using that theoretical approach” (Stevens, 2009, p. 572). In our case, the results of the CFA for Model 7 imply that S-LOMS can also further understood as a 10- domain model with four overarching categories. In summary, while the fit indices implied that Model 4 was the best model for S-LOMS, inspection of factor loadings led to the creation of Model 7, which was retained for further consideration in the next step, where Model 4 and Model 7 were examined for their stability on gender by using multi-sample analysis. Multi-sample Analysis Multi-sample analysis, or the factorial invariance test, is especially suitable for testing whether a particular model structure or relationships between factors in a model is applicable across samples by different types of categorization (Schumacker & Lomax, 1996). The dataset was divided into two samples by gender (248 male and 364 female). The demographic profile, including mean age and academic backgrounds, for the two sub-samples is listed in Table 3 below. The missingness of both male and female samples revealed an acceptable pattern, with around or over 80% of the responses did not contain any missing responses. As with the previous © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 15 analysis, the FIML method with the Yuan-Bentler Correction was employed in model estimation, given the incomplete data with the multivariate nonnormal pattern for both samples (see Table 3). Table 3. Demographics & Missingness of the Groups in the Multi-sample Analysis Male Female No. of Respondents 248 364 Mean Age 20.21 20.71 Academic Background Arts 6.9% 16.9% Social Science 11.0% 16.9% Business 16.3% 21.9% Engineering & Science 59.8% 27.5% Healthcare 6.1% 16.7% Service-Learning Experience Without 36.2% 32.8% With 63.8% 67.2% Missingness No. (percentage) of participants without missing responses 194 (78.2%) 314 (86.3%) No. of missing patterns 41 37 The mean percentage of missing responses in an item 0.6% 0.3% Multivariate Normality Yuan, Lambert, & Fouladi's coefficient 857.15 1,029.60 Normalized estimate 84.23 122.24 © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 16 We followed the approach recommended by Tabachnick and Fidell (2013) to perform the multi- sample analysis. We began with the baseline model for the two samples, and constrained a different parameter in each round to test whether the chi-square difference for each group between the less restrictive and more restrictive model was statistically significant. In EQS, this result is presented as the overall chi-square values of the two models against their summative degrees of freedom, in accordance with Bentler's (2006) recommendation. In this procedure, if the result is insignificant, the next step is to add another set of constraints followed by another test, with further steps taken until the result is significant. For our analysis, the parameters comprised, in order, factor loadings, factor coefficients, and factor covariances, but disturbance variances and error variances were not tested due to concern about the sub-group sample size. Model 4 and Model 7 were tested by means of the above method. The results of the multi-sample analyses for Model 4 and Model 7 are given in Table 4. All standardized residual covariances were small over both the male and female samples, showing that both models fit the data well. For Model 4, the analysis by gender revealed that the baseline model is acceptable (summative Y-B c2 = 4,814.65; df = 2,858; p = .00; NNFI = .904; CFI = .911; RMSEA = .046, CI = .044, .049). Further constraining on factor loadings of the items on their domains (summative DY-B c2 = 70.07; df = 56; p = .10; NNFI = .905; CFI = .910; RMSEA = .046, CI = .044, .048), as well as on factor covariance (summative DY-B c2 = 66.28; df = 55; p = .14; NNFI = .906; CFI = .910; RMSEA = .046, CI = .044, .048) between the domains showed insignificant Y-B c2 change and acceptable goodness of fit, indicating that Model 4 was equivalent across gender, in terms of model structure, factor loadings, and factor covariance. Regarding Model 7, which structures the measurement instrument items with 10 domains under the four overarching categories, the multi-sample analysis showed similar patterns, depicted in Table 4. The analysis by gender revealed that the baseline model had acceptable goodness of fit (summative Y-B c2 = 5,017.24; df = 2,940; p = .00; NNFI = .901; CFI = .905; RMSEA = .047, CI = .045, .049). Further constraining on factor loadings between items and domains (summative DY-B c2 = 54.49; df = 46; p = .18; NNFI = .902; CFI = .905; RMSEA = .047, CI = .045, .049), and factor coefficients between domains and their categories (summative DY-B c2 = 15.82; df = 9; p = .07; NNFI = .902; CFI = .905; RMSEA = .047, CI = .045, .049), as well as factor covariances between overarching categories (summative DY-B c2 = 1.57; df = 6; p = .95; NNFI = .902; CFI = .905; RMSEA = .047, CI = .045, .049) showed insignificant chi-square change with acceptable goodness of fit. The multi-sample analyses indicated that both Model 4 and 7 were stable across the sample by gender, with acceptable goodness of fit (NNFI and CFI at .90 or above; and RMSEA <.06). © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 17 Reliability Analysis Table 5 and 6 display the reliability results of the developmental outcome domains and overarching categories of the two models. These results indicate satisfactory reliability (see Lance, Butts, & Michels, 2006), with most Cronbach's alpha scores above .80 and a small number just below .80. This was the case for the entire scale (.981), for the four overarching categories (.866 to .957 for Model 7), for the 10 developmental outcome domains in Model 7 (.794 to .925) and for the 11 domains in Model 4 (.790 to .915). Selected Models and Summary Based on the above analyses, Model 4 and Model 7 were selected as potential final models, but with inclination toward Model 4 because of its lower AIC value. The final findings for Model 4 and Model 7, in terms of factor loadings, factor coefficients, factor correlations, and reliability indices are illustrated in Figure 1 and 2, and Table 5 and 6. All items in Model 4 and Model 7 were loaded on their designated domains and categories, except that for Model 7 the domain of sense of social responsibility was combined with that of community commitment and understanding. As a result, the constituent domains within the category of civic orientation and engagement distinguish interpersonal-level issues, i.e., caring and respect, from community-level issues. Although the structure of both models received confirmation, the high factor correlations and coefficients illustrated that a more parsimonious solution could be obtained (Brown, 2015). We will discuss this further in the next section. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 18 Table 4. Multi-sample Analysis Results Y-B c2 * df p NNFI CFI RMSEA RMSEA CI DY-B c2 Ddf Dp Average Absolute Standardized Residual Male Female Model 4: The 11-domain model without upper level categories Baseline model 4,814.65 2,858 .00 .904 .911 .046 .044, .049 N/A N/A N/A .038 .041 Factor loadings equivalent 4,884.72 2,914 .00 .905 .910 .046 .044, .048 70.07 56 .10 .055 .052 Factor covariances equivalent 4,951.00 2,969 .00 .906 .910 .046 .044, .048 66.28 55 .14 .054 .050 Model 7: The 10-domain model with upper level categories Baseline model 5,017.24 2,940 .00 .901 .905 .047 .045, .049 NA NA NA .049 .043 Factor loadings equivalent 5,071.73 2,986 .00 .902 .905 .047 .045, .049 54.49 46 .18 .070 .047 Factor coefficients equivalent 5,087.55 2,995 .00 .902 .905 .047 .045, .049 15.82 9 .07 .065 .048 Factor covariances equivalent 5,089.12 3,001 .00 .902 .905 .047 .045, .049 1.57 6 .95 .062 .048 Note: * Y-B c2 denotes the Yuan-Bentler scaled chi-square values with robust correction applied. The fit indices in the table are also adopting the version of robust correction. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 19 Figure 1. Standardized Item-Domain Factor Loadings* of Model 4 Notes: *All item-domain factor loadings are statistically significant at the .05 level. The path of the constant variable V999 to all measurement variables is not shown in the figure, for the sake of clarity. KA: Knowledge Application; RTS: Relationship & Team Skills; CPS: Creative Problem Solving Skills; SRS: Self-reflection Skills; CTS: Critical Thinking Skills; CCU: Community Commitment & Understanding: CR: Caring & Respect; SSR: Sense of Social Responsibility; SE: Self-efficacy; SU: Self-understanding; CSI: Commitment to Self-improvement Item 026 Item 019 Item 066 Item 061 Item 037 Item 008 Item 072 Item 087 Item 021 Item 077 Item 068 Item 051 Item 092 Item 100 Item 058 Item 030 Item 029 Item 048 Item 059 Item 044 Item 014 Item 093 Item 070 Item 073 Item 001 Item 022 Item 007 Item 091 Item 047 Item 045 Item 040 Item 101 Item 049 Item 094 Item 098 Item 046 Item 085 Item 079 Item 084 Item 005 Item 032 Item 067 Item 027 Item 023 Item 015 Item 050 Item 075 Item 060 Item 034 Item 010 Item 006 Item 063 Item 025 Item 069 Item 083 Item 090 KA RTS CPS .782 .769 .766 .636 .796 .795 .712 .658 SRS .768 .760 .719 .704 SE .793 .790 .780 .663 SU .842 .825 .783CSI .784 .759 .757CTS .749 .743 .726SSR .800 .797 .764 .761 .739 .695 .671 .632 .785 .778 .732 .722 .679 .674 .655 .614 CC U .801 .791 .788 .766 .754 .746 .680 .594 CR .781 .723 .718 .714 .711 .709 .594 For intercorrelation between learning domains, please refer to Table 5 © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 20 Table 5. Standardized Factor Correlations* and Reliability between Domains of Model 4 Domain** Reliability KA RTS CPS SRS CTS CCU CR SSR SE SU CSI KA .866 1.00 RTS .912 .859 1.00 CPS .909 .935 .970 1.00 SRS .818 .816 .955 .911 1.00 CTS .794 .771 .853 .831 .978 1.00 CCU .915 .806 .910 .888 .917 .865 1.00 CR .907 .790 .910 .891 .899 .823 .932 1.00 SSR .790 .822 .924 .917 .941 .883 .964 .962 1.00 SE .843 .807 .890 .898 .886 .828 .859 .924 .948 1.00 SU .805 .769 .854 .884 .897 .855 .887 .899 .943 .959 1.00 CSI .807 .742 .818 .824 .844 .840 .854 .825 .923 .854 .900 1.00 Notes: *All correlations are statistically significant at the .05 level **: KA: Knowledge Application; RTS: Relationship & Team Skills; CPS: Creative Problem Solving Skills; SRS: Self-reflection Skills; CTS: Critical Thinking Skills; CCU: Community Commitment & Understanding: CR: Caring & Respect; SSR: Sense of Social Responsibility; SE: Self-efficacy; SU: Self-understanding; CSI: Commitment to Self-improvement © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23258| January 2, 2020 21 Figure 2. Standardized Item-Domain Factor Loadings and Factor Correlations * of Model 7 Notes: * All item-domain factor loadings are statistically significant at the .05 level. The path of the constant variable V999 to all measurement variables is not shown in the figure, for the sake of clarity. For Domain: KA: Knowledge Application; RTS: Relationship & Team Skills; CPS: Creative Problem Solving Skills; SRS: Self-reflection Skills; CTS: Critical Thinking Skills; CCU&SSR: Community Commitment & Understanding, and Sense of Social Responsibility: CR: Caring & Respect; SE: Self-efficacy; SU: Self-understanding; CSI: Commitment to Self-improvement; For Category: KA: Knowledge Application; PPS: Personal & Professional Skills; COE: Civic Orientation & Engagement; SA: Self-awareness Item 026 Item 019 Item 066 Item 061 Item 037 Item 008 Item 072 Item 087 Item 021 Item 077 Item 068 Item 051 Item 092 Item 100 Item 058 Item 030 Item 029 Item 059 Item 048 Item 044 Item 093 Item 014 Item 070 Item 073 Item 001 Item 007 Item 022 Item 091 Item 045 Item 047 Item 027 Item 040 Item 101 Item 049 Item 023 Item 046 Item 085 Item 079 Item 084 Item 005 Item 032 Item 067 Item 015 Item 094 Item 098 Item 050 Item 075 Item 060 Item 034 Item 010 Item 006 Item 063 Item 025 KA RTS CPS .777 .760 .730 .617 .798 .786 .686 .658 SRS .773 .756 .700 .690 SE .792 .781 .770 .658 SU .842 .815 .779CSI .788 .754 .736CTS .698 .656 .585 .795 .793 .755 .749 .728 .686 .665 .626 .783 .773 .715 .711 .673 .660 .644 .611 CCU & SSR .778 .776 .773 .758 .754 .748 .739 .737 CR .774 .717 .716 .708 .708 .705 .573 KA PPS COE SA 1.00 .975 .973 .971 .900 .981 .963 .969 .976 .909 .888 .958 .952 .819 .932 .800 Item 069 Item 083 Item 090 © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 22 Table 6. Reliability of Model 7’s Domains & Overarching Categories Reliability Category Knowledge Application* .866 Personal & Professional Skills .957 Civic Orientation & Engagement .947 Self-awareness .920 Domain Knowledge Application .866 Relationship & Team Skills .912 Creative Problem Solving Skills .909 Self-reflection Skills .818 Critical Thinking Skills .794 Community Commitment & Understanding, & Sense of Social Responsibility .925 Caring & Respect .907 Self-efficacy .843 Self-understanding .805 Commitment of Self-improvement .807 Note: *The cognominal category was created above the domain "Knowledge Application" for the sake of providing a clear model structure Conclusion By using CFA with a relatively large sample, the current study sought to confirm the dimensionality and factor structure of S-LOMS that had been obtained through EFA in a previous study (Snell & Lau, 2020). Seven alternative models were specified and tested. The results indicated that an 11-domain model without overarching categories (Model 4) was the best fit, outperforming the single factor model (Model 1) and four-category level model (Model 2) in terms of the AIC values and goodness of fit indices. By contrast, the analysis indicated that both models that contained 15 developmental outcome domains (Model 3 and Model 5) could not fit the data well, because of ill-fitting patterns in factor correlations and coefficients between particular pairs of domains. Thus, in Model 3, there was a factor correlation of 1.0 between the domains of creativity and problem solving skills, and between the domains of commitment to social betterment and understanding community; while in Model 5 a factor coefficient of 1.0 was found between the domain of understanding community and its overarching category of civic orientation and engagement. The discovery of factor correlations or coefficients approaching 1.0 indicates that there may be more parsimonious model structures (Brown, 2015), and is consistent with the EFA results in the prior study (Snell & Lau, 2020). © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 23 Model 6, with a structure of 15 developmental outcome domains under four overarching categories was also rejected due to its factor coefficient of 1.0 between the domain of sense of social responsibility and its overarching category of civic orientation and engagement. Model 7 was therefore created based on a modification of Model 6, with the two domains subsumed under the overarching category of civic orientation and engagement. The first of these domains, a composite of community commitment and understanding and sense of social responsibility, reflects concern for societal level issues. The second domain, caring and respect, reflects interpersonal-level sensitivity. Acceptable goodness of fit was found between the data and Model 7, albeit with an AIC that was larger than for Model 4. Both the 11-domain model without overarching categories (Model 4) and the 10-domain model with four overarching categories (Model 7) were found to be invariant in terms of factor structure, factor loadings, factor coefficients, and factor correlations between male and female groups in the sample, indicating the stability of both models across gender (Schumacker & Lomax, 1996). The results, indicating preference for 11 over 15 developmental domains, confirmed the previous EFA findings of Snell and Lau (2020). Specifically in Model 4, creativity and problem solving skills were combined into creative problem solving skills; relationship skills and team skills were combined under relationship and team skills; community understanding and commitment to community were integrated under the community commitment and understanding; and empathy and caring for others along with respecting diversity were subsumed under caring and respect. The four overarching categories confirmed in Model 7 are consistent with typologies of the major developmental outcomes of service-learning in the past literature, which include academic enhancement, personal growth and civic learning (e.g. Driscoll et al., 1996; Elyer & Giles, 1999; Elyer et al., 2001; Felton & Clayton, 2011). Model 7 also includes self-awareness as an overarching category, which was created by Snell & Lau (2020) to capture the developmental outcomes associated with Confucian self-cultivation, which has influenced tertiary education policy in Hong Kong. At the developmental outcome domain level, Model 7 further reduces the number of domains from 11 to 10, by combining sense of social responsibility with community commitment and understanding. In summary, by comparing the 11- and 15-domain structure through CFA with a new sample, the current study confirmed that S-LOMS can be structured as an 11-domain model (Model 4) without an overarching category level, which was a better fit with the data than the alternative models. Nonetheless, the study also offered some support for a model with 10 developmental outcome domains under four overarching categories (Model 7), resembling the findings of past literature. Multi-sample analysis indicated that both Model 4 and Model 7 were stable across male and female groups in the current sample. Practical Implications Because of the satisfactory factor validity and internal consistency reported above, S-LOMS offers flexibility in how the developmental impacts on students engaging in service-learning can © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 24 be measured, with a number of options besides using the entire 56-item scale. For example, an instructor with a specific interest only in the two developmental outcome domains of critical thinking skills, which has three items, and creative problem-solving skills, which has eight items, need only use those 11 items for measurement, thereby streamlining the data collection process. Another example is that an investigator, who wishes to focus on measuring impact within the overarching category of civic orientation and engagement need only use the 18 associated items instead of the entire S-LOMS. Thus, S-LOMS can be administrated flexibly in accordance with instructors’ or researchers’ needs. Overall scores for any particular developmental outcome domain can be derived by averaging the scores of the associated items. It is assumed that investigators would adopt a pretest-posttest research design for measuring developmental impacts. Limitations and Further Studies The first limitation lies in the level of fitness of the models in the current study. Although both Model 4 and 7 achieved acceptable goodness of fit indices (i.e..90 or above for NNFI and CFI ), they did not meet the satisfactory level, which is .95 for NNFI and CFI indices (Hu & Bentler, 1999). Further studies should apply S-LOMS into more new samples to test whether consistently satisfactory goodness of fit indices can be obtained, and if so, discover what modifications are necessary in order to achieve this. The second limitation arises from the multivariate non- normality of the data from the current sample, resulting in bias over the ML methods in model estimation. Despite our attempt to apply corrections through Yuan-Bentler correction with the FIML method, other researchers have stated that better results can be achieved by adopting a two-stage robust method for non-normal missing data (e.g. Tong, Zhang, & Yuan, 2014), and further studies can consider adopting the latter approach. Third, the numerous factor correlations and factor coefficients exceeding .85 that were found for both Model 4 and Model 7 warrant attention. They imply poor discriminant validity (Brown, 2015) and may raise questions about the unique predictive validity of their individual factor. Further research is thus required into the predictive validity of S-LOMS's domains and categories. Further limitations, in the case of Model 7, concern the high factor coefficients between the 10 domains and their corresponding overarching categories, as well as the high factor correlations between the overarching categories. This phenomenon matches the observation by Snell and Lau (2020) that although the four overarching categories are conceptually distinct, they are empirically inter-related. The limitation of high factor correlations and coefficients suggests that S-LOMS may need further refinement, and that there is scope for testing a set of simplified models against data from new samples. Despite the above limitations, the current study has provided empirical evidence about the construct validity of S-LOMS, from which further validation work can be done. The next steps being undertaken include validating test-retest reliability over an interval of time, and testing © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 25 criterion validity by administering S-LOMS under a pretest-posttest design with service-learning as the intervention in between. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23258| January 2, 2020 26 Author Note This paper results from a cross-institutional project named “Cross-institutional Capacity Building for Service-Learning in Hong Kong Higher Education Institutions (PolyU4/T&L/16-19)” which aims to enhance and support the development of service-learning as an effective pedagogical strategy under the collaboration of Lingnan University, The Hong Kong Polytechnic University, Hong Kong Baptist University, and The Education University of Hong Kong. The project was launched in 2017 and has been funded by the University Grants Committee (UGC) of the Hong Kong Special Administrative Region (HKSAR) government. The authors wish to thank the UGC for funding the project, and the above institutions for their participation in the process of scale development and validation. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 27 References Astin, A. W., & Sax, L. J. (1998). How undergraduates are affected by service participation. Journal of College Student Development, 39 (3), 251–263. Retrieved from https://digitalcommons.unomaha.edu/slcehighered/7. Astin, A. W., Vogelgesang, L. J., Ikeda, E. K., & Yee, J. A. (2000). How service-learning affects students. Los Angeles, CA: Higher Education Research Institute, UCLA. Retrieved from https://digitalcommons.unomaha.edu/slcehighered/144. Bentler, P. M. (1990). Comparative fit indices in structural models. Psychological Bulletin, 107, 238–246. Bentler, P. M. (2006). EQS 6 structural equations program manual. Encino, CA: Multivariate Software, Inc. Blunch, N. J. (2016). Introduction to structural equation modeling using IBM SPSS Statistics and EQS. London: SAGE. Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). New York, NY: Guilford. Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods and Research, 21, 230-258. Byrne, B. M. (2008). Structural equation modeling with EQS: Basic concepts, applications, and programming (2nd ed.). New York: Psychology Press. Celio, C. I., Durlak, J., & Dymnicki, A. (2011). A meta-analysis of the impact of service- learning on students. Journal of Experiential Education, 34(2), 164–181. https://doi.org/10.5193/jee34.2.164. Conway, J. M., Amel, E. L., & Gerwien, D. P. (2009). Teaching and learning in the social context: A meta-analysis of service learning's effects on academic, personal, social, and citizenship outcomes. Teaching of Psychology, 36(4), 233–245. https://doi.org/10.1080/00986280903172969. Driscoll, A., Holland, B., Gelmon, S., & Kerrigan, S. (1996). An assessment model for service- learning: Comprehensive case studies of impact on faculty, students, community and institutions. Michigan Journal of Community Service Learning, 3, 66–71. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 28 Einfeld, A., & Collins, D. (2008). The relationships between service-learning, social justice, multicultural competence, and civic engagement. Journal of College Student Development, 49(2), 95–109. https://doi.org/10.1353/csd.2008.0017. Eyler, J. S., Giles, D. E., & Braxton, J. (1997). The impact of service-learning on college students. Michigan Journal of Community Service Learning, 4(1), 5–15. Retrieved from http://hdl.handle.net/2027/spo.3239521.0004.101. Eyler, J. S., & Giles, D. E. (1999). Where's the learning in service learning? San Francisco, CA: Jossey-Bass. Eyler, J. S., Giles, D. E., Stenson, C. M., and Gray, C. J., (2001). At a glance: What we know about the effects of service-learning on college students, faculty, institutions and communities, 1993–2000 (3rd ed.). Nashville, Tenn.: Vanderbilt University Press. Felten, P., & Clayton, P. H. (2011). Service-Learning. New Directions for Teaching and Learning, 128, 75–84. https://doi.org/10.1002/tl.470. Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modelling: guidelines for determining model fit. Electronic Journal of Business Research Methods, 6, 53–60. Hu, L.T. and Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55. Hurley, A., Scandura, T., Schriesheim, C., Brannick, M., Seers, A., Vandenberg, R., & Williams, L. (1997). Exploratory and confirmatory factor analysis: guidelines, issues, and alternatives. Journal of Organizational Behavior, 18(6), 667-683. Jacoby, B. (1996). Service-learning in higher education: concepts and practices. San Francisco: Jossey-Bass. Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria. Organizational Research Methods, 9(2), 202–220. https://doi.org/10.1177/1094428105284919. Lo, K. W. K., Kwan, K. P., Chan, S. C. F., & Ngai, G. (2016). Cross-cultural validation of the global citizenship scale for measuring impacts of international service-learning in Hong Kong setting. Paper presented at the 3rd International Conference on Service-Learning, Hong Kong. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 29 Lundy, B. L. (2007). Service learning in life-span developmental psychology: Higher exam scores and increased empathy. Teaching of Psychology, 34(1), 23–27. https://doi.org/10.1080/00986280709336644. Ma, C. (2018). Service-learning development in higher education in Hong Kong. In T. W. Lim, & W. X. Li (Eds.), Studying Hong Kong: 20 Years of Political, Economic and Social Developments (pp 43–61). New Jersey: World Scientific. https://doi.org/10.1142/9789813223554_0004. Ma, C. H. K., & Chan, A. C. M. (2013). A Hong Kong university first: Establishing service- learning as an academic credit-bearing subject. Gateways: International Journal of Community Research and Engagement, 6, 178–198. https://doi.org/10.5130/ijcre.v6i1.3286. Ma, H. K. C., Chan, C. W. F., Tse, P. H. (2019). A common outcome measurement for service- learning in Hong Kong. Journal of Higher Education Outreach and Engagement, 23(3), 3– 19. Ngai, S. S. (2009). The effects of program characteristics and psychological engagement on service-learning outcomes: A study of university students in Hong Kong. Adolescence, 44, 375–389. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/19764273. Novak, J. M., Markey, V., & Allen, M. (2007). Evaluating cognitive outcomes of service learning in higher education: A meta-analysis. Communication Research Reports, 24(2), 149-157. https://doi.org/10.1080/08824090701304881. Prentice, M. (2007). Service learning and civic engagement. Academic Questions, 20, 135–145. https://doi.org/10.1007/s12129-007-9005-y. Rama, D. V. (1998). Learning by doing: Concepts and models for service-learning in accounting. AAHE's series on service-learning in the disciplines. Washington, D.C.: American Association for Higher Education. Retrieved from https://eric.ed.gov/?id=ED449726. Richard, D., Keen, C., Hatcher, J. A., & Pease, H. A. (2017). Pathways to adult civic engagement: Benefits of reflection and dialogue across difference in higher education service-learning programs. Michigan Journal of Community Service Learning, 23(1), 60– 74. http://dx.doi.org/10.3998/mjcsloa.3239521.0023.105. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 30 Savalei V., & Bentler, P. M. (2005). A Statistically justified pairwise ml method for incomplete nonnormal data: a comparison with direct ML and pairwise ADF. Structural Equation Modeling, 12, 183–214. https://doi.org/10.1207/s15328007sem1202_1. Schumacker, R.E., & Lomax, R. G. (1996). A beginner’s guide to structural equation modeling. Mahwah, NJ: Lawrence Erlbaum Associates Inc. Shumer, R., Stanton, T. K., & Giles D. E., Jr. (2017). History and precursors of service-learning theory, development and research. In Robert Shumer (Ed.), Where’s the wisdom in service- learning? (pp. 1-32). NC: Information Age Publishing. Simons, L., & Cleary, B. (2006). The influence of service learning on students’ personal and social development. College Teaching, 54(4), 307–319. https://doi.org/10.3200/ctch.54.4.307-319. Siu, P. Y. C., Tang, H. H. E., & Lai, C. O. A. (2013). Teaching and learning global citizenship through service-learning: The success story of cross-border service-learning summer institute, Lingnan University, Hong Kong. Paper presented at 4th Asia-Pacific Regional Conference on Service-Learning: Service-Learning as a Bridge from Local to Global: Connected World, Connected Future. Hong Kong and Guangzhou, China. Retrieved from https://commons.ln.edu.hk/cgi/viewcontent.cgi?referer=https://scholar.google.com/&httpsre dir=1&article=1073&context=osl_conference. Snell, R. S., & Lau, K. H. (2020). The development of a service-learning outcomes measurement scale (s-loms). Metropolitan Universities, 31(1), 44-77. https://doi.org/10.18060/23258. Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.). New York: Routledge. https://doi.org/10.4324/9780203843130. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston: Allyn & Bacon. Tay, L., & Jebb, A. (2017). Scale creation. In S. Rogelberg (Ed.), The SAGE Encyclopaedia of Industrial and Organizational Psychology (2nd ed.) (pp. 1381–1383). Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781483386874.n242. Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington D. C.: American Psychological Association. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 31 Tong, X., Zhang, Z., & Yuan, K. H. (2014). Evaluation of test statistics for robust structural equation modeling with nonnormal missing data. Structural Equation Modeling: A Multidisciplinary Journal, 21(4), 553-565, https://doi.org/10.1080/10705511.2014.919820. Warren, J. L. (2012). Does service-learning increase student learning? A meta-analysis. Michigan Journal of Community Service Learning, 18(2), 56–61. Retrieved from http://hdl.handle.net/2027/spo.3239521.0018.205. Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: a content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838. https://doi.org/10.1177/0011000006288127. Xing, J., & Ma, C. (2010). Service-learning in Asia: Curricular models and practices. Hong Kong University Press. https://doi.org/10.5790/hongkong/9789888028467.001.0001. Yorio, P. L., & Ye, F. (2012). A Meta-analysis on the effects of service-learning on the social, personal, and cognitive outcomes of learning. Academy of Management Learning & Education, 11(1), 9–27. https://doi.org/10.5465/amle.2010.0072. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 32 Appendix 1. Items, Developmental Domains and Overarching Categories of S-LOMS Items No in scale Domain* Item Category: Knowledge Application 019 Knowledge Application I know how to transfer knowledge and skills from one setting to another. 026 Knowledge Application I can make connections between theory and practice. 061 Knowledge Application I am able to apply/integrate classroom knowledge to deal with complex issues. 066 Knowledge Application I know how to apply what I learn in class to solve real-life problems. Category: Personal and Professional Skills 008 Relationship Skills I am good at building relationships between people. 051 Relationship Skills I can easily establish effective relationships with people. 068 Relationship Skills I can build long-term relationships with people. 072 Relationship Skills I am good at keeping in touch with people. 021 Team Skills I have the necessary skills for making groups or organizations function effectively. 037 Team Skills I am good at resolving conflicts. 077 Team Skills I am confident in leading others toward common goals. 087 Team Skills I participate effectively in group discussions and activities. 030 Problem-solving Skills I am able to solve challenging real-life problems. 044 Problem-solving Skills I feel confident in dealing with a problem. 059 Problem-solving Skills I often modify my strategies to solve a problem when the situation changes. 100 Problem-solving Skills I feel confident in identifying the core of a problem. 029 Creativity I am not afraid of trying new things. 048 Creativity I am able to generate original ideas. 058 Creativity I am able to look at an issue from a fresh perspective. 092 Creativity When necessary, I can think of alternatives. 014 Self-reflection I always think how I can improve myself. 070 Self-reflection I will evaluate myself after completing a task. 073 Self-reflection I consider circumstances when reflecting on how well I have performed. 093 Self-reflection I reflect on myself regularly. 001 Critical Thinking Skills I can analyze an issue comprehensively. 007 Critical Thinking Skills I often look at complex issues from different angles. 022 Critical Thinking Skills I can understand others' viewpoints when we are making decisions together. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 33 Category: Civic Orientation and Engagement 040 Commitment to Social Betterment I think about how I can serve the community after graduating. 047 Commitment to Social Betterment I will play my part to reduce social problems. 049 Commitment to Social Betterment I always actively discuss possible improvements for our community. 091 Commitment to Social Betterment I will contribute my abilities to make the community a better place. 045 Understanding Community I can identify challenges in the community. 094 Understanding Community I can investigate the challenges faced by people in need in a community. 098 Understanding Community I can identify issues that are important for a disadvantaged community. 101 Understanding Community I can identify useful resources of a community. 005 Respecting Diversity I can respect people whose background is different from mine. 046 Respecting Diversity I am willing to try to understand people whose background is different from mine. 067 Respecting Diversity I respect the needs of people from different backgrounds. 084 Respecting Diversity I appreciate the ideas of people from different backgrounds. 032 Empathy and Caring for Others I observe others' feelings and emotions. 079 Empathy and Caring for Others I consider others' points of view. 085 Empathy and Caring for Others I care about others. 015 Sense of Social Responsibility I believe that taking care of people who are in need is everyone’s responsibility. 023 Sense of Social Responsibility I feel obligated to help those who are less fortunate than me. 027 Sense of Social Responsibility I believe that everybody should be encouraged to participate in civic affairs. Category: Self-awareness 034 Self-efficacy Most things I do, I do well. 050 Self-efficacy I have many good qualities. 060 Self-efficacy I am satisfied with my achievement so far. 075 Self-efficacy I am positive about myself. 006 Self-understanding I have a clear picture of what I am like as a person. 010 Self-understanding I know my strengths and weaknesses. © The Author 2020. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23920 | August 14, 2020 34 025 Self-understanding I have a clear understanding of my own values and principles. 063 Self-understanding I know what I need in my life. 069 Commitment to Self- improvement I am always motivated to learn. 083 Commitment to Self- improvement I always keep my knowledge and skills up-to-date. 090 Commitment to Self- improvement I look out for new skills or knowledge to acquire. Notes: * In Model 4, the following higher order domains were formed: 1) Creative Problem Solving Skills, by Creativity and Problem Solving Skills; 2) Relationship & Team Skills, by Relationship Skills & Team Skills; 3) Community Commitment & Understanding, by Commitment to Social Betterment and Understanding Community; and 4) Caring and Respect, by Empathy and Caring for Others and Respecting Diversity. In Model 7, an additional composite domain was formed, namely Community Commitment and Understanding and Sense of Social Responsibility, by combining Sense of Social Responsibility and Community Commitment & Understanding.