______________ Stacy M. Deck, PhD, MSSW, is an Associate Professor of Social Work at Spalding University, in Louisville, Kentucky. Phyllis A. Platt, PhD, MSW, is CEO of the Shawnee Christian Healthcare Center, Inc.in Louisville, Kentucky. Laneshia Conner, PhD, MSW, is an Assistant Professor in the School of Social Work at the University of North Carolina at Charlotte in Charlotte, North Carolina. Copyright © 2015 Advances in Social Work Vol. 16 No. 2 (Fall 2015), 233-248, DOI: 10.18060/18302 Engaged Teaching-Learning: Outcome Evaluation for Social Work Students in a Graduate-Level Service Learning Research Course Stacy M. Deck Phyllis A. Platt Laneshia McCord Abstract: The challenges of engaging social work students in the research curriculum are well-documented, and the literature supports the use of service learning to increase engagement. This study explores self-efficacy as one measure of learning outcomes. Changes in students’ (n=88) assessment of their ability to perform research and program evaluation tasks skillfully were measured by administering the Evaluation Self-Efficacy Scale (ESE) on the first and last days of a graduate-level advanced research class that included a service learning project. ESE scores on the last day of class were significantly higher than on the first day of class. The effect size was larger than in prior similar studies, suggesting that service learning contributed to students’ sense of mastery of course content. These results support the use of an engaged-learning model such as a service learning project in advanced social work research courses to improve students’ evaluation self-efficacy. Keywords: Service learning, engaged teaching, social work, research self-efficacy, evaluation self-efficacy The challenges of engaging students in the research curriculum are well-documented in the social work education literature (Harder, 2010; Hebert, Kulkin, & Ahn, 2014; Phillips, MacGiollaRi, & Callaghan, 2012; Wells, 2006). Because students commonly lack interest in research, and may fail to see connections between learning and applying research, it can be difficult to design courses that help students move beyond their reluctance (Lundahl, 2008; MacIntyre & Paul, 2013). On the other hand, students are more inclined to express interest in research when they find it to be relevant, useful in practice, congruent with their values and ethics, and collaborative (Gibbs & Stirling, 2013). As social work education has moved toward a competency-based approach to measuring program outcomes, the challenge of integrating research and practice has become even more pressing (Phillips, 2011). Focusing on student self-efficacy provides a theory-based outcome measure useful for monitoring students’ academic development. This study explores self-efficacy, not as a measure of knowledge, but as a measure of change in students’ perceptions of context-specific abilities as related to program evaluation and research. Evaluation Self-Efficacy Self-efficacy is a key component of Bandura’s social cognitive theory. According to Bandura, “self-efficacy is not a measure of the skills one has but a belief about what one can do… with whatever skills one possesses” (Bandura, 1997, p. 37). Lack of self-efficacy ADVANCES IN SOCIAL WORK, Fall 2015, 16(2) 234 can diminish an individual’s propensity to take action or to persevere in the context of challenging or complicated circumstances (Carpenter, 2011; Hebert et al., 2014; Quinney & Parker, 2010). Bandura maintains that self-efficacy is influenced by four types of experiences: (a) mastery, (b) vicarious experiences, (c) social persuasion, and (d) physiological states (Bandura, 1986). Successful experiences (mastery) contribute to increased self-efficacy as does observing others’ success. Modeling (i.e., providing vicarious experiences) offers an opportunity to engage in activities that enhance self- efficacy, especially in situations where students have had few prior opportunities to engage in the task. According to Bandura, modeling may be particularly useful with those who are mostly unsure of themselves, which is certainly the case for many social work students when it comes to research. Students’ lack of experience with and reluctance to engage in research activities often result in low self-efficacy for this professional competency. The third factor that affects self-efficacy, social persuasion, is the encouragement or discouragement received from others. Physiological factors (or more specifically, one’s interpretation of physiological responses in stressful situations) also impact self-efficacy (Bandura, 1997). In this theoretical framework, a student’s anxiety and related feelings about math or statistics, and the ensuing butterflies, could be interpreted as a sign of lacking adequate ability and, therefore, decreased self-efficacy. This belief in one’s efficacy is unlike self-esteem and confidence. Self-esteem is related to one’s sense of self-worth. Confidence is a less well-defined concept, and refers to the strength of one’s belief or how certain one is. Bandura explained that “perceived self-efficacy refers to belief in one's agentive capabilities that one can produce given levels of attainment. A self -efficacy assessment, therefore, includes both an affirmation of a capability level and the strength of that belief” (Bandura, 1997, p. 382). In other words, confidence is necessary, but not sufficient. To have self-efficacy, a person must also possess personal agency (i.e., affirm his or her own skill or ability to perform at a particular level). Social work educators have found self-efficacy to be a relevant measure of educational outcomes in social work research courses. Holden, Barker, Meenaghan, and Rosenberg (1999) described their Research Self-Efficacy Scale (RSE) as offering a means for assessing the impact of social work instruction in this area. Unrau and Grinnell (2005) subsequently used the RSE in a study of social work educational outcomes. They found that undergraduate and graduate students’ research self-efficacy increased during a semester in which they were enrolled in a required research course and that those who began with lower levels achieved larger gains. A review of the literature also supports the use of service learning as a teaching method to increase academic self-efficacy. Service Learning Service learning is a pedagogical approach that offers a systematic structure for linking classroom instruction and service experiences (Rhoades, 1998). In higher education, academic service learning takes on a variety of forms and is integrated into course work in multiple formats. In social work education, volunteering, and field practicum experiences often have been classified as service learning (Harder, 2010). Yet, for purposes of this study, service learning is more distinctly defined as “intentional learning (both curricular Deck et al./ENGAGED TEACHING-LEARNING 235 and civic) because of its emphasis on the reciprocal relationship between students, their instructors, and community members” (Phillips, 2011, p. 2). The use of service learning must provide opportunities that meet both academic and community needs (Campbell, 2012; Lucas, 2000; Wells, 2006). In their meta-analysis, Yorio and Ye (2012) found that students who participate in service learning generally “demonstrate a more positive understanding of social issues, alter their personal insight, and experience gains in cognitive development” (p. 25). Similarly, Cronley, Madden, Davis, and Preble (2014) found that social work educators’ use of service learning is predicated on a belief that it positively impacts students’ personal development and their ability to resolve moral and ethical dilemmas. Yet, surprisingly, educators who used service learning also tended not to agree with the idea that it increases students’ ability to apply what they have learned (Cronley et al., 2014). Wells, Maschi, and Slater (2012) found that among three approaches to integrating research and practice, the problem-solving approach—which included service learning activities—was least frequently described by the social work research faculty they surveyed. Similarly, an examination of research education in the United Kingdom found that while faculty were aware that students value experiential learning and hands-on experience, this method was not commonly used (MacIntyre & Paul, 2013). Self-efficacy has been identified as a “motivational outcome,” which suggests that building confidence in one’s ability to complete a difficult task increases the likelihood that one will engage in that behavior in the future (Carpenter, 2011, p. 128). Self-efficacy takes the concept of confidence further, however, to include “an affirmation of a capability level” (Bandura, 1997, p. 382). This requires knowing through assessment that one’s ability to do something is at a high enough level to be successful. Harder (2010) found that real- life experiences gained through service learning increased students’ motivation to engage in research by heightening their perception of its relevance to practice. Thus, it is plausible that as students’ self-efficacy increases, their likelihood to engage in research activities as part of their future practice also increases. The literature related to socializing students into the profession by helping them to attain and use professional knowledge and expertise in general (Wilson & Kelly, 2010), as well as to research- or evaluation-related self-efficacy specifically (Holden, Barker, Rosenberg, & Onghena, 2007), is limited. One literature review of published studies on social work education outcomes measured over time using quantitative or mixed methods identified only 31 English-language articles published between 2004 and 2010 and concluded that “rigorous evaluation of the outcomes of social work education is still at an early stage of development” (Carpenter, 2011, p. 137). Even as the body of evidence is growing for the use of service learning in social work, some argue that more research is needed to assess its “goodness of fit” as an effective teaching-learning approach (Charles, Alexander, & Oliver, 2014; Lemieux & Allen, 2007; Phillips, 2011). The purpose of this study was to explore the following research questions: What are the effects of a service learning approach to teaching social work research and program evaluation methods? Specifically, does students’ post-course evaluation self-efficacy differ from their pre- course evaluation self-efficacy? ADVANCES IN SOCIAL WORK, Fall 2015, 16(2) 236 Methods Study Design This study used a one group, pretest-posttest design. Pre- and post-course data were collected by three faculty members from four consecutive cohorts of students who were enrolled in graduate-level advanced research courses between 2010 and 2013. The course is required for the Master of Social Work (MSW) degree and is offered in the concentration year. In the first three years, it was offered as a traditional face-to-face course that was enhanced with a service learning component of community-based practice (a program evaluation project for a local non-profit agency). In 2013, the course was offered in a hybrid-learning format that included 24 hours of face-to-face class time, approximately 16 hours of online learning, and the service learning component. Sample During the four academic years in which data were collected, 106 students were enrolled in the advanced research course. Of these, 88 submitted pre- and post-course data collection forms that could be linked using a matching identification number and were included in the study. To ensure anonymity, demographic information was not collected from study participants. However, program-level administrative data indicate that 26.1% of MSW students enrolled during the most recent year of the study were under age 25, 39.1% were between the ages of 25 and 34, 52.2% were between the ages of 35 and 44, and 17.4% were over age 45. Most were female (82.6%). Half of the students (50.0%) were African American or another racial/ethnic minority. About two-thirds (64.8%) of the students in the study sample reported that they had entered the graduate program with an undergraduate degree in social work. Of these 57 advanced standing students, 51 provided information about how long it had been since their last research course. The timeframe ranged from less than one year to 14 years. While the most frequent response was one year (15 participants), the median elapsed time was three years. The remaining students were in their second year of a two-year program and had completed their first graduate-level research course in the prior year. Data Collection Data were initially collected as part of the curriculum evaluation plan for Spalding University School of Social Work’s continuing accreditation through the Council on Social Work Education (CSWE). The original purpose for collecting these data was to assess the degree to which course objectives were achieved and to evaluate the effectiveness of the engaged-learning strategy (service learning approach) used in the course. In the current study, the investigators conducted a secondary analysis of the existing data. The university’s Research Ethics Committee approved the study methods. All students enrolled in the research course completed a print version of the data collection instrument during a face-to-face class meeting on the first and last days of the course. To protect students’ identity, promote greater willingness to disclose perceptions of their own ability, and decrease the potential for social desirability bias, students were Deck et al./ENGAGED TEACHING-LEARNING 237 asked not to write their names or any identifying information on the data collection forms. To match pre- and post-course data, students created and recorded a unique identification number on their forms. These identification numbers were not linked to student names, making it impossible for instructors to connect students with their responses. Variables/Measures In the study reported here, evaluation self-efficacy was the dependent variable. Changes in students’ confidence in their own ability to perform specific social work tasks related to research and program evaluation (evaluation self-efficacy) were measured by administering the Evaluation Self-Efficacy Scale (Holden et al., 2007; Holden, Barker, Rosenberg, & Onghena, 2008) on the first and last days of class. The Evaluation Self- Efficacy Scale (ESE) is an 11-item self-report scale that measures students’ confidence in their ability to perform tasks related to social work program and practice evaluation (Holden et al., 2007; 2008). Items include skills such as “search electronic databases and the World Wide Web to obtain the scholarly literature necessary to design your evaluation of practice,” “design a measurement approach for your evaluation of some aspect of practice,” and “carry out all the elements of your evaluation in a practice setting” (Holden et al., 2007; 2008). Respondents rated each item on a Likert-type scale in 10-point increments reflecting level of confidence in their own ability from 0 (cannot do at all) to 100 (certain can do). Approximately 10 minutes are required for participants to complete the assessment. To score the instrument, responses are summed and divided by 11 (the total number of items). In the current study, a summed score was only calculated for students who rated all 11 scale items. Only those students with a matched pair of pre- and post- course scale scores were included in the study sample. The psychometric properties of the ESE have been evaluated. Holden et al. (2007) presented evidence of its reliability (Cronbach’s alphas of 0.94 at pretest and 0.96 at posttest), content validity, construct validity (demonstrated by establishing a correlation between the scale scores of the ESE and the Social Worker Empowerment Scale), and sensitivity to differences in students’ evaluation-related self-efficacy and to changes in self- efficacy over time. The same authors replicated the original validation study, again demonstrating its content and construct validity, and reporting Cronbach’s alphas of 0.95 at both pretest and posttest (Holden et al., 2008). Cronbach’s alpha for the current study was 0.95 at pretest and posttest. To contextualize their ESE responses, students were asked to report whether they had entered the program with advanced standing, and if so, how long it had been since they had taken their last research course. As noted above, students in the second year of a two-year program had completed their last research course in the prior year. Data collection forms were also coded with the academic year that the survey was completed (four categories) and an instructor identification variable (three categories). The teaching-learning intervention (i.e., research course using a service learning approach) represents the independent variable of interest. Course content for this graduate- level advanced research course focuses on evaluating service delivery in social work practice. The goal is for students to develop a critical appreciation of and the ability to ADVANCES IN SOCIAL WORK, Fall 2015, 16(2) 238 conduct research and program evaluation. In all four years, course sessions (whether face- to-face or online) included the following topics: (a) ethical issues in program evaluation, (b) using logic models to describe programs and their theory of change, (c) needs/assets assessment and formative/process evaluation, (d) client satisfaction and cost effectiveness, (e) qualitative/mixed methods in evaluation, (f) measurement tools and strategies, (g) inferential statistics, (h) data analysis and interpretation, and (i) writing and presenting evaluation reports. Across all class sessions, a portion of class time was used for a statistics lab, in which students were engaged in a faculty-guided, hands-on process of analyzing and interpreting increasingly complex sets of data. For this course, Microsoft Excel was selected as the software package for data management and analysis because this is a resource that students are likely to have in their subsequent professional careers. In partial fulfilment of course requirements, class members worked together to conduct a pro bono program evaluation for a local non-profit agency, a day shelter for homeless men (i.e., the service learning project). Research faculty guided students through the phases of evaluation design, data collection, data entry, analysis, and reporting. The project began with a visit to the service learning partner agency for a presentation on the agency’s services and program evaluation needs and culminated with student presentations of results and recommendations to the agency’s executive director and board of directors on the last day of class. This service learning project was intended as an engaged-learning strategy to help students gain skills and knowledge for determining the effectiveness of practice and the critical use of research in social work decision-making. A desired outcome of the course was that students would view research as a core component of ethical practice and gain a set of tools to use in their social work careers. Each student contributed to the service learning program evaluation project by completing a 30- to 60-minute participant observation at the non-profit agency using ethnographic methods and completing five structured, one-to-one interviews with clients. Students entered interview data into a password-protected, online data collection site, and faculty exported and compiled this data into a single Excel file. Small groups of four to six students were assigned a topic area that related to a subset of the interview questions (e.g., education and work history, health/mental health, substance use, trauma, and utilization of/satisfaction with services). These small learning groups were coached by the faculty as they completed a literature review, and analyzed, interpreted, and reported on data from their assigned interview questions. Throughout the project, course assignments and class discussions supported students in gaining skills needed to complete the project as well as reflecting on what they were experiencing and learning. Data Analysis Paired samples t-tests were used to compare mean ESE scale scores at the pre- and post-course points in time. Because an exploratory, pre-experimental design was used in this study, it was impossible to conclusively establish a causal relationship between the independent variable (the service learning course) and the dependent variable (the post- course ESE score) or to rule out alternative explanations. However, to explore potential relationships between post-course outcomes and other plausible influences, the pairwise relationships between post-course ESE score and the contextual variables (years elapsed Deck et al./ENGAGED TEACHING-LEARNING 239 since last research course, instructor, and course year) were tested using ANOVA and regression analyses. Last, to compare the mean post-course ESE scale score in this study to post-course scores in prior, similar studies, one-sample t-tests were conducted using the mean posttest scores in other studies as the test values. When results of t-tests were statistically significant, effect size was also calculated using the formula for Cohen’s d. Data were analyzed using IBM SPSS 19 software. Results Results are presented in three sections: (a) differences in pre- and post-course evaluation self-efficacy, (b) exploration of potential alternative explanations, and (c) comparison of post-course evaluation self-efficacy in this and other study samples. Changes in Evaluation Self-Efficacy Consistent with Cronbach’s observation that instructional outcomes are multidimensional (as cited in Holden et al., 2007 and Holden et al., 2008) and therefore should be considered separately as well as in aggregate form, Table 1 presents a summary of responses to the ESE scale items at the beginning and end of the course. Item ratings can range from 0 to 100. On the first day of class, there was substantial variation in students’ confidence ratings for the scale items. On average, students felt most confident that they could successfully search electronic databases for scholarly literature in a manner that a social work supervisor would consider excellent; the mean pre-course rating for this item was 80.11 (SD=20.71). They felt least confident in their ability to design an inferential data analysis for evaluation of some aspect of practice (M=33.2, SD=25.17). On the last day of class, these two items remained the areas with the highest and lowest average ratings (M=92.3, SD=9.31 and M=75.2, SD=18.51 respectively). However, by the end of the course, mean ratings had increased for each of the 11 items. These increases ranged from 12.2 points (search electronic databases) to 42.05 points (design an inferential data analysis). Importantly, the mean rating for the item carry out all elements of evaluation in a practice setting increased by 38.9 points (from a mean of 45.3 at the beginning of the course to 84.3 at the end). The amount of variability in ratings was also lower at the end of the course for all ESE items. There was also substantial variation in pre- and post-course ESE scale scores. On the first day of class, individual ESE scale scores ranged from 4.6 to 93.6. On the last day of class, scores ranged from 38.2 to 100.0. The paired differences in pre- and post-course ESE scale scores were normally distributed. Paired samples t-tests were used to compare mean ESE scale scores at these two points in time. ESE scale scores on the last day of class (M=82.3, SD=12.44) were higher than on the first day of class (M=48.1, SD=20.84), t(87)=- 17.2, p<.001, d=1.83. Given the large effect size (Cohen’s d=1.83), the increase in evaluation self-efficacy was both statistically significant and practically important. ADVANCES IN SOCIAL WORK, Fall 2015, 16(2) 240 Table 1. ESE Item Ratings and Scale Scores on the First and Last Days of Class (n=88) Variable Pre-Course Post-Course Pre-Post Change M SD M SD M Search electronic databases and web for scholarly literature 80.1 20.71 92.3 9.31 12.2 Review a particular area of social science theory/ research and write a balanced, comprehensive lit review 60.8 22.4 86.5 13.16 25.7 Design a program implementation study 41.3 23.33 78.8 13.96 37.5 Create a single system design 48.5 28.27 81.4 15.84 32.8 Create a group research design 43.0 26.27 79.7 15.04 36.7 Design a sampling strategy 46.0 26.5 80.6 16.36 34.6 Design a measurement approach 43.5 25.28 80.9 15.13 37.4 Design a descriptive data analysis 37.1 24.92 79.1 16.99 42.0 Design an inferential data analysis 33.2 25.17 75.2 18.51 42.1 Design an evaluation of practice that incorporates social work values and ethics 49.9 27.66 86.9 15.11 37.1 Carry out all elements of evaluation in a practice setting 45.3 27.62 84.3 15.91 38.9 Average of all items (scale score) 48.1 20.84 82.3 12.44 34.3 Exploration of Potential Alternative Explanations An analysis of variance showed that the effect of academic year on post-course ESE scale scores was non-significant, suggesting that the traditional and hybrid course models were equally effective. Similarly, analysis of variance showed that the effect of the instructor on post-course ESE scores was non-significant. Finally, for advanced standing students, regression analysis was used to test whether the number of years elapsed since last research course was independently associated with the post-course ESE score after controlling for the pre-course ESE score. The two predictors explained 29% of the variance in post-course ESE score, R2=.29, F(2, 48)=9.85, p<.001. However, after accounting for an expected statistically significant effect of the pre-course ESE score (B=.55, p<.001), time since last research course did not independently predict post-course ESE scores. In summary, in this population, academic year, instructor, and time elapsed since last research class were not associated with the post-course ESE score, suggesting that these potential alternative explanations were unlikely. Comparison of Study Results With Other Study Samples This study replicates two prior studies that used the ESE to measure social work educational outcomes of advanced research courses focused on program and practice evaluation (Holden et al., 2007; 2008). In the first study of 85 Master's-level students, Holden et al. (2007) reported that students’ mean ESE score increased from 44.8 near the beginning of the 14-session course to 68.3 at the end of the course. In a follow-up study of 111 Masters-level students enrolled in the same course in a subsequent year, Holden et al. Deck et al./ENGAGED TEACHING-LEARNING 241 (2008) reported a change in the mean ESE score from 44.1 at pretest to 73.1 at posttest. In the current study, the mean scale score at the pre-course point in time was comparable to the mean pretest scale scores in the studies conducted by Holden and colleagues; the differences were non-significant. However, the mean post-course scale score in the current study was over 14 points higher than the posttest mean in the first study conducted by Holden and colleagues, and over nine points higher than the posttest mean in their replication study. One-sample t-tests using the mean posttest scores in the Holden studies as the test values indicated that these differences were each statistically significant and practically important, (t(87)=10.58, p<.001, d=1.13 and t(87)=6.96, p<.001, d=0.74, respectively). Discussion The importance of self-efficacy for graduate level social work students should not be underestimated. As previously mentioned, the belief students have about their capabilities to perform practice-specific tasks is affected by their behavior and motivation (Bandura, 1986). As observed by these authors, using an agency-related research project provided the platform for a facilitative process of learning to occur. Through this process of practice, students were conditioned to use a set of skills to engage in practice-informed research. As evidenced by the increase in Evaluation Self-Efficacy Scale (ESE) scores, students’ self- efficacy was strengthened. The learning environment provided the opportunity for students to credit their successes (i.e., performing primary data collection, analyzing the data, reporting to constituents) to their own performance, exemplifying performance mastery. The results support the psychometric quality of the ESE, as well as the theoretical basis for using this measure to assess the change in self-efficacy of student learners. As in prior studies, students’ self-efficacy related to research and program evaluation increased from the beginning to the end of the course. Importantly, the effect was larger in the current study than in prior, similar studies, and it is plausible that the use of an engaged-learning strategy (the service learning project) contributed to this outcome. This exploratory study provides initial support for the use of service learning as a teaching method, suggesting that it is a good fit for increasing self-efficacy through mastery, modeling, and social persuasion. As an example of course activities designed to promote mastery through direct experience, students presented their findings to the service learning partner agency’s executive director and board of directors at the end of the course. Their report provided a tangible community benefit, and it also allowed students to participate in a verbal exchange that offered feedback to them about their work, identified them as valuable human resources, and generated new energy regarding the benefits of applied social work research for all involved. A great challenge in social work education is transferring what is learned to practice. As a fervent attempt in “making learning purposeful” (Wells, 2006, p. 401) and supporting students’ work to acquire practical skills, the level of involvement with such a project can be seen as supportive for modeling career commitment, providing students the opportunity ADVANCES IN SOCIAL WORK, Fall 2015, 16(2) 242 to see the real and practical effects of their evaluation work on service delivery systems and clients. Through modeling (using the cycle of action and reflection) students were encouraged to develop a deeper understanding of their skills and capacity, which can lead to positive impacts on learning and personal outcomes for adult learners (Knowles, Holton, & Swanson, 2005). As a result of the collaborative nature of the project over time, students were able to interact with actual consumers of social services and to work in groups, which is reflective of real research-informed practice. Given this experience and the aforementioned fact that students may have lacked prior opportunities to actively learn and practice these skills, the gains made by the end of the course are yet another way to conceptualize a positive development in their learning (Chang et al., 2014). Finally, the experience of social persuasion through working in small groups can foster leadership skills and academic learning, another benefit of engagement that positively affects self-efficacy (National Youth Leadership Council, 2007). Students in this project typically identified which members of their project group possessed particular skills related to the program evaluation project (e.g., data analysis, critical thinking/interpretation of results, professional writing, public speaking, group facilitation, etc.). Teams had the freedom to assign group roles as they chose. This resulted in students being able to play to their strengths and to learn from one another in areas where skills were still developing. Course assignments provided opportunities for students to actively reflect on and evaluate each group member’s contributions as well as the overall experience of working in a group. The role of faculty as coaches was also critical to the success of the project. In summary, using service learning as an engaged-learning strategy in this course created opportunities to intentionally use mastery, modelling and social persuasion in order to build students’ confidence in their own abilities to perform skills related to program evaluation in social work settings. This resulted in a measurable change in students’ self- efficacy. Self-efficacy theory suggests that students who are confident as well as competent will be more motivated—and therefore more likely—to engage in research and program evaluation activities in their future professional practice (Carpenter, 2011; Harder, 2010; Hebert et al., 2014). The effectiveness of service learning is supported by the results of this study. As a familiar challenge in developing social work curricula, the task of training students on research and evaluation methods calls into question ideological differences regarding research and instructional delivery methods (Gibbs & Stirling, 2013; MacIntyre & Paul, 2013; Wells et al., 2012). In short, there is not one best way to teach students about this process. It is a worthy goal, however, to streamline the task, provide insight on what is demonstrated to be effective, and increase awareness that research improves the quality of practice (Chilvers, Maidment, & Crichton-Hill, 2012). Common to academicians and researchers is an understanding that most work is done in groups focused on integrating learning and generating knowledge. Through service learning projects such as this one, students can be shepherded into that same realm of knowledge generation within a social service setting in their community. In keeping with Bandura’s theory (1977) that learned behavior results from a social process of observation, Deck et al./ENGAGED TEACHING-LEARNING 243 imitation, and modeling, service learning is a complement to other experiential learning activities. In a major shift to support students as they become consumers of evidence, experiential activities in social work training are increasingly important (Chilvers et al., 2012). Integrating real-world experiences with classroom training improves students’ ability to understand complex and ambiguous circumstances, increases their interpersonal development and social responsibility, helps them to connect with professionals for career opportunities, and importantly, enhances community relations (Vanderbilt University Center for Teaching, n.d.). Additionally, active reflection on the nature of service learning assignments adds a layer of meaning to the process of community engagement. Limitations Although promising, these results should be interpreted with caution. A convenience sample was used from a single school over four academic years. While the results of this study are not generalizable beyond the sample population, this replication of results from prior studies contributes to the body of literature related to research and evaluation self- efficacy in an important way. It is encouraging that across study populations and sites, these studies have yielded very similar results: an overall improvement in self-efficacy. A second limitation is the use of self-reported data in this study. It should be acknowledged that participants in this and other similar studies could have responded in socially desirable ways and may have had different levels of awareness as it relates to perceptions of their own ability (Pedrazza, Trifiletti, Berlanda, & Di Bernardo, 2013). However, while there are noted issues with self-reports, social science research embraces the use of this type of data. Students are one source of credible information about their own abilities. Further, this study was intended to measure perceptions of confidence in their abilities, not the underlying abilities, and students arguably are in the best position to rate their own confidence. As such, this was a study of one element in the multidimensional construct of learning. To explore the relationship between confidence and competence, future studies could integrate other sources of data including independent faculty or field instructor assessments of students’ knowledge and ability in the areas of research and program evaluation. Gibbs and Stirling (2013) emphasize that learning new knowledge and skills does not necessarily translate into their regular use in professional practice, so ongoing training and support are necessary. A longitudinal study could explore outcomes over time to objectively assess whether a student’s level of self-efficacy while enrolled in a graduate-level social work program is predictive of actual use of relevant skills in subsequent practice. Finally, this measure could be combined with other measures of social work education outcomes to provide a multifaceted appraisal of learning. A third limitation is the use of a one-group, pretest-posttest design. While the findings of this study are consistent with a conclusion that the course was associated with changes in student self-efficacy, causation cannot be definitively inferred. Carpenter (2011) notes that observed changes also could be attributed to (a) maturation effects, (b) Hawthorne effects, and (c) the effects of testing. While common in the literature on social work education outcomes, simple pre-post designs cannot rule out these threats to validity. In replication, future studies should consider comparing the reported results with a control group comprised of comparable students exposed to a more traditional teaching strategy ADVANCES IN SOCIAL WORK, Fall 2015, 16(2) 244 and should control for other maturational factors that might account for the observed changes. Yet, this design is sufficient to establish that change in the desired direction did occur from the beginning to end of the course. Further, some have suggested the utility of self-efficacy measures as diagnostic instruments (for formative assessment, adjustment of course content, and identification of tailored supports) and for periodic student self- assessment in combination with identification of goals and activities to strengthen confidence (Quinney & Parker, 2010). Finally, comparing results of this study to results of prior studies conducted at a different school suggests the possibility of using a between- program comparison group design. While the groups were not tested for similarity on any characteristics other than mean pretest scores in this study, future studies could do so. A fourth limitation is that much of what was captured in this study and other related studies of self-efficacy focuses on personal agency. This course, however, also illustrated the interdependent nature of research through working in collaborative groups. What was not measured in this study is the aggregate ability of a group to achieve an outcome, or the notion of collective efficacy (Bandura, 2000; 2006). It would be important in future studies to separate how individuals appraise their personal capabilities from perceptions of the group’s capability to operate as a whole (Bandura, 2006). While group and individual outcomes may be similar for some members of the group (e.g., dominant or highest functioning members), it is nonetheless important to consider the possibility of within- and between-group differences because group functioning is based upon its membership. A final limitation is that self-efficacy was treated as a unidimensional construct in this study. Future studies should explore self-efficacy as a multidimensional construct. Previous research has shown that there are elements of self-efficacy that should be considered when operationalizing and measuring a belief in one’s ability to carry out tasks as well as the related beliefs needed to produce the desired outcome (i.e., Bandura, 1997; Choi, Fuqua, & Griffin, 2001; Miller, Coombs, & Fuqua, 1999). This study also may not have captured other factors that relate to students’ future use of research and evaluation skills in practice. As an example, it is likely that students must be convinced that evidence-based practice is important and effective as well as having confidence in their own skills. Incorporating an additional measure such as the Gregory Research Beliefs Scale (Gregory, 2010), a validated instrument that measures social work students’ beliefs related to the value of research for practice, would complement the measurement of confidence in one's ability and broaden the assessment of outcomes related to research courses. Conclusion It has long been recognized that students need to make connections between knowledge and its application in order to enhance their adult learning experience (Knowles et al., 2005). Today’s adult learners depend heavily upon course delivery formats that use innovative teaching-learning strategies such as service learning to simultaneously provide career experience and skills training. Through community involvement, students gain an understanding of community issues and processes related to change. Deck et al./ENGAGED TEACHING-LEARNING 245 Because service learning is intended to help students develop both competence and confidence, it is equally important for social work programs to assess students’ level of skill as well as their belief that they are able to practice these skills. Specifically, students’ confidence in their own ability to perform social work tasks related to research and program evaluation can be assessed by using the ESE to measure change in self-efficacy. As other studies have investigated how to best assess the effects of self-efficacy on learning and performance in various domains (Chang et al., 2014; Luzzo, Hasper, Albert, Bibby, & Martinelli, 1999; Shively, Smith, Bormann, & Gifford, 2002), this study contributes to the social work literature in this regard. The results of this study provide initial evidence that using an engaged-learning model such as service learning improves students’ self-efficacy. Our hope is that as students’ self-efficacy increases, the likelihood of them engaging in research activities as part of their future practice will also increase. References Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191-215. doi: http://dx.doi.org/10.1037/0033- 295X.84.2.191 Bandura, A. (1986). The explanatory and predictive scope of self-efficacy theory. Journal of Clinical and Social Psychology, 4, 359-373. doi: http://dx.doi.org/10.1521/jscp.1986.4.3.359 Bandura, A. (1997). Self-efficacy: The exercise of control. New York: W. H. Freeman. Bandura, A. (2000). Exercise of human agency through collective efficacy. Current Directions in Psychological Science, 9, 75-78. doi: http://dx.doi.org/10.1111/1467- 8721.00064 Bandura, A. (2006). Guide for constructing self-efficacy scales. In F. Pajares, & T. Urdan (Eds.), Self-efficacy beliefs of adolescents (Vol. 5, pp. 307-337). Greenwich, CT: Information Age Publishing. Campbell, E. (2012). Implementing service learning into a graduate social work course: A step-by-step guide. Journal of Teaching in Social Work, 32, 300-313. doi: http://dx.doi.org/10.1080/08841233.2012.688935 Carpenter, J. (2011). Evaluating social work education: A review of outcomes, measures, research designs and practicalities. Social Work Education: The International Journal, 30(2), 122-140. doi: http://dx.doi.org/10.1080/02615479.2011.540375 Chang, C., Liu, E., Sung, H., Lin, C., Chen, N., & Cheng, S. (2014). Effects of online college student’s Internet self-efficacy on learning motivation and performance. Innovations in Education and Teaching International, 51(4), 366-377. doi: http://dx.doi.org/10.1080/14703297.2013.771429 Charles, G., Alexander, C., & Oliver, C. (2014). Overcoming isolation: Making the case for the development of blended service learning and social work interprofessional field education experiences to improve university-community engagement. Currents: Scholarship in the Human Services, 13(1), 1-17. http://dx.doi.org/10.1037/0033-295X.84.2.191 http://dx.doi.org/10.1037/0033-295X.84.2.191 http://dx.doi.org/10.1521/jscp.1986.4.3.359 http://dx.doi.org/10.1111/1467-8721.00064 http://dx.doi.org/10.1111/1467-8721.00064 http://dx.doi.org/10.1080/08841233.2012.688935 http://dx.doi.org/10.1080/02615479.2011.540375 http://dx.doi.org/10.1080/14703297.2013.771429 ADVANCES IN SOCIAL WORK, Fall 2015, 16(2) 246 Chilvers, D., Maidment, J., & Crichton-Hill, Y. (2012). Promoting learning and teaching about research through collaborative work integrated learning: Implications for students, agencies and practitioners. Paper presented at Australian Collaborative Education Network National Conference. Retrieved from http://acen.edu.au/2012conference/wp-content/uploads/2012/11/50_Promoting- learning-and-teaching-about-research-through-collaborative-work-integrated- learning.pdf Choi, N., Fuqua, D., & Griffin, B. (2001). Exploratory analysis of the structure of scores from the multidimensional scales of perceived self-efficacy. Educational and Psychological Measurement, 61(3), 475-489. doi: http://dx.doi.org/10.1177/00131640121971338 Cronley, C., Madden, E., Davis, J., & Preble, K. (2014). Factors influencing service- learning utilization in social work: Results from an online survey of faculty. Journal of Teaching in Social Work, 34, 147-166. doi: http://dx.doi.org/10.1080/08841233.2014.890692 Gibbs, A., & Stirling, B. (2013). "It's about people and their environment": Student social workers' definitions of social work research. Social Work Education, 32(3), 317-330. doi: http://dx.doi.org/10.1080/02615479.2012.658365 Gregory, V. L. (2010). Gregory Research Beliefs Scale: Factor structure and internal consistency. Research on Social Work Practice, 20(6), 641-650. doi: http://dx.doi.org/10.1177/1049731509353049 Harder, J. (2010). Overcoming MSW students' reluctance to engage in research. Journal of Teaching Social Work, 30(2), 195-209. doi: http://dx.doi.org/10.1080/08841231003705404 Hebert, C., Kulkin, H., & Ahn, B. (2014). Facilitating research self-efficacy through teaching strategies linked to self-efficacy theory. American International Journal of Social Science, 3(1), 44-50. Holden, G., Barker, K., Meenaghan, T., & Rosenberg, G. (1999). Research self-efficacy: A new possibility for educational outcomes assessment. Journal of Social Work Education, 35, 463-476. Holden, G., Barker, K., Rosenberg, G., & Onghena, P. (2007). Assessing progress towards accreditation related objectives: Evidence regarding the use of self-efficacy as an outcome in the advanced concentration research curriculum. Research on Social Work Practice, 17, 456-465. doi: http://dx.doi.org/10.1177/1049731506297474 Holden, G., Barker, K., Rosenberg, G., & Onghena, P. (2008). The Evaluation Self- Efficacy Scale: A replication. Research on Social Work Practice, 18(1), 42-46. doi: http://dx.doi.org/10.1177/1049731507303954 http://acen.edu.au/2012conference/wp-content/uploads/2012/11/50_Promoting-learning-and-teaching-about-research-through-collaborative-work-integrated-learning.pdf http://acen.edu.au/2012conference/wp-content/uploads/2012/11/50_Promoting-learning-and-teaching-about-research-through-collaborative-work-integrated-learning.pdf http://acen.edu.au/2012conference/wp-content/uploads/2012/11/50_Promoting-learning-and-teaching-about-research-through-collaborative-work-integrated-learning.pdf http://dx.doi.org/10.1177/00131640121971338 http://dx.doi.org/10.1080/08841233.2014.890692 http://dx.doi.org/10.1080/02615479.2012.658365 http://dx.doi.org/10.1177/1049731509353049 http://dx.doi.org/10.1080/08841231003705404 http://dx.doi.org/10.1177/1049731506297474 http://dx.doi.org/10.1177/1049731507303954 Deck et al./ENGAGED TEACHING-LEARNING 247 Knowles, M., Holton, E., & Swanson, R. (2005). The adult learner: The definitive classic in adult education and human resource development (6th ed.). Woburn, MA: Butterworth-Heinemann. Lemieux, C., & Allen, P. (2007). Service learning in social work education: The state of knowledge, pedagogical practicalities, and practice conundrums. Journal of Social Work Education, 43(2), 309-325. doi: http://dx.doi.org/10.5175/JSWE.2007.200500548 Lucas, E. (2000). Linking social work and service-learning. Journal of Baccalaureate Social Work, 5(2), 167-178. Lundahl, B. (2008). Teaching research methodology through active learning. Journal of Teaching in Social Work, 28(1/2), 273-285. doi: http://dx.doi.org/10.1080/08841230802179373 Luzzo, D., Hasper, P., Albert, K., Bibby, M., & Martinelli, E. (1999). Effects of self- efficacy-enhancing interventions on the math/science self-efficacy and career interests, goals, and actions of career undecided college students. Journal of Counseling Psychology, 46(2), 233-243. doi: http://dx.doi.org/10.1037/0022- 0167.46.2.233 MacIntyre, G., & Paul, S. (2013). Teaching research in social work: Capacity and challenge. British Journal of Social Work, 43, 685-702. doi: http://dx.doi.org/10.1093/bjsw/bcs010 Miller, J., Coombs, W., & Fuqua, D. (1999). An examination of psychometric properties of Bandura's multidimensional scales of perceived self-efficacy. Measurement and Evaluation in Counseling and Development, 31(4), 186-197. National Youth Leadership Council. (2007). Growing to greatness 2007: The state of service-learning. Retrieved from http://www.nylc.org/sites/nylc.org/files/files/320G2G07.pdf Pedrazza, M., Trifiletti, E., Berlanda, S., & Di Bernardo, G. (2013). Self-efficacy in social work: Development and initial validation of the self-efficacy scale for social workers. Social Sciences, 2(3), 191-207. doi: http://dx.doi.org/10.3390/socsci2030191 Phillips, A. (2011). Service-learning and social work competency-based education: A "goodness of fit"? Advances in Social Work, 12(1), 1-20. Phillips, J., MacGiollaRi, D., & Callaghan, S. (2012). Encouraging research in social work: Narrative as the thread integrating education and research in social work. Social Work Education, 31(6), 785-793. doi: http://dx.doi.org/10.1080/02615479.2012.695200 Quinney, A., & Parker, J. (2010). Developing self efficacy in research skills: Becoming research-minded. In H. Burgess, & J. Carpenter (Eds.), The outcomes of social work education: Developing evaluation methods (pp. 16-25). Southampton, England: http://dx.doi.org/10.5175/JSWE.2007.200500548 http://dx.doi.org/10.1080/08841230802179373 http://dx.doi.org/10.1037/0022-0167.46.2.233 http://dx.doi.org/10.1037/0022-0167.46.2.233 http://dx.doi.org/10.1093/bjsw/bcs010 http://www.nylc.org/sites/nylc.org/files/files/320G2G07.pdf http://dx.doi.org/10.3390/socsci2030191 http://dx.doi.org/10.1080/02615479.2012.695200 ADVANCES IN SOCIAL WORK, Fall 2015, 16(2) 248 Higher Education Academy Subject Centre for Social Policy and Social Work (SWAP). Rhoades, R. (1998). In the service of citizenship: A study of student involvement in community service. Journal of Higher Education, 69(3), 277-297. doi: http://dx.doi.org/10.2307/2649189 Shively, M., Smith, T., Bormann, J., & Gifford, A. (2002). Evaluating self-efficacy for HIV disease management skills. AIDS and Behavior, 6(4), 371-379. doi: http://dx.doi.org/10.1023/A:1021156914683 Unrau, Y., & Grinnell, R. (2005). The impact of social work research courses on research self-efficacy for social work students. Social Work Education, 24(6), 639-651. doi: http://dx.doi.org/10.1080/02615470500185069 Vanderbilt University Center for Teaching. (n.d.). What is service learning or community engagement? Retrieved from http://cft.vanderbilt.edu/guides-sub-pages/teaching- through-community-engagement/ Wells, M. (2006). Teaching notes: Making statistics "real" for social work students. Journal of Social Work Education, 42(2), 397-404. doi: http://dx.doi.org/10.5175/JSWE.2006.200400466 Wells, M., Maschi, T., & Slater, G. (2012). Integration of research and practice: Innovations and challenges in social work programs. Social Work Education, 31(3), 331-346. doi: http://dx.doi.org/10.1080/02615479.2011.551827 Wilson, G., & Kelly, B. (2010). Evaluating the effectiveness of social work education: Preparing students for practice learning. British Journal of Social Work, 40, 2431- 2449. doi: http://dx.doi.org/10.1093/bjsw/bcq019 Yorio, P., & Ye, F. (2012). A meta-analysis on the effects of service-learning on the social, personal, and cognitive outcomes of learning. Academy of Management Learning & Education, 11(1), 9-27. doi: http://dx.doi.org/10.5465/amle.2010.0072 Author note Address correspondence to: Stacy M. Deck, PhD, MSSW, Spalding University School of Social Work, 545 South 3rd Street, Louisville, KY 40203. Email: sdeck@spalding.edu http://dx.doi.org/10.2307/2649189 http://dx.doi.org/10.1023/A:1021156914683 http://dx.doi.org/10.1080/02615470500185069 http://cft.vanderbilt.edu/guides-sub-pages/teaching-through-community-engagement/ http://cft.vanderbilt.edu/guides-sub-pages/teaching-through-community-engagement/ http://dx.doi.org/10.5175/JSWE.2006.200400466 http://dx.doi.org/10.1080/02615479.2011.551827 http://dx.doi.org/10.1093/bjsw/bcq019 http://dx.doi.org/10.5465/amle.2010.0072 mailto:sdeck@spalding.edu