Microsoft Word - 05_Forgey_and_Ortega-Williams_20877_Can_Social_Work _________________ Mary Ann Forgey, PhD, LCSW, is a Professor, Graduate School of Social Work, Fordham University, New York, New York 10023. Anna Ortega-Williams, LMSW, is a doctoral candidate at the Graduate School of Social Service, Fordham University, New York, New York 10023. Copyright © 2016 Advances in Social Work Vol. 17 No. 1 (Spring 2016), 59-77, DOI: 10.18060/20877 Effectively Teaching Social Work Practice Online: Moving Beyond Can to How Mary Ann Forgey Anna Ortega-Williams Abstract: Schools of social work are increasingly developing online courses and programs. While the majority of research comparing online and face–to-face courses has found equivalent outcomes, skepticism still exists, particularly about the ability to teach practice courses effectively online. This study adds to the growing body of research within social work that specifically examines the comparative effectiveness of online and face-to-face practice courses. Using an anonymous survey, 23 face-to-face and 12 online students enrolled in two separate sections of social work generalist practice rated the quality of the learning environment, the extent to which the course objectives were met, and the effectiveness of the teaching strategies from the students’ perspective. In addition, scores on assignment rubrics and student course evaluations were also compared. Results indicate no significant differences in learning outcomes as measured by assignment rubric scores, student perceptions of the extent to which learning objectives were met, the quality of the learning environment, and the effectiveness of five of the six teaching strategies used. We recommend that research moves beyond determining if online practice courses are as effective as face-to-face courses, and instead focus on a closer examination of the factors responsible for teaching effectiveness. Keywords: Distance education; online education; social work; direct practice; asynchronous learning The platform of 21st century social work education is transforming. In particular, graduate schools of social work in the U.S. are developing online courses and programs that include curriculum components thought to be primarily taught on campus, such as practice (Coe-Regan & Freddolino, 2008; Khaja, Ouellette, Barkdull, & Yaffe, 2008; Kurzman, 2013; McAllister, 2013; Roberts, Irani, Telg, & Lundy, 2005; Siebert, Spaulding-Givens, & Siebert, 2006). The Commission on Accreditation of the Council on Social Work Education (CSWE) does not keep an exhaustive list of all accredited online social work programs; however, they do list 40 master’s programs and 7 bachelor’s programs that are online, approximately 6% of the 775 accredited schools of social work (CSWE, 2016). Many social work programs also incorporate various forms of online learning into traditional face-to-face classes, including video conferencing, computer mediated technology, and Facebook activities (Barczyk & Duncan, 2013; Flynn, Maiden, Smith, Wiley, & Wood, 2013; Forgey, Loughran, & Hansen, 2013; Moisey, Neu, & Cleveland-Innes, 2008; Seabury, 2005). The impetus for online education in social work is varied and includes reaching students with geographical and time-related constraints (Larsen, Sanders, Astray, & Hole, 2008; Vernon, Pittman-Munke, Vakalahi, Adkins, & Pierce, 2001; Wilson, Brown, Wood, & Farkas, 2013). Additionally, for institutions like the University of Southern California, which created the first national online MSW program, virtual learning centers are Forgey & Ortega-Williams/CAN SOCIAL WORK PRACTICE 60 considered strong business models that can expand enrollment in social work schools, which often struggle with sustainability due to scanty sources of revenue (Flynn et al., 2013). The benefits of online social work education have been documented, and clear evidence has been mounting as to its effectiveness. Over the past two decades, a significant body of research has accumulated within social work comparing online and face-to-face programs as a whole (e.g., Cummings, Chaffin, & Cockerham, 2015; Forster & Rehner, 1998; Freddolino & Sutherland, 2000; Wilke & Vinton, 2006) as well as a range of individual courses, including practice (e.g., Coe & Elliot, 1999; Cummings, Foels, & Chaffin, 2013; Siebert et al., 2006; Thyer, Artelt, Markward, & Dosier, 1998) and research courses (e.g., Faul, Frey, & Barber, 2004; Petracchi & Patchner, 2000). An early comprehensive review of the state of social work research on distance courses and programs provided guidelines for future research (Macy, Rooney, Hollister, & Freddolino, 2001). For the most part, based on the empirical evidence to date, comparable levels of effectiveness have been found in relation to learning outcomes and student satisfaction with the instruction and learning environment. These results mirror the overall findings from the 2010 U.S. Department of Education (US DOE, 2010) meta-analysis of research from multiple disciplines. Based on a review of 99 experimental or quasi-experimental studies contrasting online and face-to-face conditions in relation to the learning outcomes achieved, the study concluded that students in online conditions had modestly better outcomes, on average, than their face-to-face counterparts (US DOE, 2010). Despite these fairly consistent findings of comparable effectiveness across multiple fields, concerns about the effectiveness of online courses in social work continue. These concerns, which have been present within social work throughout the development of distance education, have often focused on practice courses (Groshong et al., 2013; Khaja et al., 2008; Moore, 2005; Siebert et al., 2006; Siegel, Jennings, Conklin, & Flynn, 1998; Vernon et al., 2001).. For example, Siegel et al. (1998), found that social work educators have a bias against offering practice or methods courses online emanating from a belief that practice skills can only be taught in person. Similarly, Moore (2005) found that faculty perceived online education to be less effective than face-to-face instruction, particularly in the teaching of practice courses and clinical skills. A more recent example of this skepticism in relation to online practice courses is the 2013 report released by the Clinical Social Work Association (CSWA). CSWA questioned the effectiveness of online social work education to prepare social work practitioners without face-to-face instruction (Groshong et al., 2013). In particular, the report questioned if the nature of web-based learning was antithetical to the teaching of foundational practice skills, such as building empathy and conducting holistic assessments. The report also critiqued online education’s inconsistent delivery methods, lack of attention to implicit learning, and weak protocols around cornerstone pedagogical elements, like field education. Within the report, asynchronous methods were viewed as a form of rote learning, and the authors raised concerns about the ability of this method to facilitate the development of critical thinking skills. Lastly, in reference to online social work practice ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 61 coursework, CSWA recommended that the CSWE review how much training should be completed in-person to best transfer knowledge (Groshong et al., 2013). Given the continued expression of concern about the use of online teaching methods in social work, particularly in relation to practice courses, and the rapid expansion of online education in social work, an urgent need continues for studies that examine the efficacy of online instruction (Cummings et al., 2015). This study is an effort to respond to this need and in doing so, build specifically upon the research within social work on the comparable effectiveness of face-to-face and online practice courses. Literature Review There is a small but growing body of research that has specifically examined the effectiveness of online social work practice courses. These studies can be grouped into three design categories. The first group compared the effectiveness of an online practice course to its face-to-face counterpart. The second type of study compared the learning outcomes of students within a single online practice course, and the third type compared the learning outcomes in online practice courses to non-practice online courses. In the most recent face-to-face versus online comparative study found, Cummings and colleagues (2013) compared online students (n=37) and face-to-face students (n=63) enrolled in a course on evidence-based practice with groups. No significant differences were found between face-to-face and online students in exam scores, log grades, or course evaluation scores. Moreover, while both groups were found to have increased in leadership skills, as measured by a 22-item leadership pre-post scale, no significant differences were found between groups. However, Cummings and colleagues (2013) did not look at the sense of classroom community or teaching strategies in comparing student outcomes. Siebert and colleagues (2006) used a retrospective design to compare face-to-face (n=78) and online students’ (n=25) perceptions of their skill improvement using a four- item Likert scale. No significant differences between the two groups were found in the development of their brief treatment and crisis intervention skills. Additionally, Siebert and colleagues (2006), like Cummings and colleagues (2013), found that face-to-face and online student learning outcomes were comparable, with no significant differences in mean scores on a common assignment graded by the same instructor using a standardized rubric. However, when assessing student satisfaction using a post-course survey with eight items, two items were found to be significantly different. Compared to online students, face-to- face students were significantly more satisfied with instructor availability and the course’s ability to facilitate their learning (Siebert et al., 2006). The third comparative study of an online and face-to-face practice course, completed by Coe and Elliot in 1999, reflects the type of technology available at that time. It compared outcomes for 30 on-campus students with 47 students enrolled in a face-to-face satellite television instruction format. The experimental group attended class at either of two distance locations, with a live instructor present at each location. On average, on-campus learners were found to have higher assignment and final course grades when compared to distance learners; however, none of these differences were statistically significant. Coe and Elliot also found barriers to learning for the distance learners related to the problems Forgey & Ortega-Williams/CAN SOCIAL WORK PRACTICE 62 experienced with the technology. Recommendations included more enhanced training for instructors and increased use of visuals such as PowerPoint and video conferencing to enhance interaction both in and out of the classroom. In an earlier study by Thyer and colleagues (1998), online and face-to-face delivery formats were compared; however, each format was delivered in separate intervals to students within two separate practice courses. No significant differences in quality of teaching were found among the students in the Assessment and Psychopathology class; however, the students in the Treatment of Substance Abusers course evaluated the quality of the live instruction significantly higher than the televised teaching. Technological factors, including sound and visual transmission, were recognized as difficulties that may have biased the results. In relation to the second type of study design, two studies were found that examined students’ perceptions of their learning and satisfaction rates within an online practice course. Wilson and colleagues (2013) explored the impact of 3-D online technology to improve home visiting skills of social work practice students. Debriefing sessions with students participating in virtual home visits indicated that the experience was considered meaningful for skill development. Khaja and colleagues (2008) evaluated online social work practice students (n=21) using a participatory action research approach. They found that online social work practice students were impressed with the rigor of their online course, and the development of their skills throughout the process. However, student perception of learning was found to be contingent on comfort with technology and access to the software and hardware needed to participate (Khaja et al., 2008). Lastly, one study was found that compared the learning outcomes of multiple online foundation level courses including practice. Noble and Russell (2013) inquired into student satisfaction with their online social work program, surveying 242 students across multiple foundation courses using a pre-experimental, mixed method research design conducted over the course of three years. The primary measurement tool was a 41-item survey, through which they found that online social work practice students had the highest rate of satisfaction compared to research or policy online students. In addition to the three types of studies reviewed above, researchers have recognized the need to learn more about certain factors within online courses and how these factors compare to face-to-face courses (Cummings et al., 2015). In particular, the ability of online academic environments to intentionally produce the level of social presence necessary to build community among students and support their learning has been considered crucial (Bentley, Secret, & Cummings, 2015). An additional challenge identified for online practice courses is the need to increase understanding about how courses which require students to demonstrate skill competencies in every phase of practice can provide evidence of their effectiveness (Khaja et al., 2008; Siebert et al., 2006). ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 63 Study Background Description of Course Content The content of both the online generalist practice course and the face-to-face course evaluated in this study was the same, including the topics addressed, required readings, class exercises, and written assignments. The overall objective of both courses was to teach the knowledge and skills necessary to effectively engage in the beginning phase of the social work helping process with individuals, families, and groups. Given this objective, emphasis in both courses was placed on teaching students the preparation, engagement, assessment, and contracting skills to intervene in an empathic and culturally responsive way with both voluntary and involuntary clients. An outline of the course content is shown in Figure 1. Figure 1. Outline of Content for Generalist Practice I Preparation for Practice  Characteristics of a professional relationship  The parameters of practice: The role of the social work knowledge base, ethical code and laws and regulations  Impact of the agency environment on practice  Preparation tasks prior to meeting a client Engagement of Voluntary and Involuntary Clients  Clarifying one’s professional role and responsibilities and boundaries with the client  Exploration of the presenting issues  Basic communication and interviewing skills  Introduction to the stages of change and motivational interviewing Social Work Assessment  Overview of the phases of assessment o Information-gathering o Formulation of the issues o Goal-setting and intervention planning  Defining evidence-based assessment, strengths-based assessment, and culturally- responsive assessment  Methods of information-gathering (e.g., observation, interview, structured questionnaire, genogram, ecomap)  The problem formulation process  Developing a contract with voluntary and involuntary clients Case Management  Models of case management  Social work roles and functions Description of Differences in the Face-to-face and Online Course Delivery Methods While the content of the two courses mirrored each other, and both courses were taught by the same instructor, the way in which the content was delivered was very different. The face-to-face course was delivered over 15 class sessions lasting two hours each. The online Forgey & Ortega-Williams/CAN SOCIAL WORK PRACTICE 64 course was delivered asynchronously in eight modules each one week in duration. Figure 2 presents an overview of how the same content was delivered within each format. A more detailed description of the nuances between each delivery method is provided in Figure 2. Figure 2. Differences in the Delivery of the Face-to-face and Online Content Face-to-face Delivery Online Delivery Time Frame 15 Sessions; Two-hour sessions once per week 8 modules; One module completed asynchronously each week Lectures PowerPoint lecture delivered live with PowerPoint made available online after class. PowerPoint lecture delivered asynchronously online with instructor voice recording. Remains available online. Modeling of Client Engagement and Social Work Assessment Interviews In class video of practice interviews followed by an in-class structured analysis of the skills observed. Videos of practice interviews delivered online followed by an online submission of a structured analysis of the skills observed. Discussions Live discussions of lecture material during and after delivery. Asynchronous discussion boards of lecture material after online lecture made available online. Practice Exercises Simultaneous role plays conducted within small groups within the classroom with student observers providing feedback to the small group followed by a discussion of each group’s learning with the larger group. Synchronous small group role plays with student observer audio recorded through a free conference call followed by a review and analysis of the recording submitted by each group on the discussion board for class and instructor feedback. Self-Reflections on Practice Intermittent written reflections on questions about the application of the various skills within their practice setting submitted in hard copy by each student to the professor for feedback. Intermittent written reflections on questions about the application of the various skills within their practice setting submitted as a journal entry by each student to the professor for feedback. Assessment of Knowledge and Skill Integration Three written paper assignments that required the students to describe and critique their preparation, engagement, and assessment skills using a case(s) within their field agency. Three written paper assignments that required the students to describe and critique their preparation, engagement, and assessment skills using a case(s) within their field agency. Lectures The PowerPoint lectures delivered in each course introduced the students to the same material on preparation, engagement, assessment, and contracting. However, the lecture was delivered in person to the face-to-face students whereas the online students heard a recorded lecture. Both sets of students were provided the PowerPoint lecture online, which allowed the face-to-face students to also have access to the lecture at any time and to potentially review it again. ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 65 Class Discussions Following each PowerPoint lecture, the face-to-face students participated in live discussion and class exercises with the instructor, whereas the online students participated asynchronously via an online discussion board. The online asynchronous nature of the discussion board allowed the online students more time to think through their responses and to have their participation more systematically assessed by the instructor. For example, following the lecture on the parameters of practice, one of the discussion board assignments required the online students to make a preliminary post stating how they would respond to two different ethical dilemmas and to justify their responses based on what they had learned about their ethical obligations and any relevant laws and regulations. After all students had made their preliminary posts, they were required to read everyone else’s posts and then make a final post in which they could amend their responses incorporating what they had learned from their fellow classmates. The instructor then evaluated the quality of each student’s participation in this discussion and provided individualized feedback. This same ethical dilemma exercise was used in the face-to-face class as an in-class exercise, where each group worked together on their preliminary response and then had the opportunity to amend their response after hearing from the other groups. This more spontaneous discussion did not allow the same level of systematic evaluation or instructor feedback. Modeling of Practice Skills To teach engagement and assessment, these processes were first modeled for both sets of students via sequential video clips of a social worker engaging an individual client, a family, and a group, and in a later session, a social worker conducting an assessment. Following the viewing of these clips, the students in both courses analyzed what they observed through a series of questions. The face-to-face students shared their analyses in class as part of a class discussion. The online students submitted their analyses as journal entries, which again allowed for more systematic evaluation of their understanding and more opportunity to receive individualized feedback. Practice Exercises Opportunities for students to assess their own ability to practice engagement, assessment, and contracting were also provided within each course through exercises that required the students to examine their practice on actual cases from their field work, as well as through role plays. The role plays in both courses were conducted synchronously in small groups. However, in the face-to-face course, the role plays were not recorded because they were conducted simultaneously within one classroom making the recording of all of them impossible. Students in the online course conducted their role plays using a free conference call system, which allowed them to record the role plays, review them among themselves, and then submit them to the professor for review. Assessment of Student Knowledge and Skills Students in both courses were given the same three written assignments that evaluated their understanding and ability to assess their own agency practice and their ability to engage, assess, and contract with clients. Students in both courses who did not have Forgey & Ortega-Williams/CAN SOCIAL WORK PRACTICE 66 appropriate cases from their field work were given the opportunity to practice and critique their clinical skills by participating in an audiotaped phone interview with a standardized client played by a doctoral teaching assistant. The same doctoral teaching assistant, as well as the same standardized cases, adapted from real case situations provided by the professor, were used in both classes. Method Study Design and Procedure This study used a two group posttest only quasi-experimental design to compare the effectiveness of the face-to-face and online courses. All of the MSW students enrolled in the face-to-face and online sections of the Generalist Practice with Individuals, Families and Groups course were invited via email to complete an anonymous survey designed to evaluate the quality of the learning environment, the extent to which the course objectives were met, and the effectiveness of the teaching strategies from the students’ perspective. Twelve out of 15 students in the online class and 23 out of 25 students in the face-to-face class completed the survey, yielding an online class response rate of 80%, a face-to-face class response rate of 92%, and an overall response rate of 87.5%. The online students completed the survey on SurveyMonkey. The face-to-face students completed paper and pencil versions of the survey in class at the end of the last session. In addition to the survey questionnaire, several unobtrusive measures were included in the study. To measure learning outcomes, rubrics were developed for each of the three integrative papers to assess students’ understanding of the impact of the agency and community environment on practice, the process of engagement, and the process of assessment. To measure student satisfaction within the course, the survey administered to all students in the program at the end of each course was used. Measures Survey Measures The quality of the learning environment was measured using the Classroom Community Scale (Rovai, 2002), a 20-item measure designed to assess the concept of psychological community. The psychometric properties of the CCS are well-supported. Rovai (2002) found strong internal consistency (Cronbach’s Alpha: a=.93) and reliability (split-half coefficient of reliability of .91). Other studies using CSS have reported similar measures of reliability (Rovai & Baker, 2005; Rovai & Jordan, 2004). Each question has a 5-point Likert Scale with ratings ranging from 0 (strongly disagree) to 4 (strongly agree) for a total possible score of 80. The extent to which course objectives were achieved was measured by asking students to rate each of the seven course objectives listed in the syllabus. Each question has a 4-point Likert scale with response ratings ranging from 0 (not at all) to 4 (a great deal). The effectiveness of the teaching strategies was measured by asking the students to rate the PowerPoint presentations, videos, small group peer learning activities, class discussions, reflection assignments, and ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 67 integrative papers. Each question has a 4-point Likert scale with response ratings ranging from 0 (not effective) to 4 (very effective). Three open-ended questions were included at the end of the survey that asked students to reflect upon what they liked most and least overall about the course. One question asked for recommendations. At the end of the survey, as a way of assessing the equivalence of the two groups, the students were asked to report their number of years of practice experience, their comfort with technology, and the amount of time spent on the course each week. Years of practice is a variable that has been previously examined in relation to its impact on effectiveness (Coe & Elliot, 1999; Freddolino & Sutherland, 2000). Additionally, comfort with technology has also been found to be an important variable to consider when studying the effectiveness of distance education (Khaja et al., 2008; Larsen et al., 2008). Rubrics The first assignment rubric consisted of five content areas for a total possible score of 15 and assessed the student’s ability to identify environmental factors within the agency and community that could have a positive or negative impact on the client population’s experience of help. The second assignment rubric, made up of five content areas for a total possible score of 20, measured the student’s ability to describe and critique their preparation process for a first meeting with a client and to identify and analyze, through the use of a process recording of the first interview, the tasks and skills accomplished. The third assignment rubric consisted of nine content areas for a total score of 25 and measured the student’s ability to critique a bio-psycho-social assessment that they had completed in their field placement. In this critique, the students were required to analyze the extent to which they had described relevant individual, family, and environmental strengths and limitations; the methods used in gathering this multi-level information; and the problem formulation, contracting process, and evaluation plan. Student Satisfaction The school-wide student satisfaction survey consisted of 15 items that asked students to rate their satisfaction with all aspects of the course, including the course content, assignments, grading system, instructor quality, and responsiveness. Each question had a 5-point Likert scale ranging from 1-5, one being the lowest (strongly disagree) to five (strongly agree). The measure has not been evaluated for its reliability or validity. Results Equivalency of the Groups Due to the small and unbalanced sample sizes, the non-parametrical Mann Whitney U test was used to compare the years of practice experience reported by the students. A significant difference was found between online (Mdn=3) and face-to-face students (Mdn=0) in years of practice experience (U=40.5, p=.039). Using the Chi Square Fisher’s Exact test to compare the reported comfort level with technology and the reported hours of spent on classwork, there were no statistically significant differences found. Most (72.7%) Forgey & Ortega-Williams/CAN SOCIAL WORK PRACTICE 68 of online students expressed being very comfortable/comfortable with technology as compared to 50% of face-to-face students (p=.275). Most (80%) of the face-to-face students spent six or more hours on classwork compared to 90.9% of online students. Quality of Learning Environment When comparing the median scores of the online students (Mdn=60) and the face-to- face student scores (Mdn=57.5) on the Classroom Community Survey (Rovai, 2002) using the Mann Whitney U, no statistically significant difference was found (U=118.5, p=.626). Student Perceptions about Learning Objectives Given the distribution of the responses, the categories a great deal and mostly were collapsed into one, as were the categories, somewhat and not at all. No statistically significant difference was found between online and face-to-face students in their perceptions of the extent to which the course learning objectives were achieved, using the Chi Square Fisher’s Exact test (See Table 1). Table 1. Student Self-Report Scores of Meeting Learning Objectives (Mostly/A Great Deal) Items % Rating Mostly/A Great Deal p* Face-to-face (n=23) Online (n=12) Demonstrate understanding of first session tasks with clients. 100% 100% No difference Effectively engage voluntary and involuntary clients. 90.9% 100% 0.529 Understand the role of diversity when working with clients. 77.3% 100% 0.137 Articulate and critically apply strengths-based conceptual framework to guide assessment and evaluation. 86.4% 91.7% 1 Demonstrate capacity during the assessment and case formulation process to distinguish, appraise, and integrate multiple sources of knowledge. 100% 100% No difference Demonstrate capacity to develop collaborative and mutually agreed upon intervention goals. 86.4% 100% 0.537 Understand relationship between goal-setting, intervention, and evaluation. 90.9% 100% 0.529 *p<.05, Fischer's Exact Test Effectiveness of Teaching Methods Given the distribution of the responses, the categories very effective/effective were collapsed, as were the categories, somewhat effective/not at all effective. Five of the six teaching methods measured for effectiveness were not found to have a significant ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 69 difference between what was ranked very effective/effective and somewhat effective/not effective at all. However, a significant difference was found in one of the six teaching methods. This difference was found in relation to the reflection assignments, the goal of which was to have the students reflect upon the specific skills they were learning in relation to client engagement and problem exploration and if and how their prior or current experience impacted their practice. The very effective/effective rating was significantly higher for the online students (p=.030) using the Fischer’s Exact Test (See Table 2). Table 2. Scores for Effectiveness of Teaching Methods (Very Effective/Effective) Items % Rating Very Effective/Effective p* Face-to- face (n=23) Online (n=12) PowerPoint 63.6% 91.7% 0.113 Videos 90.9% 100% 0.529 Small Group Peer Learning Activities 54.5% 41.7% 0.721 Class Discussions 95.5% 83.3% 0.279 Integrative Papers 90.9% 100% 0.529 Reflection Assignments 61.9% 100% .030* *p< .05, Fischer’s Exact Test Open-Ended Questions The qualitative data generated from the open-ended questions were coded for prominent categories, as is common in qualitative approaches such as phenomenology and grounded theory (Creswell, 2013). The content analysis consisted of placing responses into the category that was best aligned. The number of times that a response was given was counted to arrive at larger themes, which is also considered best practice among qualitative approaches (Creswell, 2013). Overall, both online and face-to-face students made similar observations and recommendations. Both online and face-to-face students liked the course videos. One face-to-face student remarked, “Watching practitioners deal with various clients” was impactful. Likewise, an online student said, “I really learned a lot from watching the videos that demonstrated certain interview skills.” Group projects were liked the least, but each group gave different reasons. A few online students expressed frustration with the synchronous format, given that the majority of content was delivered asynchronously. An online student stated, “While I appreciated actually connecting with classmates, arranging a time for a conference call was incredibly difficult.” Another online student expressed a preference for the activity to be delivered asynchronously, “They really served little purpose to me; none at any rate that couldn’t have been achieved via discussion board.” Several face-to-face students also said being “put in random groups” was what they liked least. Students in the face-to-face class also said that they wanted to “socialize with other people besides the people in the group.” Students also commented on other class features. Face-to-face and online students mentioned teacher responsiveness as what they liked most. “Quick turn-around time” was Forgey & Ortega-Williams/CAN SOCIAL WORK PRACTICE 70 noted several times by online students. Having “an open forum” to express oneself was noted as important in the face-to-face class. Both online and face-to-face students noted that while the course work was rigorous, the readings, assignments, and “multimodal learning” were considered positive. Rubric Scores Using the Mann Whitney U to compare the median rubric scores on each of the assignments, the median scores were found to be the same for the face-to-face and online students on the field entry paper (Mdn=14) and on the engagement paper (Mdn=18). The median rubric score for the online students on the assessment paper was 24 and for the face-to-face students it was 23. No significant differences were found when comparing the median scores for the field entry (U=183.5, p=.746), the engagement paper (U=159.0, p=.417), or the assessment paper (U=171.0, p=.788). See Table 3. Table 3. Rubric Scores Assignment Rubrics Face-to-face Median (n) Online Median (n) p* Field Entry Paper 14 (26) 14 (15) 0.746 Engagement Paper 18 (25) 18 (15) 0.417 Assessment Paper 23 (24) 24 (15) 0.788 *p<.05 Student Satisfaction Using the Mann Whitney U test to compare the median scores on the Student Satisfaction survey, the online score (Mdn =5) and the face to face score (Mdn = 4.9), although both quite high, were found to be significantly different (U=32.5, p=.001). However, given that the actual point difference was only 0.1, the practical significance of this finding is questionable. Discussion While skepticism remains about the effectiveness of online courses for teaching social work practice, the results from this study suggest otherwise, and are consistent with previous studies. The learning outcomes, as measured by the rubric scores on the three integrative assignments, were found to be comparable, as were the students’ perceptions of the quality of the learning environment, the extent to which they believed the course objectives were achieved, and the effectiveness of five out of six teaching strategies used. The one difference found in relation to teaching strategies was a result of the online group reporting a higher rate of effectiveness in relation to the reflection assignments. High course satisfaction scores were also found for both groups, with the online group’s scores being slightly but significantly higher. The overall findings from this study provide a basis for continued optimism about the ability of online courses to teach social work practice concepts and skills as effectively as face-to-face classes. However, this optimism must be grounded in an understanding of this study’s strengths and limitations. ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 71 Many of this study’s strengths can be found in its design components, specifically the use of a comparison group that received the same course content within the same timeframe and was delivered by the same instructor. The only difference was the method of delivery. Another design strength was the multiple measures used to understand three different dimensions of effectiveness: student perceptions of effectiveness, student satisfaction, and learning outcomes. Furthermore, the study design and measures used were also fairly easy to implement, which allows for easy replication in the future. This last strength, ease of implementation, is of particular importance in light of this study's major limitation, the small sample size, which limits its generalizability. In addition to the small sample size, another limitation of the study is the way in which effectiveness was measured. While the students’ level of knowledge was measured and compared between the two cohorts using the assignment rubric scores, this measurement did not evaluate their actual skill level. To do so effectively would have required the students be rated on their skill levels. Standardized client role plays are one method that could have been used to measure the students' skill levels. Role plays were considered given the principal investigator’s previous experience with using this method to assess practice skills (Forgey, Badger, Gilbert, & Hansen, 2013). However, after considering the time and resources that would be required to develop role plays to assess specific practice skills, to hire and train standardized clients, to schedule the student role plays, and to implement a rating system including the hiring and training of raters, role play was deemed an unrealistic method for this study. Another limitation was the lack of measurement of the actual change that occurred in relation to the students' knowledge. Even though the findings indicated that the knowledge in both cohorts, based on the assignment rubric scores, was not significantly different at the end of the course, it is unknown to what extent, if any, that students’ knowledge changed over time as a result of the practice course. A pre- and posttest design would be needed to measure this variable. Pretests were not pursued due to time and cost constraints and the desire to develop realistic and implementable evaluation procedures that had more of a chance to be replicated across the curriculum. The findings from this study also provide some guidance about what factors make an online and face-to-face course equally effective. Gaining more insight into these factors and processes in future research will provide valuable information as to what it was about the curriculum or students themselves that resulted in the level of comparability found. One factor that may have contributed to the equivalent results was the level of curriculum consistency between the two formats. Having the same instructor deliver both courses played a role in this consistency, but other strategies used in the curriculum design may have further assured the sameness of the content delivered. The same PowerPoint lectures and practice model videos were used in both classes, and all students were able to download and reference the PowerPoints. Further, the major discussion questions, small group activities, course assignments, and material sequencing was the same for both classes. As online programs grow, concerns about the consistency across face-to-face and online courses will increase due to the greater likelihood of faculty teaching solely in one format or the other. While instructor training will always be a critical factor in achieving Forgey & Ortega-Williams/CAN SOCIAL WORK PRACTICE 72 curriculum consistency, course design strategies can be implemented by faculty to further ensure face-to-face and online course consistency. However, this will require organizational structures and resources that support both online and face-to-face faculty involvement in the early stages of course design and development. Understanding what accounted for relatively high ratings for both sets of students on the classroom community scale and the lack of differences is another area for future exploration, given the importance of establishing a social presence within an online environment (Bentley et al., 2015). A closer look at the way in which interactivity happened among the students and between students and instructor is necessary to accomplish this understanding. Within both classes, interactivity occurred in response to the planned discussion questions, the video analysis questions, and the intermittent small group activities that required students to practice the skills together and to discuss among themselves how they were applying what they were learning to their specific cases. In addition, for small group activities, detailed instructions were used in both classes, with the only difference being the medium used to have the interaction. These interactive activities also took place within the same sequence within each class. The amount of individual student interaction with the instructor is also an important element to the student’s experience of classroom community and to the instructor establishing their presence within the class with each student. While the amount of student questions to the instructor cannot be planned for, the amount of other types of student/instructor interactivity can be planned by the type, amount, and timing of the assignments for which individual feedback is provided. Within this class there were three major assignments where extensive feedback was provided individually to each student. In addition, the professor also provided individual feedback on the five reflection assignments. Certain characteristics of the students themselves may also partly explain the comparability of the data. On two of the three student characteristics, the amount of time reportedly spent on classwork each week and comfort with computer technology, there were no significant differences. However, one of the student characteristics that was significantly different was the amount of practice experience. The online students were found to have significantly more years of practice experience. This one difference may be a possible explanation for the significantly higher value placed on the reflection assignments by the online students, who perhaps as a result of having more practice experience had more to share when asked to apply and discuss what they were learning in relation to their prior and current experience. Attention needs to be given to ways in which the value of reflection could be increased for students with little or no practice experience. In the quest to identify factors that may help explain the level of comparable effectiveness, class size is another factor that deserves more attention in future studies of course effectiveness (Jones, 2015). While this study found no significant differences in learning outcomes or the quality of the learning environment, it is not known how dependent these outcomes were on the particular class sizes in this study. As online social work programs expand, the question of how large an online class can be before impacting the quality of the learning environment and learning outcomes needs to be addressed. ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 73 Future program-wide evaluations involving multiple sections and courses need to explicitly examine this question by analyzing if and how class size impacts the learning outcomes in the class, as well as the faculty work load, which has been reported by some experienced online instructors to be 40% higher in online courses (Jones, 2015; Pelech et al., 2013). Further investigation is especially important in light of the review of existing multidisciplinary research on the appropriate class size for online courses carried out by Taft, Perkowske, and Martin (2011). They found mixed and at times contradictory results. And although none of the studies reviewed included graduate online social work programs, the review did identify a number of factors that may be responsible for the variation found in the studies on the impact of online class sizes on student outcomes. Some of these include: a) the type of course and specifically the extent of factual information versus application, analysis and/or synthesis within it; b) the level of course and amount of teaching intensity and interaction expected; c) the extent to which the course is delivered synchronously vs. asynchronously; d) the presence or absence of technology support and teaching assistants; e) level of faculty expertise in online education; and f) the type of evidence of student learning. Given that the practice course evaluated in this study was asynchronous with a high level of application, analysis and synthesis as well as interaction within it, it seems imperative that future effectiveness research examine if and how class size may impact the effectiveness of this type of course. Conclusion Online programs are developing rapidly within schools of social work. Based on the findings in this study, as well as previous studies that have compared online and face-to- face practice courses, the student learning experience in these courses appears to be comparable. However, learning outcomes in social work practice courses are difficult to measure effectively. To do so requires the measurement of both knowledge and skills, as well as the extent to which the skill levels changed as a result of the course. While there has been an increased focus on the need to measure learning outcomes in reaction to the proliferation of online social work courses, and in particular practice courses, this need also exists and deserves equal attention in the face-to-face curriculum. Ironically, the development of online curricula and the concerns raised about the learning outcomes in these courses have resulted in a much needed focus on the evaluation of learning outcomes in both online and face-to-face courses. While the findings to date, which demonstrate comparable learning, may allay some of the concerns about the effectiveness of online courses, the measures used need to be further developed to more accurately measure the knowledge and skills gained as a result of enrollment in each method of delivery. Doing so in a way that is easily implementable across the curriculum and provides results in a timely and useful way to the faculty responsible for monitoring and improving course quality is one of the major challenges ahead. References Barczyk, C. C., & Duncan, D. G. (2013). Facebook in higher education courses: An analysis of students’ attitudes, community of practice, and classroom community. Forgey & Ortega-Williams/CAN SOCIAL WORK PRACTICE 74 International Business and Management, 6(1), 1-11. doi: http://dx.doi.org/10.3968/j.ibm.1923842820130601.1165 Bentley, K. J., Secret, M. C., & Cummings, C. R. (2015). The centrality of social presence in online teaching and learning in social work. Journal of Social Work Education, 51(3), 494-504. Coe, J. A. R., & Elliott, D. (1999). An evaluation of teaching direct practice courses in a distance education program for rural settings. Journal of Social Work Education, 35(3), 353-365. Coe-Regan, J. A., & Freddolino, P. (2008). Integrating technology into the social work curriculum. Alexandria, VA: Council on Social Work Education. Council on Social Work Education [CSWE]. (2016). Distance education. Retrieved from http://www.cswe.org/cms/39516.aspx Creswell, J. W. (2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). Los Angeles, CA: Sage. Cummings, S. M., Chaffin, K. M., & Cockerham, C. (2015). Comparative analysis of an online and a traditional MSW program: Educational outcomes. Journal of Social Work Education, 51(1), 109-120. doi: http://dx.doi.org/10.1080/10437797.2015.977170 Cummings, S. M., Foels, L., & Chaffin, K. M. (2013). Comparative analysis of distance education and classroom-based formats for a clinical social work practice course. Journal of Social Work Education, 32(1), 68-80. doi: http://dx.doi.org/10.1080/02615479.2011.648179 Faul, A. C., & Frey, A. J., & Barber, R. (2004). The effects of web-assisted instruction in a social work research methods course. Social Work Education: The International Journal, 23(1), 105-118. doi: http://dx.doi.org/10.1080/0261547032000175764 Flynn, M., Maiden, R. P., Smith, W., Wiley, J., & Wood, G. (2013). Launching the virtual academic center: Issues and challenges in innovation. Journal of Teaching in Social Work, 33(4-5), 339-356. doi: http://dx.doi.org/10.1080/08841233.2013.843364 Forgey, M., Badger, L., Gilbert, T., & Hansen, J. (2013). Using standardized clients to train social workers in intimate partner violence assessment. Journal of Social Work Education, 49(2), 292-306. doi: http://dx.doi.org/10.1080/10437797.2013.768482 Forgey, M. A., Loughran, H., & Hansen, J. (2013). Utilizing videoconferencing to introduce an international perspective to foundation practice. Journal of Teaching in Social Work, 33, 449-466. doi: http://dx.doi.org/10.1080/08841233.2013.829550 Forster, M., & Rehner, T. (1998). Part-time MSW distance education: A program evaluation. Computers in Human Services, 15(2-3), 9-21. doi: http://dx.doi.org/10.1300/J407v15n02_02 ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 75 Freddolino, P. P., & Sutherland, C. A. (2000). Assessing the comparability of classroom environments in graduate social work education delivered via interactive instructional television. Journal of Social Work Education, 36(1), 115-129. Groshong, L., McKenna, R., Hest, K., Hadley, S., Freeman, J., …Stephenson, D. (2013). Report on online MSW programs. Retrieved from http://www.clinicalsocialworkassociation.org/Resources/Documents/CSWA%20- %20Position%20Paper%20-%20Online%20MSW%20Programs%20- %20September2013.pdf Jones, S. H. (2015). Benefits and challenges of online education for clinical social work: Three examples. Journal of Clinical Social Work, 43, 225-235. doi: http://dx.doi.org/10.1007/s10615-014-0508-z Khaja, K., Ouellette, P., Barkdull, C., & Yaffe, J. (2008). Online practice course development with action research: A case example. Advances in Social Work, 9(2), 91-105. Kurzman, P. A. (2013). The evolution of distance learning and online education. Journal of Teaching in Social Work, 33(4/5), 331-338. doi: http://dx.doi.org/10.1080/08841233.2013.843346 Larsen, A. K., Sanders, R., Astray, A. A., & Hole, G. O. (2008). E-teacher challenges in international comparative social work courses. Social Work Education, 27(6), 623- 633. doi: http://dx.doi.org/10.1080/02615470802201671 Macy, J. A., Rooney, R. H., Hollister, C. D., & Freddolino, P. P. (2001). Evaluation of distance education programs in social work. Journal of Technology in Human Services, 18(3/4), 63-84. doi: http://dx.doi.org/10.1300/J017v18n03_05 McAllister, C. (2013). A process evaluation of an online BSW program: Getting the student perspective. Journal of Teaching in Social Work, 33(4/5), 514-530. doi: http://dx.doi.org/10.1080/08841233.2013.838200 Moisey, S. D., Neu, C., & Cleveland-Innes, M. (2008). Community building and computer mediated conferencing. Journal of Distance Education, 22(2), 15-42. Moore, B. (2005). Faculty perceptions of the effectiveness of web-based instruction in social work education: A national study. Journal of Technology in Human Services, 23(1/2), 53-66. doi: http://dx.doi.org/10.1300/J017v23n01_04 Noble, D., & Russell, A. C. (2013). Research on webbed connectivity in a web-based learning environment: Online social work education. Journal of Teaching in Social Work, 33, 496-513. doi: http://dx.doi.org/10.1080/08841233.2013.829167 Pelech, W., Wulff, D., Perrault, E., Ayala, J., Bayton, M., Williams, M., … Shankar, J. (2013). Current challenges to social work distance education: Responses from the Elluminati. Journal of Teaching in Social Work, 33(4/5), 393-407. doi: http://dx.doi.org/10.1080/08841233.2013.834863 Forgey & Ortega-Williams/CAN SOCIAL WORK PRACTICE 76 Petracchi, H. E., & Patchner, M. A. (2000). Social work students and their learning environment: A comparison of interactive television, face-to-face instruction, and the traditional classroom. Journal of Social Work Education, 36(2), 335-347. Roberts, T. G., Irani, T. A., Telg, R., & Lundy, L. K. (2005). The development of an instrument to evaluate distance education courses using student attitudes. American Journal of Distance Education, 19(1), 51-64. doi: http://dx.doi.org/10.1207/s15389286ajde1901_5 Rovai, A. P. (2002). Development of an instrument to measure classroom community, Internet and Higher Education, 5, 197-211. doi: http://dx.doi.org/10.1016/S1096- 7516(02)00102-1 Rovai, A. P., & Baker, J. D. (2005). Gender differences in online learning – Sense of community, perceived learning and interpersonal interactivity. Quarterly Review of Distance Education, 6(1), 31-44. Rovai, A., & Jordan, H. (2004). Blended learning and sense of community: A comparative analysis with traditional and fully online graduate courses. International Review of Research in Open and Distance Learning, 5(2), 1-13. Seabury, B. A. (2005). An evaluation of on-line, interactive tutorials designed to teach practice concepts. Journal of Teaching in Social Work, 25(1-2), 103-115. doi: http://dx.doi.org/10.1300/J067v25n01_07 Siebert, D. C., Spaulding-Givens, J., & Siebert, C. F. (2006). Teaching clinical social work skills primarily online: An evaluation. Journal of Social Work Education, 42(2), 325-336. doi: http://dx.doi.org/10.5175/JSWE.2006.200404103 Siegel, E., Jennings, J. G., Conklin, J., & Flynn, S. A. N. (1998). Distance learning in social work education: Results and implications of a national survey. Journal of Social Work Education, 34(1), 71-80. Taft, S. H., Perkowski, T., & Martin, L. S. (2011). A framework for evaluating class size in online education. The Quarterly Review of Distance Education, 12(3), 181-197. Thyer, B., Artelt, T., Markward, M. K., & Dozier, C. D. (1998). Research notes: Evaluating distance learning in social work: A replication study. Journal of Social Work Education, 34(2), 291-295. U.S. Department of Education, Office of Planning, Evaluation, and Policy Development (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, D.C.: Author. Vernon, R., Pittman-Munke, P., Vakalahi, H., Adkins, L. F., & Pierce, D. (2001). Distance education programs in social work: Current and emerging trends. Journal of Social Work Education, 41(2), 263-275. Wilke, D., & Vinton, L. (2006). Evaluation of the first web-based advanced standing MSW program. Journal of Social Work Education, 42(3), 607-620. doi: http://dx.doi.org/10.5175/JSWE.2006.200500501 ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 77 Wilson, A. B., Brown, S., Wood, Z. B., & Farkas, K. J. (2013). Teaching direct practice skills using web-based simulations: Home visiting in the virtual world. Journal of Teaching in Social Work, 33, 421-437. doi: http://dx.doi.org/10.1080/08841233.2013.833578 Author note: Address correspondence to Mary Ann Forgey, Graduate School of Social Service, Fordham University, 113 West 60th Street, New York, N.Y. 10023, forgey@fordham.edu