Microsoft Word - 04_Brown_and_Park_20870_Longitudinal_Student_Research ___________ Jodi L. Constantine Brown, PhD, MSW is Associate Professor and Director of Online & Offsite Programs, Department of Social Work, California State University, Northridge, Northridge, CA 91330-8226. Hyun-Sun Park, PhD, MSSW, is associate professor, Department of Social Work, California State University, Northridge, Northridge, CA 91330-8226. Copyright © 2016 Advances in Social Work Vol. 17 No. 1 (Spring 2016), 44-58, DOI: 10.18060/20870 Longitudinal Student Research Competency: Comparing Online and Traditional Face-to-Face Learning Platforms Jodi L. Constantine Brown Hyun-Sun Park Abstract: This exploratory research compares longitudinal research self-efficacy and retention between a completely asynchronous Master of Social Work (MSW) online cohort and its traditional face-to-face counterpart. This study used a non-equivalent comparison groups design with two groups: online instruction only (n=16) and traditional face-to-face instruction (n=32), with pretest (Time 1), posttest (Time 2) and follow-up (Time 3) standardized measures of practice evaluation knowledge (PEKS) and research self- efficacy (RSES) in a beginning research methods course. Results indicate that students’ knowledge and research self-efficacy improved between pretest and posttest and remained significantly improved at follow-up one year later, with no significant difference between online learners and traditional face-to-face students. Students gain and maintain confidence in research methods and evaluation regardless of the learning platform utilized. Keywords: Distance learning; knowledge retention; research, self-efficacy Online learning/distance education continues to grow in popularity, and the field of social work is no exception (Buchanan & Mathews, 2013; Shorkey & Uebel, 2014). Despite skeptics criticizing online education in social work as not providing sufficient practice, engagement, and interaction time (Allen & Seaman, 2011; Knowles, 2001; Pearlman, Weston, & Gisel, 2010), the number of online MSW-degree-granting programs continues to grow. As online learning/distance education instructional offerings expand, social work literature examining differences between online and traditional classroom teaching continues to develop. Previous literature has explored different types of classes/teaching methods, with the majority of published articles focusing on practice and research methods classes (Dennison, Gruber, & Vrbsky, 2010). The bulk of early literature developed around the idea of comparing web-based or online classes with traditional, non-hybrid, face-to- face courses (Dalton, 2001; Harrington, 1999; Hisle-Gorman & Zuravin, 2006; Huff, 2000; Kleinpeter & Potts, 2003; Royse, 2000; Seabury, 2005; Stocks & Freddolino, 2000; Westhuis, Ouellette, & Pfahler, 2006) but failed to use a design controlling for pretest scores (e.g., Harrington 1999; Hisle-Gorman & Zuravin, 2006; Kleinpeter & Potts, 2003), or use standardized measures of learning outcomes (e.g., Harrington, 1999; Royse, 2000; Westhuis et al., 2006). Further, the variability in programs and classes explored makes comparisons between study outcomes difficult. One of the challenges of building research knowledge in a developing area like distance education is consistent definition of terms. Distance education is a means of asynchronously delivering a course online or through interactive television (Vernon, Vakalahi, Pierce, Pittman-Munke, & Adkins, 2009). Quinn, Fitch, and Youn (2011) argue that synchronous technologies should be included in the definition of distance education. Asynchronous online classes allow a course or assignment to be completed at a student’s discretion within a given timeframe, whereas synchronous classes are held live via the Internet with the students and instructor engaging simultaneously (Cummings, Chaffin, & Cockerham, 2015). ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 45     Studies comparing online, face-to-face, and hybrid (a mix of online and face-to-face meetings) program models continue to grow, but no studies to date explore differences in the retention of knowledge over time by learning platform. The current research study addresses this gap by examining the practice evaluation knowledge and research self- efficacy of online and face-to-face MSW students over an 18-month period. Online vs. Traditional Classroom Platforms Previous social work literature specifically comparing online vs. traditional classroom experiences in terms of research methods learning has examined differences in student satisfaction (Faul, Frey, & Barber, 2004; Ligon, Markward, & Yegidis, 1999; Westhuis et al., 2006; York, 2008) showing mixed results ranging from higher overall satisfaction with web-assisted courses, to no difference in satisfaction levels by learning platforms, to higher satisfaction in the face-to-face classroom. While student satisfaction is considered an important part of successful online education (Siebert, Siebert, & Spaulding-Givens, 2006; Stocks & Freddolino, 2000), satisfaction does not necessarily equal effectiveness. Students might be satisfied with their program, but have they learned? Previous studies operationalize learning outcomes using course grades (Harrington, 1999; Hisle-Gorman & Zuravin, 2006; Kleinpeter & Potts, 2003; O’Neill & Jensen, 2014; York, 2008), exam scores (Cummings et al., 2015; Westhuis et al., 2006), and overall grade point average (GPA; Cummings et al., 2015). O’Neill and Jensen (2014) compared final course grades and self-reported GPA for forty-four MSW students enrolled in either a face- to-face (23 students) or an online (21 students) section of the same research course and found no significant differences between the two groups of students at the end of the course. Cummings et al. (2015) found mixed results, with advanced-standing face-to-face students having statistically significant higher GPA scores than their online counterparts, but found no significant difference in GPA between non-advanced-standing face-to-face and online students. Few studies comparing online with face-to-face learning practices have used standardized measures with demonstrated reliability and validity. Stocks and Freddolino (2000) examined comfort with technology and classroom environment in a sample of 60 MSW students. Using standardized measures of attitudes toward computers and technology use at the beginning and end of a research methods class with two sections (one online, the other face-to-face), they found no significant difference in the computer attitude scale between pretest and posttest and marginal differences in technology use between the two groups, with online students reporting greater comfort using technology than their face-to-face counterparts. Buchanan and Mathews (2013) used the Kirk-Rosenblatt Research Inventory (1981) to assess MSW social work students’ beliefs, knowledge, and opinions about research, finding no statistically significant difference between main campus and satellite MSW students’ knowledge. Cummings et al. (2015) used a standardized measure of self-efficacy to explore differences between online and face-to-face students from the perspective of an overall program, finding no significant difference between online and face-to-face student outcomes (Cummings et al., 2015). These results provide additional support of no Brown & Park/LONGITUDINAL STUDENT LEARNING 46   significant difference between online and face-to-face learning platforms (WCET, 2010), but do not explore the retention of knowledge over time. Knowledge Retention One challenge educators face is choosing a learning strategy that will result in long- term retention of knowledge (Beers & Bowden, 2005). Various theories posit strategies to improve knowledge and memory including, but not limited to, problem-based learning (Beers & Bowden, 2005; Schmidt, 1993), integrating the arts in education (Hardiman, Rinne, & Yarmolinskaya, 2014), team-based learning (Macke & Tapp, 2012), and diffuse learning (Raman et al., 2010). Outlining the myriad education mechanisms suggested to increase knowledge retention is beyond the scope of this article. The importance of examining knowledge retention, however, should not be overlooked (Raman et al., 2010; Wayne et al., 2006). If students do not retain knowledge, they may be less likely to successfully use their education and skills in the field, which is particularly important in helping professions such as social work. MSW students are required to successfully complete at least two research courses to earn their degree, and many schools require additional research methods coursework. However, few studies have explored the degree to which students maintain their research methods knowledge over time. Using a small sample (n=25) of undergraduate social work students, Baker, Pollio, and Hudson (2011) found evidence that students maintain educational gains one year after a research methods class, but their study tested knowledge perception as opposed to knowledge gained and did not compare differences by learning platform. The Current Study This study builds on previous research by replicating the Baker et al. (2011), pre/post/post study of BSW students using the same measure, the Practice Evaluation Knowledge Scale (PEKS, Baker & Ritchey, 2009), with a sample of MSW students. Further, we included a measure of Research Self-Efficacy (RSES, Holden, Barker, Meenaghan, & Rosenberg, 1999) and a comparison group of online distance learning students. This study compares practice evaluation knowledge and research self-efficacy learning outcomes between a completely asynchronous online MSW research methods class and its face-to-face classroom counterpart using a quasi-experimental non-equivalent comparison groups design. Using standardized measures of students' perception and confidence, this project builds on previous social work literature and adds to the ongoing online versus face-to-face classroom debate. Based on previous literature, we hypothesized that there would be no difference between the research knowledge and self-efficacy of online students and face-to-face classroom students a) after completing a beginning research class, and b) one year after the class. Method ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 47     Program/Class Description In fall 2012, a large public university on the west coast introduced a two-year, degree- granting, fully asynchronous online Master in Social Work (MSW) program. Students in both the online program and face-to-face program follow a cohort model, meaning that students enter and exit the program together and take classes in a prescribed order. Online students complete the program in two years. Traditional students have the option of completing the program in either two or three years. All students in the current study were part of a two-year cohort. Online students and face-to-face students differ in that online students follow a quarter system, taking two eight-week classes per quarter totaling four classes per semester. Traditional face-to-face students take four classes over 16 weeks each semester. In order to graduate, all students must take and successfully complete three semesters of research methods: beginning research methods, advanced research methods, and a Capstone project. The beginning research methods class focuses on problem formulation, operationalization, conceptualization, design, and measurement concepts, and students complete a single subject design over the course of the semester. The advanced research methods course focuses on sampling, ethics, program evaluation, qualitative research methods, and survey research and touches on statistical analyses. The Capstone project gives students the opportunity to design and conduct their own small research project including data analysis and presenting results. Students may choose to collect their own data, use agency secondary data previously collected for non-research purposes, use secondary data from publicly available sources (e.g., the General Social Survey or the National Health and Nutrition Examination Survey), or conduct a program evaluation. Students in all cohorts receive the same content, and classes happen in the same semester for students in a two-year cohort (i.e., everyone in the cohort has beginning research methods their second semester, advanced research methods their third semester and Capstone the semester before graduation.) Sample The study population included MSW students enrolled in one of three master’s level beginning research methods sections. Instructor B taught one section online (n=21). Instructor B and Instructor P each taught one traditional face-to-face section (n=13 for Instructor B; n=23 for Instructor P) for a total of 57 participants. Five students (3 face-to- face; 2 online class) did not complete the pretest, and three different students (1 online class; 2 face-to-face) did not complete the posttest, resulting in a valid n=49 (n=18 online; n=31 face-to-face) between Time 1 and Time 2. Between posttest (Time 2) and one-year follow-up (Time 3), two students left the online cohort for personal reasons; the remainder of the cohort (n=16) completed the Time 3 measure. One face-to-face student who did not complete the Time 2 measure did complete the Time 3 measure resulting in a valid n=48 (n=16 online; n=32 face-to-face) between Time 1 and Time 3. Design Brown & Park/LONGITUDINAL STUDENT LEARNING 48   This study used a non-equivalent comparison groups design with two groups: one consisting of students who received online instruction only, and one group consisting of students who received only face-to-face instruction, with pretest and posttest measures of student competency for both groups. Pretest competency scores were compared for all three sections. No significant differences in demographic variables or research competency scores were found between Instructor B’s face-to-face students and Instructor P’s face-to- face students, so those face-to-face sections were combined and compared to the online student competencies. Pretest measures were taken at Time 1 in January 2013 prior to the first research methods class, Time 2 in May 2013 after the first research methods class, and Time 3 in May 2014 after two subsequent research courses (advanced research and Capstone). Measurement Two standardized measures were used to assess student achievement of research competency: the Practice Evaluation Knowledge Scale (PEKS) and the Research Self- Efficacy Scale (RSES). The PEKS was developed to measure social work practitioners’ beliefs about their knowledge of practice evaluation competencies and has demonstrated internal consistency and validity (Baker et al., 2011, p. 558). Example items include “I have been adequately trained to conduct practice evaluation” and “I am familiar with issues of reliability and validity.” The 8-item PEKS (α=.88) is measured on a scale from 1-5 where 1=strongly disagree and 5=strongly agree. The RSES developed by Holden, Barker, Meenaghan, and Rosenberg (1999) has demonstrated internal consistency reliability, evidence of construct validity, and sufficient sensitivity “to detect change in students’ research self-efficacy from the beginning to the end of their participation in a single-semester research course” (p. 472). The 9-item RSES (α=.95) is measured on a scale from 0–10 where 0=cannot do at all, 5=moderately certain can do, and 10=certain can do. Items begin with the statement “How confident are you that you can…” and include “Do effective electronic searching of the scholarly literature”? and “Design and implement the best sampling strategy possible for your study of some aspect of practice?” Data Collection Together the PEKS and RSES total 17 questions. For the purpose of this study, each measure was included in an easily readable online chart where respondents were asked to click the button next to their response for each question. After receiving approval from the University Institutional Review Board, pretest data were collected online via the class web pages. Students were directed to a link to the survey prior to the first class session via an email message from the other instructor. Students were assured that their instructor would not see their survey results until after the class ended, and then only in aggregate. Each student has a unique login, so matching pretest with posttest data occurred seamlessly. There were no duplicate entries, meaning it was unlikely that students logged in under another students’ ID to complete either the pretest (Time 1) or posttest (Time 2). ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 49     Follow-up (Time 3) data collection began at the end of the third research (Capstone) class, a full 18 months after the pretest, and continued for approximately one month. Students were contacted via email, reminded about the study, and asked to complete the posttest using a class web page that was set up to collect their data with their unique login, thus allowing for pretest, posttest, and follow-up data to be matched easily. If students had difficulty navigating the webpage they were invited to return their responses via email, fax, or in-person. These responses were then entered into the database by hand by Instructor B and checked for accuracy by Instructor P. Approximately one-third of the responses were recorded in this manner. Data Analysis Descriptive statistics were used to summarize demographics. Paired samples t-tests were used to determine differences between pretest and posttest knowledge and self- efficacy scores. One-way between groups analysis of covariance (ANCOVA) was conducted to compare differences in learning platforms for research methods instruction for MSW students. ANCOVA tests the significance of group differences between two or more groups while controlling for one or more covariates (e.g., pretest scores) that may influence the dependent variable (Tabachnick & Fidell, 2007; Wright, 2006). For the current study, the independent variable was the type of learning platform (online vs. face- to-face classroom), and the dependent variable consisted of scores on the PEKS and RSES surveys administered at the end of the first research class (Time 2) and again at the end of the third research class (Time 3). Participants’ scores on the PEKS and RSES pretest surveys (Time 1) were used as the covariate in the analysis. Results Sample Demographics Students ranged in age from 22–44 years (M=28.55; SD=5.79) and were mostly women (83%). The majority of students identified as Hispanic (n=18), followed by Caucasian (n=14). No significant differences in age, race, gender, or previous research experience were found between the online and face-to-face students (Table 1). Face-to-face students (M=2.36, SD=.96) scored significantly higher than online students (M=1.89, SD=.56) on item 1 of the PEKS (I have been adequately trained to conduct practice evaluation), t(50)=2.21, p=.03, two-tailed, d=0.59. The magnitude of the difference in the means (mean difference=.47, 95% CI: .04 - .89) was moderate (Cohen’s d=0.59). There were no statistically significant differences in any of the remaining PEKS or RSES items for face- to-face or online students. Table 1. Student Demographics by Instructional Method Student Demographics Instructional Method Face-to-Face f (%) Online f (%) Gender n=36 n=21 Brown & Park/LONGITUDINAL STUDENT LEARNING 50   Student Demographics Instructional Method Face-to-Face f (%) Online f (%) Female 30 (83) 18 (82) Male 6 (17) 3 (14) Race/Ethnicity n=33 n=19 African American 2 (6) 2 (11) Asian American 2 (6) 1 (5) Caucasian 11 (31) 3 (16) Hispanic 11 (31) 7 (37) Multiracial 5 (14) 2 (11) Other (Armenian, Jewish) 2 (6) 4 (21) Previous Research Experience n=33 n=19 None 6 (18) 2 (11) 1 class…a long time ago 1 (3) 5 (26) 1 class 10 (30) 7 (37) 2-3 classes 13 (40) 4 (21) 4+ classes / very comfortable 3 (9) 1 (5) Knowledge Retention Pretest to Posttest Online and face-to-face students were grouped together for initial knowledge retention analyses. The PEKS composite scale (M=18.25, SD=5.47) demonstrated reliability (α=.88). Results from paired samples t-tests indicate a statistically significant increase in PEKS scores from pretest (M=18.20, SD=5.34) to posttest (M=29.12, SD=4.53), t(48)=12.48, p<.001 (two-tailed), d=1.74. The mean increase in PEKS scores was 10.91 with a 95% confidence interval ranging from 9.11 to 12.72. Cohen’s d (1.74) indicated a large effect size. The RSES composite scale (M=486.92, SD=176.55) demonstrated reliability (α=.95). RSES scores increased significantly from pretest (M=486.95, SD=165.38) to posttest (M=698.16, SD=126.73), t(48)=8.06, p<.001 (two-tailed), d=1.15. The mean increase in RSES scores was 211.24 with a 95% confidence interval ranging from 158.56 to 263.84. Cohen’s d (1.15) indicated a large effect size. See Tables 2 and 3 for paired samples t-test results by individual items on the PEKS and RSES. There was a substantial difference in program evaluation knowledge (as measured by the PEKS) and research self-efficacy (as measured by the RSES) for all students (online and traditional face-to-face) after taking the foundation research methods course. We explored differences between Time 2 and Time 3 and found that only questions 1 (effective electronic searching of the scholarly literature) and 3 (review a particular area of social science theory and research, and write a balanced and comprehensive literature review) on the RSES were significantly different between posttest (Time 2) (Q1 M=87.60, SD=13.63; Q3 M=78.60, SD=15.52) and follow-up (Time 3) (Q1 M=92.60, SD=13.82), t (49)=2.29, p=.03 (two-tailed), d=0.32; (Q3 M=87.80, SD=14.88), t (49)=3.77, p < .001 (two-tailed), d=0.53. None of the items on the PEKS, and no other items on the RSES were significantly different between Time 2 and Time 3. Table 2. Paired Samples T-Test Results for Online Student Responses by Item, Pretest (Time 1) and Posttest (Times 2 & 3) ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 51     Item Time 1 Pretest n=18 M (SD) Time 2 Posttest n=18 M (SD) t d Time 1 Pretest n=16 M (SD) Time 3 Posttest n=16 M (SD) t d PEKS 1 1.89 (0.58) 3.78 (0.64) -8.31* 0.8 1.94 (0.57) 4.25 (0.44) -15.36* 0.94 2 2 (0.68) 3.56 (0.78) -6.33* 0.7 1.88 (0.5) 3.5 (0.73) -6.78* 0.45 3 2.11 (0.83) 3.78 (0.8) -7.79* 0.78 2.06 (0.85) 3.81 (0.91) -7.00* 0.48 4 2.28 (1.01) 3.61 (0.69) -6.23* 0.69 2.25 (1.06) 4.06 (0.85) -6.53* 0.46 5 2.83 (1.09) 3.78 (0.8) -3.01* 0.35 2.63 (1.08) 4.12 (0.62) -5.47* 0.42 6 2.06 (0.72) 3.67 (0.76) -6.98* 0.74 2 (0.73) 3.75 (0.93) -5.91* 0.44 7 2.11 (0.83) 3.22 (0.8) -4.16* 0.5 1.88 (0.62) 3.18 (0.98) -4.39* 0.37 8 3 (1.18) 3.83 (0.7) -2.48* 0.27 2.81 (1.16) 4.06 (0.68) -3.87* 0.34 RSES 1 72.22 (24.86) 90.56 (10.55) -3.57* 0.43 73.75 (24.18) 96.25 (7.18) -3.73* 0.48 2 78.33 (24.31) 92.22 (10.6) -2.55* 0.28 76.88 (24.14) 94.37 (7.27) -2.69* 0.32 3 60 (22.75) 77.78 (18.96) -2.67* 0.3 61.88 (23.43) 89.37 (11.23) -4.15* 0.53 4 58.89 (24.22) 80 (16.8) -4.03* 0.49 59.38 (24.89) 87.5 (9.3) -3.97* 0.51 5 50 (23.01) 75.56 (18.22) -4.29* 0.52 51.88 (24.82) 78.75 (15.43) -3.98* 0.51 6 47.22 (24.92) 72.78 (16.01) -4.29* 0.52 48.13 (26.38) 76.25 (16.27) -4.03* 0.52 7 46.67 (23.51) 73.89 (16.85) -4.72* 0.57 47.5 (24.9) 75 (18.97) -3.90* 0.5 8 45 (25.49) 72.22 (18.64) -4.42* 0.53 45.63 (27.07) 73.12 (18.51) -3.56* 0.46 9 56.11 (23.04) 82.22 (16.64) -4.20* 0.51 56.88 (24.41) 85 (18.25) -4.43* 0.57 Note. PEKS=Practice Evaluation Knowledge Scale RSES=Research Self-Efficacy Scale *p<.05 Table 3. Paired Samples T-Test Results for Face-to-Face Student Responses by Item, Pretest (Time 1) and Posttest (Times 2 & 3) Item Time 1 Pretest n=31 M (SD) Time 2 Posttest n=31 M (SD) t d Time 1 Pretest n=32 M (SD) Time 3 Posttest n=32 M (SD) t d PEKS 1 2.29 (0.94) 3.68 (0.65) -6.74* 0.6 2.34 (0.97) 3.68 (0.93) -6.29* 0.56 2 1.94 (0.63) 3.58 (0.77) -10.01* 0.77 1.94 (0.61) 3.59 (0.87) -11.32* 0.8 3 1.9 (0.75) 3.52 (0.85) -9.08* 0.73 1.84 (0.72) 3.68 (0.96) -10.93* 0.79 4 2.26 (0.93) 3.74 (0.82) -8.91* 0.72 2.25 (0.98) 3.71 (0.95) -7.71* 0.65 5 2.74 (1.12) 4.13 (0.56) -6.42* 0.58 2.75 (1.16) 3.78 (0.75) -4.73* 0.43 6 2.23 (1.02) 3.9 (0.75) -8.67* 0.71 2.25 (1.07) 3.71 (1.02) -7.14* 0.62 7 1.9 (0.65) 3.06 (0.77) -6.44* 0.58 1.94 (0.76) 3.28 (1.02) -6.03* 0.54 8 2.9 (1.13) 3.45 (0.99) -2.02* 0.12 2.91 (1.14) 3.59 (0.87) -3.23* 0.25 RSES 1 73.55 (19.41) 86.45 (14.5) -3.92* 0.34 71.56 (23.43) 91.87 (14.46) -4.40* 0.38 2 71.61 (19) 86.13 (13.34) -4.43* 0.4 71.88 (19.08) 90.93 (13.99) -5.60* 0.5 3 55.48 (21.1) 77.42 (16.32) -5.76* 0.53 55.63 (22.99) 87.5 (15.45) -7.34* 0.63 4 56.13 (22.76) 77.42 (18.43) -4.14* 0.36 56.25 (25.62) 80.93 (18.89) -5.39* 0.48 5 44.52 (20.3) 72.26 (16.87) -7.06* 0.62 44.38 (22.99) 76.56 (19.27) -7.69* 0.66 6 40.32 (18.88) 70.32 (20.08) -6.70* 0.6 40 (21.09) 75 (20.63) -7.50* 0.65 7 38.71 (19.1) 71.94 (20.07) -6.64* 0.6 38.44 (20.8) 74.06 (19.81) -7.93* 0.67 8 37.42 (19.14) 67.1 (22.98) -5.78* 0.53 37.81 (22.1) 73.43 (21.03) -7.60* 0.65 9 53.23 (27.98) 78.06 (19.4) -5.08* 0.46 53.13 (29.77) 85 (18.13) -6.21* 0.55 Note. PEKS=Practice Evaluation Knowledge Scale RSES=Research Self-Efficacy Scale *p<.05 Online vs. Traditional Face-to-Face Platform Brown & Park/LONGITUDINAL STUDENT LEARNING 52   Our hypothesis stated that there would be no difference between the self-reported knowledge and self-efficacy of online and face-to-face students after the beginning research methods class and one year after the beginning research methods class. Preliminary checks were conducted to ensure that there was no violation of the assumptions of normality, linearity, homogeneity of variances, homogeneity of regression slopes, and reliable measurement of the covariate. After controlling for pre-test scores, there was no significant difference between online and traditional face-to-face classroom students on posttest (Time 2) PEKS scores F(1, 46)=0.01, p=.91, and posttest (Time 2) RSES scores, F(1, 46)=0.36, p=.55 after the first research class. There was no significant difference between online and traditional face-to-face classroom students on follow-up (Time 3) PEKS scores F(1, 45)=1.51, p=.22 or follow-up (Time 3) RSES scores, F(1, 45)=0.09, p=.76 one year after the beginning research class. Discussion The online students in this program completed a two-year, completely asynchronous, online MSW program with the same 16 hours per week in a local field placement as students in the face-to-face cohort. The standards for the online program are the same as those for the traditional classroom, and the same faculty members teach in both programs. As such, and based on previous literature, the investigators expected to find no difference in learning outcomes between the online and traditional face-to-face classroom students. Although not social work specific, meta-analyses comparing distance education and classroom instruction reveal somewhat mixed results with support leaning toward distance education being similar to traditional classroom instruction. Allen et al. (2004) and Sitzman, Kraiger, Stewart and Wisher (2006) found no differences in educational effectiveness for distance learners, with effectiveness defined as assessment of student performance (e.g., grades) or acquisition of declarative knowledge. However, Bernard et al. (2004) found wide variability and low effect sizes on various outcomes including student achievement, attitude, and retention. Note that Bernard et al. (2004) define retention as “the opposite of dropout” (p. 388) as opposed to the maintenance of knowledge. Creating subsets of synchronous and asynchronous applications resulted in effect sizes for asynchronous applications favoring distance education (Bernard et al. 2004). Sitzman et al. (2006) found web-based instruction 6% more effective than classroom instruction for teaching declarative knowledge. The current study examined pre and posttest practice evaluation and research self-efficacy scores of students in an asynchronous, web-based, distance-learning classroom compared to a traditional face-to- face classroom for research methods (declarative knowledge). Meaningful gains between pretest, posttest, and follow-up support the effectiveness of online and face-to-face education as seen in significant gains between the three time periods for the two groups. Further, finding no differences in the learning outcomes between the two learning platforms provides additional support that online and face-to-face learning modalities are equally effective. Finding differences in learning outcomes would have resulted in adjustments being made to either course, depending on the nature and direction of those differences. ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 53     Despite a growing body of evidence that online learning or distance education is as effective as traditional face-to-face classroom instruction, the various types of online education make comparisons difficult. Online learning or distance education ranges from in-service training on-demand via television and satellite (Williams, Nicholas, & Gunter, 2005) to asynchronous electronic software content (Harrington, 1999) to hybrid models combining face-to-face instruction with distance learning applications (Ayala, 2009; Osguthorpe & Graham, 2003; York, 2008). The current study adds to the developing body of literature by using standardized measures of learning outcomes, a pre/post/post quasi-experimental longitudinal design, and controlling for instructor and content differences in that the same instructor taught both the asynchronous online and face-to-face beginning research classes. Knowledge retention over time is one indicator of teaching effectiveness, and our results suggest that online learning platforms are at least as effective as traditional face-to-face classroom strategies in students maintaining their practice evaluation knowledge and research self-efficacy one year after taking a beginning research methods class. Limitations While we were able to control for instructor and content differences between Time 1 and Time 2, we were not as fortunate between Time 2 and Time 3. Online students maintained the same professor for all three research courses (beginning, advanced, and capstone), whereas the face-to-face students had the ability to choose their instructor for the advanced and capstone research classes. While faculty work together to ensure students are receiving standard content, there were likely differences in how that content was delivered. Since there were four potential instructors for the advanced research class that occurred fall 2013 (Instructor P and three additional faculty), and six potential capstone instructors during spring 2014 (Instructors B and P and four additional faculty), there was too much variability to statistically control for potential instructor differences. The sample size of this study was small. Although the use of small samples is common in social psychology (Johnson & Bachan, 2013) and education (Cook & Hatala, 2015), this limits the statistical power. A finding of no statistically significant difference between online and face-to-face students in this study may not indicate that there is no true difference but may also be a result of inadequate power. However, considering this study employs a theory-based prediction, strong design with longitudinal data, and standardized measures, the findings of this study may have meaningful educational implications. Findings indicate increased student self-efficacy in research methods, which may translate into greater comfort recognizing and employing evidence-based practices in the field. However, results should be interpreted with caution considering the current study used self-reports of a non-random, convenience sample of graduate social work students from a single university. Although there were no statistically significant differences between the face-to-face and online students at pretest, participants were not randomized into experimental and control groups, and it is possible that there are inherent differences between the two groups that account for the knowledge and self-efficacy gained. Although randomization may not be realistic in this type of education research, selection bias remains Brown & Park/LONGITUDINAL STUDENT LEARNING 54   a threat to internal validity with this non-randomized design. With the exception of the first item on the PEKS, there were no significant differences between the face-to-face and online students. Conclusion As online learning and distance education continues to develop, MSW programs have an opportunity to be on the cutting edge of this growth in the social work field. However, remaining on the cutting edge involves conducting research that goes beyond student satisfaction or course evaluations. Our results indicate that students gain and maintain confidence in research methods and evaluation regardless of the learning platform utilized. Using this model, we encourage future researchers to explore competencies and areas of social work practice beyond research methods. Despite the limitations of the current study, our results add to the growing body of literature showing that successful student research learning and knowledge retention may occur equally well through online and traditional face-to-face learning platforms. Future studies should continue exploring longitudinal research knowledge retention since the timing of content delivery in this sample differed by eight weeks (face-to-face students had a 16-week semester; online students had an eight-week course). It is possible that the shorter learning time could negatively affect longer-term retention of knowledge. Furthermore, research about the quality of programs from the perspective of learning outcomes triangulated with faculty-measured student competency could provide useful knowledge for informed practice and policy. A major strength of this study is the use of standardized learning outcome measures taken at three different time points over 18 months, with findings suggesting that the modality of content delivery is less important than the content itself. Implications for social work education include effectively utilizing a broad range of information and communication technologies and increasing accessibility to social work students in traditionally underserved areas. References Allen, I. E., & Seaman, J. (2011). Going the distance: Online education in the United States. BABSON Survey Research Group, The Sloan Consortium. Retrieved from http://www.onlinelearningsurvey.com/highered.html Allen, M., Mabry, E., Mattrey, M., Bourhis, J., Titsworth, S., & Burrell, N. (2004). Evaluating the effectiveness of distance learning: A comparison using meta-analysis. Journal of Communication, 54(3), 402–420. doi: http://dx.doi.org/10.1111/j.1460- 2466.2004.tb02636.x Ayala, J. S. (2009). Blended learning as a new approach to social work education. Journal of Social Work Education, 45(2), 277-288. doi:  http://dx.doi.org/10.5175/JSWE.2009.200700112 Baker, L. R., Pollio, D. E., & Hudson, A. (2011). Assessing student perception of practice evaluation knowledge in introductory research methods. Journal of Social ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 55     Work Education, 47(3), 555-564. doi: http://dx.doi.org/10.5175/JSWE.2011.201000022 Baker, L. R., & Ritchey, F. J. (2009). Assessing practitioner’s knowledge of evaluation: Initial psychometrics of the practice evaluation knowledge scale. Journal of Evidence-Based Social Work, 6(4), 376-389. doi: http://dx.doi.org/10.1080/15433710902911097 Beers, G., & Bowden, S. (2005). The effect of teaching method on long-term knowledge retention. Journal of Nursing Education, 44(11), 511-514. Bernard, R. M., Abrami, P.C., Lou Y., Borokhovski, E., Wade, A., Wozney, L… Huang, B. (2004). Does distance education compare with classroom instruction? A meta- analysis of the empirical literature. Review of Educational Research, 74(3), 379-439. doi: http://dx.doi.org/10.3102/00346543074003379 Buchanan R. L., & Mathews, D. A. (2013). A comparison of student knowledge and attitude toward research: Are main campus students different from those in a hybrid environment? Journal of Teaching in Social Work, 33, 467-480. doi: http://dx.doi.org/10.1080/08841233.2013.828668 Cook, D. A., & Hatala, R. (2015). Got power? A systematic review of sample size adequacy in health professions education research. Advances in Health Sciences Education, 20, 73-83. doi: http://dx.doi.org/10.1007/s10459-014-9509-5 Cummings, S. M, Chaffin, K. M., & Cockerham, C. (2015). Comparative analysis of an online and a traditional MSW program: Education outcomes. Journal of Social Work Education, 51, 109-120. Dalton, B. (2001). Distance education: A multidimensional evaluation. Journal of Technology in Human Services, 18(3-4), 101-115. doi: http://dx.doi.org/10.1300/J017v18n03_07 Dennison, S. T., Gruber, K. J., & Vrbsky, L. (2010). Research literature review on social work education instructional methods: 1998-2008. Journal of Teaching in Social Work, 30(4), 399-419. doi: http://dx.doi.org/10.1080/08841233.2010.517732 Faul, A. C., Frey, A. J., & Barber, R. (2004). The effects of web-assisted instruction in a social work research methods course. Social Work Education, 23(1), 105-118. doi: http://dx.doi.org/10.1080/0261547032000175764 Hardiman, M., Rinne, L., & Yarmolinskaya, J. (2014). The effects of arts integration on long‐term retention of academic content. Mind, Brain, and Education, 8(3), 144-148. doi: http://dx.doi.org/10.1111/mbe.12053 Harrington, D. (1999). Teaching statistics: A comparison of traditional classroom and programmed instruction/distance learning approaches. Journal of Social Work Education, 35(3), 343-352. Hisle-Gorman, E., & Zuravin, S. (2006). Teaching social work research: A comparison of web-based and in-class lecture methods. Journal of Technology in Human Services, Brown & Park/LONGITUDINAL STUDENT LEARNING 56   24(4), 77-93. doi: http://dx.doi.org/10.1300/J017v24n04_05 Holden, G., Barker, K., Meenaghan, T., & Rosenberg, G. (1999). Research self-efficacy: A new possibility for educational outcomes assessment. Journal of Social Work Education, 35, 463-476. Huff, M. (2000). A comparison study of live instruction versus interactive television for teaching MSW students. Research on Social Work Practice, 10, 400-416. Johnson, D. R., & Bachan, L. K. (2013). What can we learn from studies based on small sample sizes? Comment on Regan, Lakhanpal, and Anguiano (2012). Psychological Reports, 113(1), 221-224. doi: http://dx.doi.org/10.2466/21.02.07.PR0.113x12z8 Kirk, S. A., & Rosenblatt, A. (1981). Research knowledge orientation among social work students. In S. Briar, H. Weissman, & A. Rubin (Eds.), Research utilization is social work education: Major articles and final recommendations of the project on research utilization in social work education (pp. 29-39). New York: Council on Social Work Education. Kleinpeter, C., & Potts, M. (2003). Teaching practice methods using interactive television: A partial replication study. Journal of Technology in Human Services, 22, 77-87. doi: http://dx.doi.org/10.1300/j017v22n01_03 Knowles, A. (2001). Implementing web-based learning: Evaluation results from a mental health course. Journal of Technology in Human Resources, 18(3/4), 171-187. doi: http://dx.doi.org/10.1300/j017v18n03_11 Ligon, J., Markward, M. J., & Yegidis, B. L. (1999). Comparing student evaluations of distance learning and standard classroom courses in graduate social work education. Journal of Teaching in Social Work, 19(1-2), 21-29. doi: http://dx.doi.org/10.1300/J067v19n01_03 Macke, C., & Tapp, K. (2012). Teaching research to MSW students: Effectiveness of the team-based learning pedagogy. Journal of Teaching in Social Work, 32(2), 148-160. doi: http://dx.doi.org/10.1080/08841233.2012.668161 O'Neill, M., & Jensen, J. (2014). Comparison of student outcomes in a campus and distributed learning class. Journal of Technology in Human Services, 32(3), 186-200. doi: http://dx.doi.org/10.1080/15228835.2014.908157 Osguthorpe, R. T., & Graham, C. R. (2003). Blended learning environments: Definitions and directions. Quarterly Review of Distance Education, 4(3), 227-233. Pearlman, C., Weston, C., & Gisel, E. (2010). Enabling meaningful learning through web-based instruction with occupational therapy students. Educational Technology Research & Development, 58, 191-210. doi: http://dx.doi.org/10.1007/s11423-008- 9097-2 Quinn, A., Fitch, D., & Youn, E. (2011). Considering construct validity in distance educational research in social work education: Suggestions for a multivariate approach to researching efficacy. Journal of Social Work Education, 47(2), 321-336. ADVANCES IN SOCIAL WORK, Spring 2016, 17(1) 57     doi: http://dx.doi.org/10.5175/JSWE.2011.200900123 Raman, M., Mclaughlin, K., Violato, C., Rostom, A., Allard, J., & Coderre, S. (2010). Teaching in small portions dispersed over time enhances long-term knowledge retention. Medical Teacher, 32(3), 250-255. doi: http://dx.doi.org/10.3109/01421590903197019 Royse, D. (2000). Teaching research online: A process evaluation. Journal of Teaching in Social Work, 20(1-2), 145-158. doi: http://dx.doi.org/10.1300/J067v20n01_09 Schmidt, H. G. (1993). Foundations of problem-based learning: Some explanatory notes. Medical Education, 27, 422-432. doi: http://dx.doi.org/10.1111/j.1365- 2923.1993.tb00296.x Seabury, B. (2005). An evaluation of on-line, interactive tutorials designed to teach practice concepts. Journal of Teaching in Social Work, 25, 103-116. doi: http://dx.doi.org/10.1300/J067v25n01_07 Shorkey, C. T., & Uebel, M. (2014). History and development of instructional technology and media in social work education. Journal of Social Work Education, 50(2), 247- 261. Siebert, D. C., Siebert, C. F., & Spaulding-Givens, J. (2006). Teaching clinical social work skills primarily online. Journal of Social Work Education, 42, 325-336. doi: http://dx.doi.org/10.5175/JSWE.2006.200404103 Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59(3), 623-664. doi: http://dx.doi.org/10.1111/j.1744-6570.2006.00049.x Stocks, T. J., & Freddolino, P. P. (2000). Enhancing computer-mediated teaching through interactivity: The second iteration of a World Wide Web-based graduate social work course. Research on Social Work Practice, 10, 505-518. Tabachnick, B. G., & Fidell, L.S. (2007). Using multivariate statistics (5th ed). Boston, MA: Pearson Education. Vernon, R., Vakalahi, H., Pierce, D., Pittman-Munke, P., & Adkins, L. F. (2009). Distance education programs in social work: Current and emerging trends. Journal of Social Work Education, 45, 263-276. doi: http://dx.doi.org/10.5175/JSWE.2009.200700081 Wayne, D. B., Siddall, V. J., Butter, J., Fudala, M. J., Wade, L. D., Feinglass, J., & McGaghie, W.C. (2006). A longitudinal study of internal medicine residents’ retention of advanced cardiac life support skills. Academic Medicine, 81(10), S9-S12. doi: http://dx.doi.org/10.1097/00001888-200610001-00004 WCET. (2010). No significant difference. Retrieved from: http://www.nosignificantdifference.org/about.asp. Westhuis, D., Ouellette, P. M., & Pfahler, C. L. (2006). A comparative analysis of online and classroom-based instructional formats for teaching social work research. Brown & Park/LONGITUDINAL STUDENT LEARNING 58   Advances in Social Work, 7(2), 74-88. Williams, P., Nicholas, D., & Gunter, B. (2005). E-learning: What the literature tells us about distance education: An overview. Aslib Proceedings, 57(2), 109-122. doi: http://dx.doi.org/10.1108/00012530510589083 Wright, D. B. (2006). Comparing groups in a before-after design: When t-test and ANCOVA produce different results. British Journal of Educational Psychology, 76, 663-675. doi: http://dx.doi.org/10.1348/000709905X52210 York, R. O. (2008). Comparing three modes of instruction in a graduate social work program. Journal of Social Work Education, 44(2), 157-172. doi: http://dx.doi.org/10.5175/JSWE.2008.200700031 Author note Address correspondence to: Jodi L. Constantine Brown, California State University Northridge, Department of Social Work, 18111 Nordhoff Street, Northridge, CA 91330- 8226. Phone: 818-677-6010. Fax: 818-677-7662. Email: jodi.brown@csun.edu