Learner Participation in the Online Learning Experience: Help or Hindrance? Developments in Business Simulation and Experiential Learning, Volume 33, 2006 LEARNER PARTICIPATION IN THE ONLINE LEARNING EXPERIENCE: HELP OR HINDRANCE? E. Brian Peach The University Of West Florida bpeach@uwf.edu Richard G. Platt The University Of West Florida rplatt@uwf.edu Martin Hornyak The University Of West Florida mhornyak@uwf.edu ABSTRACT The investigation examined the relationship between learner participation in an online learning experience, and performance on tests covering course topics. Data from four online courses compared the various levels of performance on tests with the amount of online participation in a threaded discussion. The results indicate that on average, as online participation decreases, average performance on tests decreases. INTRODUCTION The use of Internet technology to support online learning is becoming increasingly common (Amerine & Potosky, 2005; Arbaugh, 2000). Fewer than 100 colleges or universities offered Internet-based courses in 1993 (Hankin, 1999; Gibson, Tesone, & Blackwell, 2001). By 1999 the number had risen to almost two thirds of the 3200 accredited four year colleges and universities (Clarke, 1999). In 2002, one out of every four students taking an MBA was getting it online through a single university (Anonymous, 2002). The reasons can range from the ability to access more students to a belief it results in economic savings. The trend is accelerated by technological advances in computing power, improved interconnectivity through higher bandwidth, as well as the competitive pressures from stakeholders and alternative sources of education (Moore, 1997; Rahm & Reed, 1997). Traditional education=s share of the market is expected to decline as corporate online learning offerings increase (Gold, 2001). For MBA programs with declining enrollments, distance learning is seen as a recruiting source (MacLellan & Dobson, 1997). Some view online learning only as a supplement to traditional pedagogies and retain face-to-face communication to some degree. But for the pure online course, all interaction becomes electronic. For those who believe interaction among students is necessary for learning, whether for all disciplines or just for their specific disciplines, the requirement for meaningful participation through online interaction becomes an imperative. There is a growing body of research on the nature and outcomes of online learning. One important question is whether we should even expect the same outcomes between the two pedagogical approaches. Arbaugh (2000) proposed two questions: (1) will the Internet support effective learning, and (2) what factors influence online learning. This study focuses on what affects online learning. Although a number of possible drivers of learning have been identified (Golladay, Prybutok, & Huff, 2000), Arbaugh (2000) concluded the only variable to influence learning was interaction. This study looks at interaction in the form of online participation in a threaded discussion as a possible influence on learning. Specifically, this study tests for a relationship between online participation in a threaded discussion, and learning as measured in the form of written tests. ONLINE LEARNING Educational institutions are increasingly looking to supplement traditional classroom experiences with distance learning technology. Traditionally distance learning used voice and video transmission from fixed locations (Gibson &Gibson, 1995). One popular approach to distance learning is online learning, where technology such as the Internet is used to place the entire course pedagogy into an electronic environment (Smith, 2005). One important consequence of the fully electronic approach is the loss of face-to-face contact between student and instructor, as well as between students (Gibson, et. al., 2001). This loss may make some courses more appropriate for online learning than others (Potosky, 2002). For the online learning approach to establish itself as a viable 170 mailto:bpeach@uwf.edu mailto:rplatt@uwf.edu Developments in Business Simulation and Experiential Learning, Volume 33, 2006 alternative to the traditional approach, it must somehow establish that student learning is, if not directly equal, at least comparable to traditional methods (Krug, 2002). To some, this means online learning must emulate the traditional approaches, while to others the path is not as important as the result (Brower, 2003; Feinstein, 2004). For some, the belief equivalency begins with pedagogy runs deep (Simonson, Schlosser, & Hanson, 1999). A growing number of writers contend course design and the role of participants such as the instructor must adopt new approaches to create effective online learning programs (Bigelow, 1999; Clark, 2001; MacKinnon, 2000; Shrivastava, 1999). There are a number of reasons online learning is becoming more widely used. Some administrators believe it reduces the need for the expensive overhead of classrooms and buildings. Others tout the flexibility it brings to students, enabling them to >attend= classes although physically remote from the university site. Certainly a university has no geographic boundaries limiting the students it may attract. However, with the impetus to move to online learning, difficulties and challenges inherent in the new medium challenge many of these assumptions. Savings in physical overhead expense is counterbalanced by the need for increased time and new skills on the part of instructors to construct a viable online learning experience. Even experienced online instructors find that it typically requires more time and effort to construct an online course than a traditional course (Arbaugh, 2000). For those new to the medium, the experience can be overwhelming. The >flexibility= for students in online learning can only be realized through a relatively sophisticated use of technology to create a user friendly web experience. Simply placing course materials, supplementary materials, and assignments online is a recipe for disaster. Increasingly, creators of an online learning experience are finding major differences in how courses are constructed and administered. One is that the role of the instructor must evolve to that of a facilitator (Shrivastava, 1999; Clark, 2001; Palloff & Pratt, 1999). A second major difference is in the nature of interaction between student and instructor and between students. Research in the area of interaction has looked at methods to enhance interaction, and the question of whether interaction affects learning. ONLINE PARTICIPATION A fully electronic online learning experience provides no opportunity for face-to-face feedback or interaction among students. For many courses, interaction is considered to be a major concern in course design (Chisamore, 2004). Interaction as a part of the learning environment has been noted for its importance in the research on learning (Dewy, 1938; Vigostky, 1978; Vrasidas & McIsaac, 1999; Chisamore, 2004). Interaction has been identified as one of the 12 most critical factors for online learning (Golladay, Prybutok & Huff, 2000), one of the 9 lessons for creating an online course (Gibson et. al., 2001), or where learning begins (Pallof & Pratt, 1999). Thus the contention is face-to-face classroom interaction must be replaced with some alternative technique when using an electronic medium. The challenge is how to electronically accomplish the activity of face-to-face interaction. In this area there is not much guidance (Potosky, 2002). Non-verbal communication typically accounts for half of the information transmitted (Hamilton & Parker, 2001). Another challenge is face-to-face communication is realtime: it is simultaneous, or synchronous. One of online learning=s attractions is its potential for flexibility. To achieve this flexibility, students must be able to communicate on their own schedule. Thus online methods for interaction are typically asynchronous, there is no requirement that students be logged on to the website simultaneously. One asynchronous approach is an online threaded discussion, which provides students with a virtual collaborative learning environment. A threaded discussion addresses a topic selected by the instructor. Students make comments about the topic, read other students= comments, and respond to those comments. The discussion is asynchronous to allow flexibility in times students participate, and is monitored by the instructor. This flexibility allows the discussion to become a virtual learning space that supports collaborative learning (Arbaugh, 2000). In addition, the flexibility in the online learning environment is particularly attractive to graduate students and professionals (Dumont, 1966; Greco, 1999). Given the literature=s support for high interaction as a component of the online learning environment, the online version of the course used in this study made interaction a key component of its structure. This study examined the research question: is there a relationship between course performance on tests and the level of participation in online threaded discussions? METHOD The course used in this study is a foundation proficiency course in information systems. It is one of six foundation proficiency courses used in the university=s MBA program. Each of the six proficiency courses is taken by students when the MBA Director determines that they do not have the necessary background or currency in a particular foundation subject area prior to enrolling in the MBA curriculum=s core courses. Typically requiring a proficiency course would be for students with non-business backgrounds such as engineering, or whose business degree was not sufficiently recent. The course used in this study was initially developed for a face-to- face delivery format using the university learning management systems to supplement course content. Because all students did not require every foundation course, the variability in class size made it difficult to schedule and staff face-to-face sessions. Due to variability in demand, the course was converted to a totally online delivery format using the university=s online, internet-based learning management system. The course content consists of five modules. Each module contains a reading assignment in the textbook, definition 171 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 exercises, an annotated Power Point presentation, one to three short videos showing information systems applications related to the module topics, and three discussion questions on the subject of the module. The discussion questions are managed using the learning management system=s threaded discussion group tool. The instructor moderates the discussions and participates as appropriate, just as an instructor would act in a face-to-face classroom environment. The second test for a relationship used groups based on typical grade-bands of the test scores, e.g., 90-100, 80-90, etc. Correlations were run between test performance scores and participation event scores within the grade-band based groupings. The test was to determine if within any one or more of the groups, performance and participation tended to move together. Although the study=s purpose was to ask if there is a relationship between test performance and participation, we did not necessarily expect a smooth relationship, and due to visual observation of individual outliers in the data we expected them to affect the within group correlation results. The major effect we were looking for was an overall relationship between test performance and participation, and we believe averaging each group=s test scores and participation event scores would smooth the variability. Thus, for the final test, averages for each of the performance groups were calculated. If there was any relationship at all between test performance scores and participation event scores, we would expect it to show as a correlation (positive or negative) between the group averages. As a test for this relationship, the correlation between grade-band group test performance score averages and grade-band group participation event score averages was calculated. As part of the course syllabus, the students are advised that participation in the discussion groups counts for 10% of the grade (80% of the course grade comes from the final exam). Based on results of the first offering of the online course, the students in subsequent courses were further cautioned that lack of involved participation in the discussion groups has been correlated with poor performance on the final exam. Data from four administrations of the online course were used in this study. The class compositions were similar in student demographics and size. Tests scores were from the final examinations. Participation scores were the number of quality events by a student. The instructor reviewed all student comments and comments such as AI agree@ were not counted. Only substantive comments that reflected a contribution to the discussion were included. The review of all online comments can be a very time-consuming effort, and some researchers elect to use proxies such as logged-in time or total number of replies (Amerine & Potosky, 2004). In this study, the class sizes were relatively small, typically about ten students, and it was feasible to review each participation event. Due to the small class sizes, the data from all four courses were combined into a single dataset of forty four students. RESULTS The first test we accomplished was a broad, overall test of the complete dataset. Using every data point, the correlation between each student=s test score and participation event score was .35, and it was significant at p=.01. Thus, the first test found a significant positive relationship between test scores and participation event scores, although the relationship was not a large one. Based on the literature, it was not clear whether increased participation would have an effect on course performance as reflected in test scores. In addition, our data were limited by the number of responses on two variables. Thus we did not hypothesize there would be an effect; the study question was to determine if we could find a relationship. The data variables we used were the final exam test score, and the number of quality online participation events in a threaded discussion accomplished by the students. In testing for a relationship, the choice existed to use the dataset as a whole, or to subdivide it into groups. The second test was for within group relationships. With the responses sorted by test score, scores in the grade-band of 90-100 were placed in the first group, 80-90 in the second group, and similarly through all the responses. Due to the low n-size, all scores below 60 were placed in a single group. The results are mixed as only the AD@ group had a significant correlation of .79 (p=.05), and there was a negative although not significant correlation for AC@ and AF@ students. The first test used the complete dataset of all student responses. If there was a relationship between test score performance and participation event scores, we would expect it to show as a correlation (positive or negative) between the test scores and participation event scores. Table 1 - Relationship within groups Test Score n Correlation A - F 44 .35** A - 90-100 10 .20 B - 80-90 15 .11 C - 70-80 10 -.09 D - 60-70 4 .79* F - <60 5 -.05 For the next set of tests we elected to use groupings of the data. The data were sorted based on test performance, and a visual inspection of the data revealed some potential problems. There were no apparent natural groups in the data. A number of possible groupings was considered, but the use of the traditional letter grade grouping seemed a reasonable approach. Although there was visually a general downward trend of participation with performance, there was substantial variation in participation within performance groups at all levels. *p<.05 **p<.01 ***p<.001 172 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 The lack of significant within-group relationships might be attributed to the small n sizes. It also likely the extreme variability of participation made the relationship impossible to detect. For all groupings, the existence of outliers increased the participation score variability and distorted the relationship. For example, if a group=s average participation was six, there might be one or two at the bottom of the range with score of 20 and 22. Figure 1 illustrates a comparison of standardized test and participation scores to highlight the variability in participation event scores versus test scores. The next test for a relationship looked for a correlation between grade-band group test performance score averages and grade-band group participation event score averages. First we calculated the average test scores and participation event scores for each grade-band group as shown in Table 2. We then looked for a correlation between the group averages. The correlation between average test score and average participation event score for all groups, A-F, was .84 (p=.05) (Table 3). We had previously noted that the >A= group did not seem to follow the overall trend. This anomaly in the relationship between group test performance score averages and group participation event score averages is illustrated in Figure 2. From the graph it was apparent the top group=s participation and performance relationship was different from the remaining groups. Figure 1: Standardized Score Relationship for Test Scores and Participation Scores -6.00 -5.00 -4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 Test Score Participation The average participation for Group A was below both the B and C groups. (Note: For clarity of presentation the participation scores were multiplied by 10). To determine the nature and degree of difference this could cause, we accomplished a second correlation excluding the >A= group. When we excluded the AA@ group, the correlation between average test score and average participation event score for groups B-F was .99 (p=.001). The results are tabulated in Table 3. Figure 2: Average Participation Score vs Average Test Score 0.0 20.0 40.0 60.0 80.0 100.0 120.0 140.0 90-100 80-90 70-80 50-60 Test Score Participation 173 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 Table 2 - Group Averages GROUP n TEST SCORE AVERAGE PARTICIPATION AVERAGE A - 90-100 10 93.1 8.9 B - 80-90 15 85.4 11.5 C - 70-80 10 76.5 10.4 D - 60-70 4 66.9 4.5 F - <60 5 51.3 2.0 Although the overall dataset returned a significant positive correlation, our expectation was the group average comparison was the strongest test. With a correlation of .84 (p=.05), this test provided strong confirmation that there is a relationship between interaction and performance. Eliminating the A group led to an extremely high correlation at .99, p<.001. Even though we found significant relationships, we cannot conclude there is a causal relationship between interaction and performance. It might be that higher performing students tend to participate more. In fact, the A group=s lower participation rate than the B and C groups might indicate that there is no causal influence of interaction on performance. Another possible explanation is the best students do not need as much interaction as weaker students to perform well. High performers may have study habits that allow them to master material with less assistance. It may be high performers do not like to share their insights. For courses graded on a curve, high performers could easily view sharing information as a zero sum game. Table 3 - Relationship Between Groups GROUPS n CORRELATION A-F 5 .84* B-F 4 .99*** *p<.05 **p<.01 ***p<.001 DISCUSSION Although we might intuitively expect it, the literature has not established that higher levels of interaction lead to higher levels of learning. In this study we used performance on a final exam as a proxy for learning, which is likely common to most course learning measures. To measure interaction, we used the number of quality participation events in an asynchronous threaded discussion. In a comparison of test score performance and number of participation events, we found a correlation of .34 (P=.05). This provides evidence that interaction by students is indeed positively related to course performance and hopefully learning. Although significant, we concluded the relatively low correlation was due to the high variability in the participation event scores. CONCLUSION This study was designed to determine if we could identify a relationship between interaction as measured by participation in a threaded discussion, and learning as measured by exam scores. A positive correlation was found for the entire dataset, and between groups of the data. We can reasonably conclude that there is a positive relationship. We cannot conclude that increased levels of interaction lead to higher performance. It does appear higher performing students tend to interact more. Certainly, lower performing students tend to participate less. Future research should use course designs which test for causality. Interventions to increase the participation rate in a test group, along with a non-intervention control group could be used to test for causality. The factors that lead to increased online performance are a rich area for further research. We expected a larger correlation if we used groups to average out the data and smooth the variability. Having the groups identified, we decided to test for within group correlations. The anomalous results are to be expected. The small n-sizes of the groups and the high variability of the data all contributed to masking any possible relationship. The identification of groups was relatively arbitrary, as there was a fairly smooth decrease in test scores with no obvious group breaks. Using grade-band limits for identifying groups disallows any investigator bias in selecting group boundaries. It also likely allowed the possibility of including scores at the low or high end of a group that might more properly be in an adjoining group. In the final analysis, it was neither critical nor anticipated that we would find a within group relationship. 174 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 175 REFERENCES Amerine, K., & Potosky, D. 2005. Learner behavior in the online classroom experience. Developments in Business Simulation and Experiential Learning, 32, 1-9. Arbaugh, J.B. 2000. How classroom environment and student engagement affect learning in Internet-based MBA courses. Business Communication Quarterly, 63(4), Summer 2000, 9-26. Anonymous. 2002. MBA anywhere anytime. Canadian HR Reporter, June 3, 2002, 15, 11 Bigelow, J. D. 1999. The web as an organizational behavior learning medium. Journal of Management Education, 23. 635-650. Brower, H. H. 2003 On emulating classroom discussion in a distance-delivered OBHR course: Creating an on-line learning community. Academy of Management Journal of Learning and Education, 2(1), 22-36. Chisamore, D. B. 2004. Structuring case analysis for asynchronous delivery. Proceedings, 11th annual international distance education conference. Clark, L. J. 2001. Web-based teaching: a new educational paradigm. Intercom, 48, 20-23. Clark, R. D. 1999. Going the distance. Black Enterprise, 29, April 1999, 113-118. Dumont, R. A. 1996. Teaching and learning in cyberspace. IEEE Transactions on Professional Communication, 39(4), 192-204. Feinstein, A. H. 2004. A model for evaluating online instruction. Developments in Business Simulation and Experiential Learning, 31, 32-39. Gibson, C., & Gibson, T. 1995. Lessons learned from 100+ years of distance learning. Adult Learning, 7(1), 15. Gibson, J. W., Tesone, D. V., & Blackwell, C. W. 2001. The journey to cyberspace: Reflections from three online business professors. S.A.M. Advanced management Journal, Winter 2001, 66(1), 30-34. Gold, S. 2001. E-learning: The next wave of experiential learning. Developments in Business Simulation and Experiential Learning, 28, 76-79. Golladay, R. M., Prybutok, V. R., & Huff, R. A. 2000. Critical success factors for the online learner. The Journal of Computer Information Systems, 40(4), Summer 2004, 69- 71. Greco, J. 1999. Going the distance for MBA students. Journal of Business Strategy, 20(3), 30-34. Hamilton, C., & Parker, C. 2001. Communicating for results: A guide for business and professionals, 6th ed. Belmont, CA: Wadsworth/Thomson learning. Hankin, J. N. 1999. Alice, the college teacher, and the Rotweiler in wonderland: The prospects and problems of distance education. Executive Speeches, 14(2), 18-21. Krug, D. H. 2002. Pedagogy, multimedia, and distance education: Developing critical inquiry through virtual communities. Proceedings of the Annual Conference on Distance learning and Teaching. MacKinnon, G. R. 2000. The dilemma of evaluating electronic discussion groups. Journal of Research on Computing in Education, 33, 125-132. MacLellan, C., & Dobson, J. 1997. Women, ethics, and MBAs. Journal of Business Ethics, 16(11), 1201-1209. Moore, M. G. 1997 Three types of interaction. The American Journal of Distance Education, 3(2), 1-6. Palloff, R. M., & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco: Jossey-Bass Publishers. Potosky, D. 2002. Virtually experiential classrooms. Developments in Business Simulation and Experiential Learning, 29, 172-178. Rahm, D., & Reed, B. J. 1997. Going remote: The use of distance learning, the World Wide Web, and the Internet in graduate programs of public affairs and administration. Public Productivity and Management Review, 20(4), 459- 471. Shrivastava, P. 1999. Management classes as online learning communities. Journal of Management Education, 23(6), 691-703. Smith, J. A. 2005. Evaluating the direction of research in online education: are we going anywhere? Developments in Business Simulation and Experiential Learning, 32, 340-343. Simonson, M., Schlosser, C., & Hanson, D. 1999. Theory and education: a new discussion. American Journal of Distance Education, 13, 60-75. Table of Contents Volume 33, 2006 Learning Assurance Using Business Simulations Applications To Executive Management Education Team Teaching In An Integrated Business Course Using Critical Problem Based Learning Factors In An Integrated Undergraduate Business Curriculum: A Business Course Success Personality Type And Strategic Planning Business Games As Strategic Management Laboratories The Relationship Between Students' Success On A Simulation Exercise And Their Perception Of Its Effectiveness As A PBL Problem Forecasting Accuracy And Learning: The Key To Measuring Simulation Performance The Business Strategy Game: A Performance Review Of The New Online Edition Using The Socratic Method And Bloom's Taxonomy Of The Cognitive Domain To Enhance Online Discussion, Critical Thinking, And Student Learning Using Negotiation Exercises To Promote Critical Thinking Skills Effective Leadership Experiences For Management Majors In A Futures Class The Role Of Learning Versus Performance Orientations When Reacting To Negative Outcomes In Simulation Games Is Pay Inversion Ethical? A Three-Part Exercise Simulations And Experiential Exercises - Do They Result In Learning? Have We Figured It Out Yet? Examining Program Management In Business Simulations: Student And Faculty Views Validating Business Simulations: Do Simulations Exhibit Natural Market Structures? Characterizing Business Games Used In Distance Education Utilizing Games In A Graduate Level Instructional Game Course Employment Interview Preparation: Assessing The Writing-To-Learn Approach Simulations - Bridging From Thwarted Innovation To Disruptive Technology Creating An Authentic Cultural Lens Using Case Dialogue Learning By Fire: Reflections Of A First Time Online Instructor An International Internship With A Service-Learning Focus Learner Participation In The Online Learning Experience: Help Or Hindrance? Any Given Sunday: Intervention In Pursuit Of Simulation Team Parity Beginning With The End: Creating An Experiential Exercise From Assessment Criteria Simulating Life Cycles: Life Span As The Measure Of Performance In Business Gaming Simulations It's Puzzling: Communications, Competition, And Cooperation Balanced Scorecard Implementation For Strategy Management: Variation Of Manager Opinion In Real And Simulated Companies Cases And Business Games: The Perfect Match! Three-Attribute Interrelationships For Industry-Level Demand Equations Using A Web-Based Module To Teach Information Literacy Decision Support System For Demand Forecasting In Business Games The Invalidity Of Profit=F(Market Share) PIMS Validation Of Marketing Games Online Market Test Laboratory With The MINSIM* Program The Gas Mileage Game - A Policy Simulation Delivered Cost And Differentiation Applied To Threshold 3rd Ed. The Effect Of Team-Leadership Modes On Team Performance: A Preliminary Study The Design And Use Of A Macroeconomics Simulation Using Maple Software: A Pilot Study The Instructor's Toolbox: A Meaning-Centered Framework For The Social Construction Of Experiential Learning Incorporating Strategic Product-Mix Decisions Into Simulation Games: Modeling The 'Profitable-Product Death Spiral' Group Composition And Groupthink In A Business Game A Direct Approach To Teaching Business Ethics A Decision Support System For Planning Sales, Production, And Plant Addition With Manager: A Computer Simulation Polish - American Entrepreneurial Business Cooperation Workshop Utilizing The Income/Outcome Simulation Student Leader Training Exercise Student Preference To Mode Of Learning In Hong Kong Experiential Learning For Technology-Based And Management Programme In Hong Kong: A China Study Tour A Price Game With Product Differentiation In The Classroom Discrete Event Modeling In A New Transportation Simulation Supply-Side Modeling In A Total Enterprise Simulation The Quality Game Towards A Massive Multiplayer Online Business Simulation Making The Connection: Improving Virtual Team Performance Through Behavioral Assessment Profiling And Behavioral Cues Individual Learning Producing A Learning Organization: 'Playing Dice With Polar Bears' Narratology and Ludology: Competing Paradigms or Complementary Theories in Simulation Framework For Evaluating Internet Research Using Children's Games To Illustrate Strategy Concepts: Is Less Better?