LEARNER BEHAVIOR IN THE ONLINE CLASSROOM EXPERIENCE Developments in Business Simulation and Experiential Learning, Volume 32, 2005 LEARNER BEHAVIOR IN THE ONLINE CLASSROOM EXPERIENCE Kelli Amerine The Pennsylvania State University Great Valley School of Graduate Professional Studies kkamerine@comcast.net Denise Potosky The Pennsylvania State University Great Valley School of Graduate Professional Studies dxp16@psu.edu ABSTRACT This investigation examined the amount and quality of student interaction in the online classroom experience. Data from two 100% online courses and two hybrid courses were analyzed in terms of the duration of logged time spent online, participation in asynchronous threaded discussions, rated quality of posted messages, and performance on graded assignments. INTRODUCTION Courses that utilize Internet technology as a replacement or as a supplement to traditional classroom learning are becoming increasingly common. This movement requires that learning typically gained through classroom discussion of an issue and instructor-led activities now be realized through online interaction and no time spent in person. This paper examines the amount and quality of interaction that occurs in courses conducted entirely online as well as “hybrid” courses in which some class meetings are conducted online rather than in the classroom. Disagreement exists among authors and educators about the role of online education, and whether the goal of online education is to emulate the traditional classroom environment using Internet technology or whether online education is altogether different (Brower, 2003). This disagreement also applies to how the effectiveness of online education is evaluated. Proponents of “equivalency theory” suggest that distance courses should be designed to be equivalent to the traditional classroom experience and the more equivalent the two are, the more equivalent the learning outcomes will be (Simonson, Schlosser and Hanson, 1999). As summarized by Russell (1999), many studies have attempted to explain the effectiveness of online instruction in relation to more traditional forms of classroom instruction. Others, however, assert that distance education requires different course design and a different role for the instructor (Bigelow, 1999; Clark 2001; MacKinnon, 2000; Ponzurick et al., 2000; Thach & Murphy, 1995; Shrivastava, 1999, Tullar, Kaiser & Balthazard, 1998). In line with this assertion, Feinstein (2004) argued that instructional designers and educators can evaluate the effectiveness of online instruction based upon student achievement of learning outcomes associated with an online course, without necessarily relying on comparisons to traditional resident instruction. Whichever approach is taken to examine online instruction, however, the amount of learning that takes place between the two formats was not found to differ (Arbaugh, 2000). Brower (2003) concluded that, “although there is some resistance to web-based education, learning is not diminished by this medium, and interaction between students and faculty may be enhanced.” While the exact equivalence of online instruction and traditional classroom instruction continues to be debated, the greater flexibility for learners provided by online instruction cannot be denied. Online instruction is especially valuable in its ability to bring students together who span time zones and lack the ability to attend classes at established times and physical locations. (Brower, 2003) To capture more learners who require greater flexibility to “attend” courses, many schools are quickly moving to the web-based, online education format. Despite the growing use of distance education and online instruction, research on the measurement of the learning behaviors of students in online classes is still in its infancy. How much time do students spend online? What is the nature of interaction and learning in the online format? Instructors can observe and interpret students’ behaviors in a classroom. What happens in an online classroom? Baugher (2004) considered some of these issues in his examination of the relation of online activity (specifically, total hits and hit consistency) to course success. These findings prompted our further investigation into measures of learner behavior in the online classroom experience. Internet technology and the features of the specific courseware used for online instruction tell us what can happen in this alternative format. For example, notes and files and even textbooks can be uploaded and accessed, and printed and downloaded. Hopefully, students read and react to this information exchange, and ideally learn from it. However, such information in an online course is 1 mailto:kkamerine@comcast.net mailto:dxp16@psu.edu Developments in Business Simulation and Experiential Learning, Volume 32, 2005 comparable to the textbook, readings, handouts, and notes that are assigned or distributed in a resident course. The challenge of teaching online is primarily concerned with the creation of an online classroom experience that replicates the value of in-class interactions and instructor-facilitated activities that characterize traditional classroom instruction. To address this challenge, web-based classroom technology offers synchronous chat rooms and shared files and asynchronous message boards. Newer Internet communication technology also offers streaming video and conferencing software. Developments in Internet technology and courseware capabilities do not necessarily translate into students’ learning behaviors online, however. To say that Internet learning is effective if it includes message boards or flash animation is akin to saying that having a state-of-the art classroom produces the desired learning outcomes in a traditional course. The technology may provide a platform for learning, but it is what transpires online that creates the class experience. Shrivastava (1999) describes this as the difference between simply delivering course content online versus creating a learning community, where people learn from each other. In light of the above information concerning the Internet classroom, we can anticipate at least two areas for related research on learning and instructional design: One area focuses on the behaviors and online facilitation skills of the instructor in such courses, and the other area focuses on the interactions, behaviors, and experiences of students in the online classroom. Several authors have begun to explore the changing role of the instructor to that of a learning facilitator in a student-centered environment (Shrivastava, 1999; Ahern & El-Hindi, 2000; Clark, 2001; Paloff & Pratt, 1999). It is the latter area of learner behavior that is the focus of the present research. One important question for the present investigation concerned the overall amount of time students spend online. For courses or segments of courses that require online participation, this is often in lieu of time spent in the classroom. We sought to explore the amount of time students spent online during four different courses that offered online class meetings. Although we did not set out to compare online classes to resident instruction, we did keep in mind established norms regarding the time that would be expected of a student in a course taught in the traditional classroom format. In addition, we included data from two courses taught primarily online via the Internet and two “hybrid” courses in which only a few class meetings were conducted online versus face-to-face in a classroom. Another question for this research probed deeper into the overall amount of time spent online by examining the amount of time in which students actually interact or participate with each other and/or the instructor using asynchronous message boards. Interaction has been noted as one of the most important components of any learning experience (Dewey, 1938; Vygostky, 1978; Vrasidas & McIsaac, 1999; Yoo et al., 2002). Generally speaking, time spent in a traditional classroom is used for class discussion and instructor-led activity during which students interact with each other. When course materials and timelines are online, it is important to distinguish between the amount of time spent accessing course materials versus the amount of time spent in discussion with others. Time spent interacting in an online classroom represents the classroom experience. The third area of interest in this research is how participation in an online class might change over time. As Brower (2003) found, the instructor needs to monitor quality of the discussion in addition to the amount of interaction by each student. In our study, the asynchronous message board was the primary medium for students to engage in discussions with the instructor and their classmates. One advantage of online classrooms is that in most cases the medium employed allows one to capture what was said, by whom, and when. As suggested by both MacKinnon (2000) & Poole (2000), we developed a coding scheme to attempt to clarify the amount of “credit” each message contributed to the overall quality of the discussion. We developed three dimensions concerning the quality of message board postings made by students, and we compared the quality of postings to a message board early in each of four different courses to the quality of postings to a message board later in the course. We anticipated that these comparisons would enhance our understanding of how familiarity with an asynchronous online format develops over time during a course, but also how the nature of students’ participation in a course might change as facilitated online discussion progresses throughout a term (or semester). Finally, this research examined the relationship between online participation during the course and performance in the course. For each course examined in our research, participation was a percentage of the student’s final overall grade. We considered not only the final grade which included this percentage, but also grades on related assignments that were dependent on the learning gained from the interaction rather than the simple act of participating. It is important to note, however, that although we explored this relationship, given the difficulties associated with establishing a causal relationship between these variables in resident courses, we did not anticipate a significant positive or predictive relationship for online courses. That is, the difficulties associated with relating classroom interaction and experiential learning activities to learning outcomes and performance in traditional classrooms may be extended to the online classroom experience. In sum, this research examined four online courses for which data were available in an effort to address the following four questions: 1. How much time do students spend in an online course? 2. How much time do students spend interacting/participating online? 2 Developments in Business Simulation and Experiential Learning, Volume 32, 2005 3. Does participation using an online message board (MB) change over time during a course? 4. Does online participation relate to performance in a course? METHOD For this research, student data from four graduate M.B.A. courses offered at a large northeastern university were examined. All of the courses included in this research were taught by the same instructor. Two of the courses included in this research required online student participation during the entire term of the course. The first of these, an organizational behavior course, was an eight- week course conducted in Spring 2003 with 25 students. The second, a human resource management course, was conducted in Fall 2003 over twelve weeks with the same 25 students that took the organizational behavior course. Both courses specified a percentage of the final grade based on student online participation in discussions (10% and 15%, respectively). A description of “A”, “B” and “C” level participation was provided to students on the course syllabus. In addition, it should be noted that the same students participated in both courses, with a required week of residency occurring toward the end the first course in May, 2003. Two other graduate M.B.A. courses were also examined in this research. Both were seven-week resident courses in human resource management. One section of this course was offered in Spring 2003 with 22 students, and a separate section of this course was offered in Fall 2003 with 25 students. These sections comprised a hybrid model in which students elected to meet asynchronously online over four days in lieu of meeting in class for three hours. In Spring 2003, students elected to have five out of 14 class meetings online, and in Fall 2003 students elected to have only two out of 14 class meetings online. A portion of each student’s final grade (5% in this case) was also earned based on class participation for these hybrid courses. Students were provided a description of what “A”, “B” and “C” level participation consisted of including both class and online interaction. The variables examined from the courses described above are as follows: Number of logins: The actual number of times each student logged in to each lesson and within the total course. Duration of logins: The actual amount of time each student remained online within each lesson and within the total course. Number of message board postings: The number of times each student posted a message or replied to a message within the course message boards. Within each course, individuals were assigned to teams for project purposes within the course. Team members could utilize the Angel message system to communicate regarding team projects. These messages were subtracted from the message board posting calculations as they did not pertain to the lesson discussion topics. Amount of time spent on message board: The amount of time each student spent in online ‘conversation’ throughout the course. Message quality measures: The extent to which each message posted by a student met three criteria utilized for message quality: Relatedness, Involvement, and Size. (1) Relatedness – the extent to which the posted message pertained to the actual lesson topic as well as the current discussion string within the lesson. (2) Involvement – the extent to which the posted message responded directly to the instructor, to a small group or specific individuals within the course, or encouraged participation of all class participants. (3) Size – the number of sentences within a posted message. In order to derive the data needed for our areas of focus within this research, first we examined the data available from each course. The university offering the courses in this study utilizes a web-based course management system, which provides a variety of measurement and reporting based on students’ actions while online. We gathered data from this system on the total number of logins by student, total login duration by student, total message board activity by student and daily activity log by student for each of the courses. Based on the calendar and syllabus established for each course, dates for each lesson were determined and applied to students’ daily login durations and message board activity. While student assignment and final grade data is also contained within the course system, this data was checked against the final grade data held by the course instructor. In addition to the course data mentioned above, web- based course management system also provided the ability to view entire message boards, including the message content, author, date and time. As part of this research, we wanted to look at the content and quality of the posts on the message boards in these courses. Since two of the courses were completely online and the other two conducted only few lessons online, we decided to select two lesson message boards from each course to further examine for message quality. Based on this, we applied the following criteria to select lesson message boards within the two courses conducted completely online: Sequence within the course. One lesson was selected from the beginning of the course and one from the second half of the course so that we could examine how participation changed over time, if at all. Relationship to a graded assignment. Lessons selected were required for a graded assignment that followed, in order to allow us to look at performance 3 Developments in Business Simulation and Experiential Learning, Volume 32, 2005 on the graded assignment as well as overall performance in the course. To begin to assess the quality of each message, three variables were created as possible indicators of message quality: relatedness, involvement, and size. The first of these criteria, relatedness, specified the degree to which the message was related to both the lesson topic as well as the current discussion string within the message board. Items were rated for relatedness along the following 4-point scale: 4 = message that clearly demonstrated ongoing participation and comprehension in the discussion, 3 = message provided an example of a previously expressed point in the discussion, 2 = message provided input or reply to the current discussion without examples or evidence of comprehension of past discussion postings, and 1 = message with no clear relation to the current topic or discussion string. The second criteria, involvement, employed the following four-point scale: 4 = message encouraged interaction and discussion with all class participants and the instructor, 3 = message interacted with the instructor and one or several other designated participant), 2 = message pertained to an exchange with only the instructor, and 1 = message showed no potential further interaction with anyone. The third criteria, size, measured the actual number of sentences within each message board posting. In this case, three ratings were used to designate each message as 1 = small (1-3 sentences), 2 = medium (4-7 sentences), or 3 = long (8 or more sentences). All messages within the designated message boards were rated based on the relatedness, involvement and size criteria by the same individual. In total 744 messages were reviewed, with 178 from the online organizational behavior course, 360 from online HRM course, 90 from the first hybrid HRM course and 116 from the second hybrid HRM course. Each message was assigned a message ID code and a rating for each category. RESULTS Our first research question focused on how much time students spend online in online and hybrid courses. Table 1 shows means, standard deviations, and ranges of logged minutes spent in four separate M.B.A. courses, two of which were conducted fully online. The first course considered was an organizational behavior course that spanned nine weeks, including a one-week residency period (Week 8) during which time students actually met in person for the first time after completing two fifteen-week semesters of their online M.B.A. program of study. The second online course, a human resource management course, was comprised of the same students who took the organizational behavior course. On average, these students spent approximately 177 minutes logged into the course per week. It is important to note that during both terms, these students were enrolled in a total of three online M.B.A. courses, each worth three credit hours. It may be safe to speculate that the overall total log time (1732.46 minutes and 1906.29 minutes, respectively) represents approximately one-third of the time these students spent online each terms. Table 1. How much time do students spend online? Total Login Time by Week 200 400 600 800 1000 1200 1400 1600 1800 Wk 1 Wk 2 Wk 3 Wk 4 Wk 5 Wk 6 Wk 7 Wk 8 Wk 9 Wk 10 Wk 11 Wk 12 0 4 Online Org Beh Online HRM Hybrid HRM-A Hybrid HRM-B 447 - 3766 460 - 5987 732 - 8513 1063 - 17510 Developments in Business Simulation and Experiential Learning, Volume 32, 2005 Table 1 also shows the login time for students who participated in only a few online class meetings as part of hybrid courses. Note that most students logged into the course web site during each week of class throughout a seven-week term. This is because course materials (handouts, notes, assignment dropboxes, etc.) were made available to these students online even when the class met in person during the seven-week term. However, in one hybrid course, students participated in five online class meetings and spent an average of 651 minutes logged into the system, and in the other hybrid courses examined students participated in two online class meetings and spent an average of 1444 minutes logged on. Examination of logged time in the system can be considered in relation to unlogged time in a course. That is, all of the classes considered were three-credit M.B.A. courses which required 37.5 hours (or 2250 minutes) of class time. In light of this fact, students in the fully online courses clearly spent less time “in class” than they would have if taking the course in residence. In terms of classroom hours, students in the hybrid courses would have spent 150 minutes in each class meeting. When focusing on the online class meetings in the hybrid courses, Table 1 suggests that students spent more time online than they would have in the classroom (e.g., 150 X 5 class meetings online = 750 minutes vs. 4109 minutes, and 150 X 2 class meetings = 300 minutes vs. 7137 minutes online). Comparison of time logged into a course provides only a rough proxy for class time, however, because logged time includes activities that typically occur outside of the classroom (e.g., accessing notes, checking the syllabus, uploading files, etc.). Our second research question focused more directly on the equivalence of “classroom time” or interaction between students and the instructor. All formal classroom interaction in these four courses occurred using asynchronous message boards (i.e., online threaded discussions). That is, no chat rooms, e-mail exchanges, Internet conferencing, or other means of interaction that included the instructor were counted toward “attendance” in an online class meeting, and students understood this at the outset of each course. In every online class meeting, the instructor provided discussion questions and facilitated discussion within threads by commenting on postings and replies made by students, by asking secondary questions, and by referring to additional information. Also, the courseware used in all four courses considered in this investigation allowed students to generate their own threads (by changing subject headings) and to upload attachments to their message board postings (e.g., if they wished to show or present something to the class). Table 2 shows the number of postings (including new postings as well as replies) for each course analyzed. In the fully online organizational behavior course, each student posted an average total of 4.27 messages per week of class, for a total of approximately 30 messages throughout the online term. In the fully online HRM course, each student posted an average total of 6 messages per week of class, for a total of approximately 66.5 messages throughout the online term. In the hybrid courses, on average, each student posted a total of 2.25 messages per online class meeting for the first HRM course, and students in the second hybrid HRM course posted an average of 2.4 messages per online class meeting. The larger number of postings per student in the fully online courses may reflect greater familiarity with as well as dependency upon the online message board format for exchanging information and opinions “in class.” Examination of the mean number of posts per student for each week in each course also partially addressed our third research question regarding how participation might change over time. The means by week in Table 2 suggest that in the fully online courses, the amount of student participation via the message boards varied from week to week. For the organizational behavior course, an upward trend in the average number of posts per student (Weeks 1 through 6) can be observed (i.e., starting out at around 4 posts per week, going down to 2.5 posts in week 3, but then increasing to between about 4 and 6 posts per week). The HRM online course was similarly variable, but little upward trend in terms of the number of posts was observed (Weeks 1 through 10). Keeping in mind that the same students took the fully online HRM course in a term subsequent to the online organizational behavior course, posts to the online HRM course may reflect the comfortable range of participation for these students (i.e., between 5 and 8 posts or replies per week). In contrast, the mean number of posts in the hybrid course showed little variation or progression in terms of increased interaction online. As noted above, students posted between 2 and 3 messages, on average, per online class meeting in the hybrid course. Results pertaining to another approach to our third research question are shown in Table 3. We compared participation (# of MB posts) regarding one lesson presented early in the course (represented by a specific week’s message board) to students’ participation regarding another lesson later in the course. In the fully online courses, we selected the MB associated with Lesson 2 (Week 2) to the fourth or fifth MB in the course. These specific MBs were selected because each of these lessons was directly relevant to two more or less equivalent graded assignments required of all students. That is, these message boards should have represented an appropriate time for students to participate in an online class in an effort to prepare for an assignment. For the hybrid courses, we compared postings for the first and fourth MBs for one course and the only two MBs available for the second course. As described in our methods section above, we applied the relatedness, involvement, and size ratings to these MBs. For all but the one hybrid course that had only two online class meetings, paired t-test results suggested that relatedness ratings were greater for the later MB than for the earlier MB. Comparisons associated with the involvement ratings were more difficult to interpret. For the hybrid courses, students’ involvement did not differ from the early MB to the later MB. However, for the fully online courses, involvement 5 Developments in Business Simulation and Experiential Learning, Volume 32, 2005 was significantly greater in the later MB than in the earlier MB for the organizational behavior course (paired t = 2.10, p < .05), but lower in the later MB for the online HRM course (paired t = -1.88, p < .1). The rated size of messages posted did not differ for the two MBs compared in each course. TOTA 0 2 3 4 5 6 7 8 9 Wk 1 Wk 2 Onlin Lessons Compar Relatedness Size Involvement Note. Eac three criteria: R message board the course. Res * p < .05, † p < Table 2. How much do students interact online? L POSTS TO MESSAGE BOARDS BY WEEK 1 Wk 3 Wk 4 Wk 5 Wk 6 Wk 7 Wk 8 Wk 9 Wk 10 Wk 11 Wk 12 e Org Beh Online HRM Hybrid HRM-A Hybrid HRM-B 1 2 4 3 h 6 - 8 Table 3. Does P Onl Org Beh ed 2 vs. 4 t 2.5 * .70 2.10* posting on two elatedness, size, were subtracted f ults for a paired t .1 28 - 15 articipation Chan ine Courses HRM 2 vs. 5 t 1.91† M=.17 .30 M=.07 -1.88† M=.09 separate message b and involvement. R rom ratings for pos comparison, two-ta 6 5 - 2 ge Over Time? Hybrid C HRM 1 vs. 4 t 2.25* M=.28 -2.04* M=-.26 .12 M=.02 oards were ratings atings for postings tings on a message iled, are shown. 2 - 1 ourses HRM 1 vs. 2 t -.53 M=-.06 .38 M=.05 .90 M=.14 according to on the earlier board later in Developments in Business Simulation and Experiential Learning, Volume 32, 2005 Our fourth research question focused on the relationship between online participation and performance outcomes associated with the course. Although at first glance this analysis might seem straightforward, it was not. For one thing, 20 percent of students’ overall grades in each course was based upon an overall class participation grade. As a result, the number of MB postings and, to a lesser extent, time spent online was confounded with overall course grades. In the hybrid courses, online participation was not considered separately from in-class participation when participation was evaluated for a grade. Also, in the 100% online courses, as in the hybrid courses, there is no optimal method for aggregating overall online participation throughout the term. Each MB may have reflected different students’ participation, interest, and online interaction skill to different degrees. Further, to evaluate MB participation for each student for every message posted (in terms of the three dimensions proposed) would require intensive time and effort. Table 4. Does Online Participation Relate to Performance?(Correlations between Measures) VARIABLES n Log of Grade for A Log of Grade for B Log of Team Grade Log of Final Course Grade Mean Related Rating Mean Involve Rating Mean Size Rating Online Org Behavior MB A Minutes 22 .32 -.25 .13 .54** -.22 .15 -.18 MB A Posts 22 .11 -.09 -.47* .35 -.14 .24 -.47* MB B Minutes 22 .31 .29 -.02 .56** .16 -.00 -.10 MB B Posts 22 .19 .03 -.08 .50* -.06 -.16 -.22 Online HRM MB A Minutes 24 .20 .45* .36 .44* -.20 .06 -.09 MB A Posts 24 .19 .41* .51* .58** -.22 -.46* -.41* MB B Minutes 24 .21 .29 .47* .47* -.40* -.15 -.12 MB B Posts 24 .02 .25 .57** .49* -.38 -.12 .05 Hybrid HRM 1 MB A Minutes 18 -.17 -.07 .01 -.21 -.17 -.18 -.18 MB A Posts 18 .23 -.07 .22 .31 -.11 .44† .38 MB B Minutes 16 .20 .27 .02 .27 -.69** -.05 -.70** MB B Posts 16 .44 .41 .30 .55* -.41 -.32 -.49† Hybrid HRM 2 MB A Minutes 20 ..01 .34 -.07 .21 -.17 -.09 -.06 MB A Posts 20 .17 .13 -.22 -.18 -.27 -.35 .03 MB B Minutes 23 .06 .17 -.42* -.13 -.41* -.09 -.16 MB B Posts 23 .14 .33 .02 .22 .11 .04 .28 Notes. Grades were transformed using the natural logarithm prior to correlation analysis. Grade A represents an individual graded assignment most closely related to an early message board in the course (MB A). Grade B represents an individual graded assignment most closely related to a message board class meeting that occurred later in the course (MB B). The team grade represents a graded team paper assignment. ** p < .01, * p < .05, † p < .10 7 Developments in Business Simulation and Experiential Learning, Volume 32, 2005 Given the above considerations, we elected to consider the correlation between online class participation and available performance measures for the two MBs evaluated above in our third research question. That is, for each MB, we considered logged minutes per student and the number of MB postings per student in relation to the following outcome variables: grades on an individual assignment that was most closely related to the first comparison MB, grades on an individual assignment most closely related to the second comparison MB, grades on a team paper assignment, final grades in the course, the mean of rated relatedness, the mean of rated involvement, and the mean of related message size. The natural logarithm was used to transform grades prior to further analyses. The correlations between logged MB time and number of posts and the outcome variables are shown in Table 4. The information presented in Table 4 does not suggest a particularly strong relationship between MB participation and the specified performance measures. The strongest relationships were observed for the fully online HRM course, which consisted of students who had already worked together on the organizational behavior course. In the fully online HRM course, MB performance was significantly and positively correlated with team paper grades. It is difficult to interpret the positive correlations shown for MB minutes and posts and final course grades, since final grades for all courses examined included a participation grade. One surprising result shown in Table 4 concerns the number of significant negative correlations between minutes spent online and the number of MB posts and average ratings for relatedness, involvement, and message size. This result is discussed below in terms of study limitations. DISCUSSION We examined detailed message board results for four courses, two 100% online and two hybrid courses. Overall, one of the most interesting observations associated with study results is that the students in the fully online courses spent approximately 81% of the time they would have spent in resident classrooms online. This may point to the possibility that students in online courses do not participate for an equivalent quantity of course participation in the online format as opposed to the classroom experience. Since logged time online does not accurately reflect what students were reading or paying attention to while logged on, this may be a conservative comparison estimate. In contrast, students in hybrid courses, which included only a small number of online class meetings, tended to log more minutes online than they would have if they attended a resident class meeting. Part of this may be due to the unfamiliarity of the online format. In terms of the number of messages posted, students in the fully online courses posted approximately four to six messages per “class week,” whereas students in hybrid courses posted only about 2.5 messages per online class meeting. It is worth noting, however, that since students in the hybrid courses met online for four day periods, whereas each lesson spanned one full week in the fully online courses, it may be that in M.B.A. classes of 20-24 adult students, most students will post approximately five messages (postings and replies) per week online. We developed ratings associated with the relatedness of a posted message to the topic of a discussion thread, the involvement among the MB participants (i.e., other students and the instructor) generated by a posted message, and the size (i.e., number of sentences) of each message posted. We applied these ratings to two message boards, one early in the course and one later in the course, for each course considered in this study. For most courses, we found that message relatedness tended to improve from early to later message boards. This result may suggest that there is a learning curve associated with the use of message boards. As students become more familiar with the online format and how to contribute to a discussion thread, the relatedness of their postings seems to improve. This finding can be utilized to include activities and discussions at the start of an online course in order to achieve a high degree of message relatedness to the discussion thread early in the course to support student learning. However, the involvement in a discussion thread did not seem to develop or change in any consistent manner over time in a course. It is possible that graduate students may not be as aware of their contributions to class discussion and the pace and timing associated with a conversation when posting messages to a MB. This criteria should be further refined for inclusion in a future study to produce data that can be meaningful. The number of sentences posted did not vary from early MBs to later MBs in a course. The correlations observed between MB postings and time spent online within MBs and performance outcomes such as grades and the ratings we applied to each message posted were not conclusive. Part of the problem may be the small sample sizes in each course used to calculate these correlations. However, it is important to note that it is difficult to establish that participation or even attendance in classroom instruction relates positively to performance, given the many other factors that could explain overall performance in a course. This difficulty is present regardless of whether class interaction occurs in residence or online. In addition to the small class sizes, it is also important to note some of the other limitations of this study. First, although the ratings we developed appeared to be fairly relevant to posted messages and independent of each other, the distinction between relatedness and involvement needs further study and development as a more precise measure. Second, the fact that only one person rated the messages in all courses limits our ability to estimate the reliability of these measures. Finally, the fact that the same instructor taught all four courses limits the generalizability of our results. 8 Developments in Business Simulation and Experiential Learning, Volume 32, 2005 Based on our examination, we can suggest several related areas for future research. The first of these involves other factors that contribute to student performance in online courses, such as student proficiency with the Internet and the web-based course technology, past experience taking online courses, and the impact of a residential component during an online learning program. The two courses examined here that required complete online participation and the same group of students required a week of residence which occurred between the two courses. Another area for future research consideration is the extent to which collaboration between students occurs in an online course and the affect it has on student performance in team-based assignments. Each of the courses in this study had both individual and team graded assignments; however, only the individual assignments were considered in our work. Finally, opportunity for more research exists regarding ways to increase student participation in online learning environments and on the impact of instructor entries to course message board participation (i.e., whether a question posed by the instructor encourages direct response to the instructor or whether participants continue to interact with all class participants in total). Overall, this study introduces new information and considerations for analyzing “the classroom experience” for online and hybrid courses. Our findings extend Baugher’s (2004) consideration of what measures of online activity are predictive of student course success. Few studies to date have examined similar data, and no studies, to our knowledge, have provided the depth of analysis required to document the characteristics of online interaction and participation in a way that can guide future expectations for research in this area. Our results can be used to frame expectations for what constitutes typical online participation in M.B.A. courses. These results may also be useful for establishing minimum expectations for contributions to message boards for instructors who teach classes online. REFERENCES Ahern, T. C., & El-Hindi, A. E. (2000). “Improving the instructional congruency of a computer-mediated small- group discussion: A case study in design and delivery.” Journal of Research on Computing in Education, 32, 385-200. Arbaugh, J. B. (2000). “Virtual classroom versus physical classroom: An exploratory study of class discussion patterns and student learning in an asynchronous Internet-based MBA course.” Journal of Management Education, 24(2), 213-234. Baugher, D., Butler, R., Morgan, S., & Varanelli, A. (2004). “Implementing Distance Approaches to Education; A Panel Discussion for ABSEL Las Vegas, 2004.” Developments in Business Simulation and Experiential Learning, 31, 62-65. Bigelow, J. D. (1999). The web as an organizational behavior learning medium.” Journal of Management Education, 23, 635-650. Brower, Holly H. (2003). “On Emulating Classroom Discussion in a Distance-Delivered OBHR Course: Creating an On-Line Learning Community.” Academy of Management Learning and Education, 2(1), 22-36. Clark, L. J. (2001). “Web-based teaching: A new educational paradigm.” Intercom, 48, 20-23. Dewey, J. (1938). Experience and education. New York: Collier Macmillan. Feinstein, Andrew H. (2004). “A Model for Evaluating Online Instruction.” Developments in Business Simulation and Experiential Learning, 31, 32-39. MacKinnon, G. R. (2000). “The dilemma of evaluating electronic discussion groups.” Journal of Research on Computing in Education, 33, 125-132. Palloff, R. M., & Pratt, K. (1999). Building learning communities in cyberspace. San Francisco: Jossey- Bass. Ponzurick, T. G., France, K. R., & Logar, C. M. (2000). “Delivering graduate marketing education: An analysis of face-to-face versus distance education.” Journal of Marketing Education, 22, 180-188. Poole, D. M. (2000). “Student participation in a discussion- oriented online course: A case study.” Journal of Research on Computing in Education, 33, 162-177. Russell, T. L. (1999). The No Significant Difference Phenomenon. Raleigh, NC: Office of Instructional Telecommunications, North Caroline State University. Shrivastava, P. (1999). “Management classes as online learning communities.” Journal of Management Education, 23(6), 691-703. Simonson, M., Schlosser, C., & Hanson, D. (1999). “Theory and distance education: A new discussion.” American Journal of Distance Education, 13, 60-75. Thach, E. C., & Murphy, K. L. (1995). “Competencies for distance education professionals.” Educational Technology Research and Development, 43, 57-59. Tullar, W. L., Kaiser, P. R., & Balthazard, P. A. (1998). “Group work and electronic meeting systems: From boardroom to classroom.” Business Communication Quarterly, 61, 53-65. Vrasidas, C., & McIsaac, M. S. (1999). “Factors influencing interaction in an online course.” The American Journal of Distance Education, 13, 22-36. Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press. Yoo, Y., Kanawattanachai, P., & Citurs, A. (2002). “Forging into the wired wilderness: A case study of a technology-mediated distributed discussion-based class.” Journal of Management Education, 26, 139- 163. 9 Table of Contents Volume 32, 2005 LEARNER BEHAVIOR IN THE ONLINE CLASSROOM EXPERIENCE THE EFFECTIVENESS OF A SIMULATION EXERCISE FOR INTEGRATING PROBLEM-BASED LEARNING IN MANAGEMENT EDUCATION DEMONSTRATION OF FOUR WEB-BASED SIMULATIONS THRESHOLD COMPETITOR: A MANAGEMENT SIMULATION ENTREPRENEUR: A NEW VENTURE SIMULATION MERLIN: A MARKETING SIMULATION MICROMATIX: A STRAGETIC MANAGEMENT SIMULATION LEARNING STYLES INFLUENCES ON SATISFACTION AND PERCEIVED LEARNING: ANALYSIS OF AN ONLINE BUSINESS GAME INTERNATIONAL INTERNSHIPS: DESIGN AND EXPERIENCES SIM MAP: TURNING ACTION-BASED LEARNING INTO SIMULATED CONSULTING PROJECTS NOTEL HEALTH SERVICES: A ROLE-PLAYING SIMULATION TEACHING EXPERIENTIALLY WITH THE MADELINE HUNTER METHOD: AN APPLICATION IN A MARKETING RESEARCH COURSE SIMULATING CUSTOMER LIFETIME VALUE: IMPLICATIONS FOR GAME DESIGN AND STUDENT PERFORMANCE VIRTUAL PROGRESS: SIMULATING ECONOMIC DEVELOPMENT ONLINE STRATEGIC MANAGEMENT: AN EVALUATION OF THE USE OF THREE LEARNING METHODS IN CHINA ADOPTION OF DISCUSSION-BASED TEACHING AND ASSESSMENT IN TEACHING STRATEGIC MANAGEMENT IN CHINA STUDENTS' VIEW ON THE USE OF CASE METHOD IN CHINA CHINESE STUDENTS' PERCEPTIONS OF BUSINESS GAMING EXPERIENCECSR - A CORPORATE SOCIAL RESPONSIBILITY SIMULATION CREATING DYNAMIC INTERACTION IN A VIRTUAL WORLD: ADD VALUE TO ONLINE CLASSROOMS THROUGH LIVE ELEARNING AND COLLABORATION: A DEMONSTRATION CAPABILITIES OF EXPERIMENTAL BUSINESS GAMING A COMPARISON BETWEEN SOLUTIONS AND DECISIONS IN A BUSINESS GAME VALIDATING BUSINESS SIMULATIONS: DOES HIGH PRODUCT QUALITY LEAD TO HIGH PROFITABILITY? ALIGNING ART AND EPISTEMOLOGY: ILLUSTRATIONS TO DISTINGUISH DISCOVERY FROM KNOWLEDGE BUILDING TUTORIALS USING WINK STUDENTS AS LAB RATS: THE ETHICS OF CONDUCTING NON-PEDAGOGICAL RESEARCH IN THE CONTEXT OF CLASSROOM SIMULATIONS AND EXPERIENTIAL LEARNING THE EFFECT ON GAME PERFORMANCE OF DIFFERENT MEASURES AND UNITS OF ANALYSIS IN QUANTITATIVE ANALYSIS ANALYZING AND THINKING WHILE PLAYING A SIMULATION COMPUTER BUSINESS SIMULATION DESIGN: THE ROCK POOL METHOD EXPANDING THE ROLE OF E-ROOMS IN DISTANCE LEARNING APPLICATIONS TO MANAGEMENT EDUCATION APPLICATION OF TRADITIONAL AND ONLINE JOURNALING AS PEDAGOGY AND MEANS FOR ASSESSING LEARNING IN AN ENTREPRENEURIAL SEMINAR DEVELOPING MANAGERIAL EFFECTIVENESS: ASSESSING AND COMPARING THE IMPACT OF DEVELOPMENT PROGRAMMES USING A MANAGEMENT SIMULATION OR A MANAGEMENT GAME INTERNATIONAL MANAGEMENT GAME Œ AN INTEGRATED TOOL FOR TEACHING STRATEGIC MANAGEMENT INTERNATIONALLY STUDENT EXPECTATIONS OF SIMULATIONS DISTANCE EDUCATION DELIVERY OF AN INTENSIVE SIMULATION BASED COURSE TEACHING SERVICE LEARNING USING A BUSINESS GAME ROLE-PLAY SIMULATION EDUCATIONAL PERSPECTIVE OF COLLABORATIVE VIRTUAL COMMUNICATION AND MULTI-USER VIRTUAL ENVIRONMENTS FOR BUSINESS SIMULATIONS SIMULATION PERFORMANCE & PREDICTOR VARIABLES: ARE WE LOOKING IN THE WRONG PLACES TO MEASURE THE RIGHT LEARNING? VIDEO CASE: JET-A-WAY INC. Œ FOCUSING ON DIVERSITY AND ENTREPRENEURIAL LEADERSHIP ACTIVE LEARNING: WHAT IS IT AND WHY SHOULD I USE IT? FACILITATING THROUGH COLLABORATIVE REFLECTIONS TO ACCOMMODATE DIVERSE LEARNING STYLES FOR LONG-TERM RETENTION ONLINE CUMULATIVE SIMULATION TEAM PERFORMANCE PACKAGE WHEN PROPHECY FAILS: A SMALL SAMPLE, PRELIMINARY STUDY USING EXPERIENTIAL LEARNING TO INTEGRATE THE BUSINESS CURRICULUM FORECASTING STOCK VALUE EMPLOYING PROGRESSIVE PRACTICES AND PRINCIPLES TO FACILITATE SEMINAR ROOM LEADERSHIP AMONG LEARNERS: SHARED POWER AND COLLECTIVE ACCOUNTABILITY INDIVIDUAL ACHIEVEMENT DOES NOT GUARANTEE TEAM PERFORMANCE: AN EVIDENCE OF ORGANIZATIONAL LEARNING WITH BUSINESS GAMES DECISION MAKING IN BUSINESS SIMULTION DESIGN ZUG UM ZUG 2015: COLLECTIVE BARGAINING AS A TWO-LEVEL GAME A NEW METHOD FOR MODELING INNOVATION AND R&D IN BUSINESS SIMULATIONS: ILLUSTRATED WITH A SIMULATION OF A NEW PRODUCT DEVELOPMENT PORTFOLIO DEVELOPING A MICRO SIMULATION EFFECT OF MARKET SHARE AND PRODUCTION EXPERIENCE ON COMPANY PROFITABILITY RE-DESIGNING A CURRICULUM THAT VALUES A WORK-INTEGRATED APPROACH TO STUDENT LEARNING HOW SIMULATIONS AND EXPERIENTIAL LEARNING FIT AS WE COMPLY WITH LEGISLATIVE AND AACSB ASSESSMENT GUIDELINES: HOW TO DEVELOP ACADEMICALLY SOUND COURSES THAT ALSO MEET STAKEHOLDER NEEDS EVALUATING SERVICE LEARNING: REFLECTION AND ASSESSMENT FROM THE STUDENT POINT OF VIEW SIMPLIFYING AND ENHANCING FINANCIAL ANALYSIS IN CASES AND SIMULATIONS OVERCOMING THE BUSINESS GAME COMPLEXITY PARADOX EXPLORING THE PREFERENCE IN LEARNING APPROACH AMONG THE HONG KONG UNIVERSITY STUDENTS: CASE STUDY, PROBLEM-BASED OR TRADITIONAL TEXTBOOK QUESTION AN EXERCISE FOR EXPLORING THE RELATIONSHIP BETWEEN JUNGIAN PSYCHOLOGICAL TYPES AND POLITICAL STYLE IN THE WORKPLACE EVALUATING THE DIRECTION OF RESEARCH IN ONLINE EDUCATION: ARE WE GOING ANYWHERE? AIS RAIL SYSTEM: A COMPUTER-BASED JOB-ORDER COST SIMULATION BLOOM BEYOND BLOOM: USING THE REVISED TAXONOMY TO DEVELOP EXPERIENTIAL LEARNING STRATEGIES DELIVERING A TECHNOLOGY-BASED CASE IN A TECHNOLOGICAL WAY: THE SMARTCART CASE USING THE INTERNET TO ENHANCE COURSE PRESENTATION: A HELP OR HINDRANCE TO STUDENT LEARNING TEACHING PRACTICES: A CLUSTER ANALYSIS OF STUDENTS IN HONG KONG TEACHING PRACTICES: A CLUSTER ANALYSIS OF TEACHING STAFF IN HONG KONG EVALUATING A SIMULATION WITH A STRATEGIC EXPLORATION TOOL A SYMBOLIC MODEL OF THE SIMULTANEOUS ACHIEVEMENT OF CONCRETE BENEFITS AND LEARNING BY PARTICIPATING GROUPS IN EXPERIENTIAL ACTIVITIES: ‚THE SPHERE OF EXPERIENTIAL LEARNING