A MODEL FOR ONLINE EDUCATION DELIVERY AND A LOOK AT ONLINE DELIVERY EFFECTIVENESS Developments in Business Simulation and Experiential Learning, Volume 30, 2003 A MODEL FOR ONLINE EDUCATION DELIVERY AND A LOOK AT ONLINE DELIVERY EFFECTIVENESS Jerry Gosen University of Wisconsin-Whitewater gosenpuj@uww.edu ABSTRACT This paper deals with delivery of education online. It has three connected purposes. The first is to survey and summarize the research up until today, in particular that which pertains to the efficacy of internet learning. The second purpose is to present a scheme categorizing the kind research that could and should be done to assess internet delivery effectiveness. This categorization will serve to suggest what kinds of research should be undertaken to gain a more complete understanding of the relative effectiveness of teaching online and a better understanding of how to attain effective outcomes. The third is to provide research- design suggestions to help internet delivery effectiveness research become more credible. INTRODUCTION Internet education is now an established phenomenon and a growth industry. While web-based courses were a rarity only a decade ago, they have become an accepted means of providing higher education, and their popularity is increasing. According to Duvall (2000) for example, in 1999 at least 1.5 million internet courses were available through 3,000 different institutions, and Clark (1999) reported that by 1999, two-thirds of accredited 4 year and graduate universities offered courses via the internet. There is ample enthusiasm for learning via this medium (Cox, 2000, Elram & Easton, 1999, Gibson, Tesone, & Blackwell, 2001). It is convenient for students, it helps to increase enrollment for universities, and the initial evidence suggests that the medium, when compared to the face-to- face classroom, does at the worst, equally well in facilitating student performance (Arbaugh & Hilz, in press). But there are also concerns. Course development and course delivery are both time consuming and labor intensive. Face-to-face interaction is lacking, and there are questions about quality compared to classroom-based delivery (Flaherty, Pearce, and Rubin, 1998, Gilbert & Moore, 1998). The pedagogy is controversial. Given these, Grossman (1999) contends that the field may be racing to adopt educational techniques without fully understanding them. As a result there is a clear call for research studies assessing the effectiveness of course delivery on the internet. This paper has three connected purposes. The first is to survey and summarize the research up until today, in particular that which pertains to the efficacy of internet learning. The second purpose is to present a scheme categorizing the kind research that could and should be done to assess internet delivery effectiveness. This categorization will serve to suggest what kinds of research should be undertaken to gain a more complete understanding of the relative effectiveness of teaching online and a better understanding of how to attain effective outcomes. The third is to provide research-design suggestions to help internet delivery effectiveness research become more credible. PREVIOUS RESEARCH Internet learning delivery is ripe with research opportunities, and there are a plethora of studies exploring many issues and potential relationships among variables. But most of the studies performed consist of case studies or produce anecdotal evidence and simple findings, such as the result that students were satisfied with a particular aspect of a course. This summary will for the most part ignore that kind of research. Instead it will present research where analytic relationships are explored. It reveals streams of research where trends appear, where more than one study finds a similar result, where there is little or no contradictory data, and where conclusions can begin to be drawn. Much of the available research that assesses Internet education attempts to measure whether students learn with online delivery, and the majority of studies undertaken have found that students perform on learning tasks at least as well in internet classes as in face-to-face classes. This has been particularly true for information system classes (Alivi et al., 1997; Hilz et al., 2000; Sankaran & Bui, 2001) and science classes (Leasure et al, 2000; Rickets et al, 2000). In business classes, Arbaugh (2000b) found no significant differences between test scores using a multiple-choice tests between students from virtual and physical MBA classrooms. However he did find that post test scores from students in the internet sections were significantly higher than pre-test scores, while such differences were not 279 mailto:gosenpuj@uww.edu Developments in Business Simulation and Experiential Learning, Volume 30, 2003 significant for classroom students. Finally, Warkinton, Sayweed & Hightower (1997) performed a study comparing Internet teams with face-to-face teams instead of Internet classes with face-to-face classes. Solving a mystery puzzle was the task to be accomplished, and both groups were equally effective in solving the puzzle. The results from these studies suggest that the Internet interaction is as effective as face-to-face interaction, at least for some tasks. When compared to face-to-face-alone delivery, classes combining online with face-to-face delivery also show performance-related superiority. In an undergraduate Sociology course which met face-to-face, Althus (1997) found that students who participated in asynchronous communications performed better than those who did not, and Thoennessen et al. (1999) found that undergraduate Physics students from a large class supplemented with online discussions had 10% higher course grades and missed 12% fewer classes than students from previous traditional classes. Also, Hartman, Dziuban, & Moskal (2000) surveyed 36 mixed-mode undergraduate courses and found that students from hybrid classes out performed students from classroom only settings. One of the claims for internet classes is that they generate interaction and learning from interaction as well or better than face-to-face classes, and the evidence supports the claim. Alavi (1994) and Borthwick & Jones (2000) found that internet delivery encourages collaborative learning and Arbaugh (2000b) found significantly greater participation for an internet-based section than in a classroom-based section. Not all studies compare internet classes with those that meet face-to-face. There are streams of research that focus on predictors of outcomes in online classes. For example, there is evidence (Arbaugh, 2002; Webster and Hackley, 1997) that outcomes (primarily satisfaction and perceived learning) are better in distance learning environments when the media is richer (multi-faceted). There is also evidence that immediacy of instructor interaction, i.e., communication behaviors that reduce psychological distance between people (Myers, Zhong & Guan, 1998), is positively associated with student learning and satisfaction with format (Arbaugh, 2002; Commeaux, 1995; Frietas, Myers & Avtgis, 1998). There is also research that deals with class size. According to Hiltz & Wellman(1997) and Arbaugh & Durray (2001), class size is negatively associated with learning for classes of 50 and greater. Discussion. The research available dealing with online education yields few conclusions, and many of the above studies are methodologically flawed. In addition, the conclusion that students from online classes do as well on tests as students from physical classrooms is hardly earthshaking. On the other hand, internet delivery advocates can be comforted with data that suggests student can perform as well taking internet classes as they can in a face-to-face environment. Also the predictive studies are valuable to internet delivery practitioners. The relationships in these studies showing antecedents of effective outcomes can be used to help practitioners create these favorable outcomes in their courses. In addition, the results of the above studies are points of departure for hypotheses for further (and perhaps methodologically improved) research. A MODEL FOR FUTURE RESEARCH Figure 1 shows a model of the delivered classroom learning experience. It is an open systems model showing outcomes, inputs, and delivery or throughput variables. Relationships between outcomes on one hand and inputs and delivery variables on the other suggest directions for research on the effectiveness of internet course delivery. In figure 1, the lists of specific variables (such as technology acceptance) are examples of potential researchable variables; they are not an exhaustive set. 280 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 Figure 1: A model for learning with Online Delivery INPUTS Students Instructors Institutions Personality Whether specified online Academic Ability Experience with web classes Whether University a virtual U. Technology Acceptance Technology Acceptance Whether program entirely online Major of Emphasis Teaching style Course requires physical presence? Pedagogical Preference Whether author instructs course Experience with web classes ▼ THROUGHPUTS Technological Factors Course Content Teaching Style Software Sophistication Degree descriptive Amount of student contact Audio, Visual, Other Supplements Degree Analytical Degree Immediacy Behavior Degree Quantitative Incentive or Grading Scheme Degree Abstract Number and Type of Assignments Degree of Difficulty Degree of Rigor Focus on Personal Awareness Degree of Flexibility Focus on Interpersonal Skills Synchronous vs. Asynchronous Instructivist vs. Constructivist ▼ OUTCOMES Student Instructor Institution Learning Satisfaction Enrollment Perceptions of Learning Time Spent Reputation General satisfaction Precision of Materials Student Job Placement Rates Satisfaction with Specifics Involvement in Discussion Profits Perceptions of Usefulness Individualized Instruction Convenience Adjustment of Style to Web Delivery Perceptions of Interaction Perceptions of Professor Access Perceptions of Interaction Difficulty Perceptions of Flexibility Perceptions of Media Richness Perceptions of Demanding ness Interaction Frequency Interaction Patterns Length of Comments 281 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 Inputs. Inputs include student characteristics, instructor characteristics, and institutional variables. These input variables are already formed before course delivery. Student characteristics should be studied because some types of students may gain more from internet delivery than others or the internet may be better suited for some students and the classroom may be better suited for others. Student characteristics might include personality variables such as introversion-extroversion, major or emphasis, and level, e.g., undergraduate (Wernet et al., 2000). Another student characteristic worth studying is acceptance of technology (Arbaugh & Duray, 2001). Arbaugh & Duray (2001) found that perceived usefulness of the technology (a component of acceptance) significantly influenced perceived learning for courses delivered on the internet. Finally, in some programs, students may be able to choose between internet and classroom sections of courses. Among those who have chosen a combination of internet and classroom sections, another variable might be preference between internet and classroom delivery. Professor characteristics should be explored, as some kinds of professors may be better suited as internet instructors than others. Specific variables might include need for affiliation, teaching style, comfort with technology, and experience teaching courses online. There is evidence (Arbaugh & Duray ,2001) of a positive association between perceptions of student learning and professor-online teaching experience. Institutional variables are important in part because institutions vary as to course, program, and technology requirements, and it is possible that outcomes may be more positive under certain institutional conditions and less positive under others. Some universities offer online MBA’s which differ from these universities’ traditional MBA’s; others offer the same MBA for internet students as they do for students who take courses in the classroom. Some universities are simply virtual Universities. Others have a physical campus, where most students attend classes. Some programs are entirely online; others offer both online and in-class courses. Some online programs consist of courses which are entirely on the web; others have courses in which online students attend a limited number of sessions on campus; still others require a combination on campus and online activities. Some programs require the person who develops the course to deliver it; for other programs delivery and authorship are separate. Some programs require specified software; others are more flexible. Different universities have different limits, on class size and ways in which tests can be taken, as examples, and programs vary as to how much they require their courses to be consistent (Swan et al, 2000). Institutions make decisions that may affect outcomes, at least for instructors, decisions that concern faculty workload, compensation for course set-up, and policies towards intellectual property. A variable often decided on by the institution is software platform or type of software package used to organize a course. Some software packages are institution specific such as SUNY Learning Network’s Course Management System (Swan et. al., 2000), others have been adopted by multiple universities such as WebCT (Kendall, 2001; Wernet et. Al, 2000), Blackboard/Course Info, and Learning Notes (Smith, 2001; Drago and Gosen, 2002). There has been one study where platforms have been compared. Arbaugh (2002) found that students using Blackboard felt that the software was easier to use and their experience was more satisfying than students using Lotus Notes. Software package is not the only technologically related variable that might affect outcomes in online learning environments. Webster and Hackley (1997) suggest that technology should be reliable and of high quality. Throughputs. Throughput or delivery variables include technological factors, course content, and teaching style. These variables guide the online learning experience. These are throughput variables and not input because they are more a feature of the individual course and the way the instructor/author chooses to deliver it. Input variables are either background characteristics of course teachers, authors, or students, or characteristics of decisions about course delivery imposed on authors and teachers by the institution. Technological factors that are throughputs are those decided on by the teacher or author and not imposed by the institution. Course content is more unique to the course taught than a characteristic of the institution, and teaching style, while a result of the teacher’s background, is a function of and applied to the course taught and not merely a background factor. One technological delivery factor is the degree of software sophistication. According to Arbaugh & Duray (2001), there is great variation in approaches with respect to delivery sophistication. Software sophistications ranges anywhere from a combination of email and electronic bulletin boards (Bailey & Kotlar; 1994; Dumont, 1996; Partee, 1996) to web-based proprietary software which allows for transmission of multimedia, threaded discussions, and chatroom capability (Greco, 1999; Phillips, 1998). Another technological delivery continuum is the extent to which the on-line environment is supplemented with other teaching pedagogies, such as the use of streaming audio, video, videotapes, video-conferencing, or conducting a portion of a course or program on campus (Greco, 1999; Phillips, 1998). Sometimes the degree of software and supplements are an institutional decision, but often instructors have some say in these matters. The course taught also impacts learning related outcomes. Some courses ask students to learn descriptive content; others call for analytical skills. Some are quantitative; some are highly abstract and call for idea sharing and brainstorming; others are conceptually difficult, and in them students might need individualized help; still others develop self-awareness skills and focus on interpersonal dynamics. Common sense would suggest that some teaching styles are better than others for teaching each of the above kinds of courses, and it is possible that online 282 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 teaching is better for some of the above kinds of courses than it is for others. The final throughput dimension in figure 1 is teaching style. Teaching style is ordinarily presumed to affect outcomes, especially satisfaction. Variables derived from this dimension include but are not limited to amount of contact a professor has with students (Swan et al, 2000), the extent to which the instructor engages in immediacy behaviors (Myers, Zhong, & Guan, 1998), incentive schemes, number and type of assignments required, rigor, flexibility (Drago & Gosen, 2002), and the amount of group work required. In addition, courses can be synchronous in that they ask students to communicate in real time, or they can be asynchronous. For asynchronous online courses where there is virtually no competition for airtime, one common sense hypothesis would be that participation rates would be higher when the incentive scheme rewards participation. Another teaching style related variable is instructivist vs. constructivist categories of pedagogy (Leidner & Jarvenpaa, 1995). The instructivist model is one-way knowledge transmission, from the professor to the student. The constructivist approach assumes knowledge is developed by every learner interacting with others. Internet advocates believe that online delivery is well suited for constructive or collaborative learning, but the evidence (Arbaugh & Benbunan-Fich, in press) is inconclusive. A teaching approach variable often neglected in the online advocate literature involves teaching traditional classes with online supplements, e.g., internet assignments or asynchronous opportunities (Althaus, 1997 and Thoennsen et al., 1999). Outcomes. Outcomes are the results of delivery, and input and delivery variables in figure 1 should impact outcomes. There are three loci of outcomes in this paper’s model: student, instructor, and institution. It is most important for a learning delivery system to affect the student positively. The most frequently measured outcomes associated with students are learning and satisfaction. Learning has been measured objectively with tests (Alavi, Yoo & Vogel, 1997; Arbaugh, 2000b) and by self-report (Alivi, 1994; Arbaugh & Duray, 2001). Self-reports of learning are in effect perceptions of learning. Some authors have explored perceptions of specific types of learning. For example, Hiltz et al (2000) studied perceptions of collaborative learning effectiveness. Satisfaction can be general satisfaction (Alavi, Wheeler & Valicich, 1995; Arbaugh & Duray, 2001; Kendall, 2001; Warkinton, Sayweed & Hightower, 1997), substitutes for satisfaction such as course ratings (Hilz, 1997) and attrition rates (Serban,1999), or satisfaction with specific aspects of the experience such as satisfaction with the use of technology (Wernet et al., 2000). Satisfaction and perceptions of learning are attitudinal variables, but they are not the only attitudinal variables measured in internet delivery studies. Researchers have measured such variables as technology usefulness (Arbaugh, 2000b), convenience (Hilz, 1997), how interesting the course was (Hilz, 1997), perceptions of interaction and participation (Alavi et al, 1997), access to the professor (Hilz, 1997), interaction difficulty (Arbaugh, 2000a), flexibility (Arbaugh & Duray, 2001), media richness (Webster & Hackley (1997), and how demanding the course was (Serban, 1999). Studies focusing on outcomes involving interaction and participation have used objective measures as well as those stemming from self-reports. For example, Arbaugh, 2000a, 2000b) assessed the medium by measuring interaction patterns among students and counting the number of times students talked (sent messages) to other students. He also has suggested (Arbaugh & Benbunan-Fich, in press) the use of frequency and length of comments as interaction- participation variables. The second locus of outcomes is the teacher. Teacher oriented outcomes include satisfaction (Ellram & Easton, 1999), time spent preparing and administering the course (Arbaugh, in press[a]; Dryud, 2000), precision in presentation of materials (Coppla, Hilz, & Rotter, 2002), involvement in class discussions, ability to address individual student concerns (Alavi et al., 1997; Gosen, 2002; Hilz& Wellman, 1997), and adjustment of teaching style to web delivery (Gosen, 2002). The final locus is the institution. Dimensions include enrollment rates, reputation, job placement for students, and profits. RESEARCH DESIGN ISSUES Measuring Learning. The best way to prove that a teaching methodology is effective is to show that learners learn in it, and many researchers have tried to use learning to assess the effectiveness of internet delivery. Researchers have measured online learning in three ways: self-reported perceptions of learning, multiple-choice tests used to grade students, and grades. Anderson and Lawton (1997) have presented a criticism on the research on the effectiveness of simulations that applies to the research on internet delivery. Much of the reason for the inability to make supportable claims about the efficacy of simulations can be traced to the selection of dependent variables and the lack of rigor with which investigations have been conducted……Virtually all research designed to measure the outcomes produced by engaging in an activity requires… assumptions concerning the expected outcomes produced by performing that activity. We cannot construct an assessment activity without knowing what it is we expect to measure. In effect, these authors suggest that in order to measure learning appropriately, learning measures should reflect consciously thought-out learning goals, something not usually done for most learning delivery assessment studies. It is also beneficial for measures to be objective, and self- reports of learning are not. Grades and tests are 283 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 performance indices that according to Wolfe (1990) are unproven proxies for learning. In addition, from the experiential-learning and simulation literature, there are very few studies that show one teaching approach yielding superior performance to another, especially on graded exams. The reasons seems obvious. Exam performance and grades are a function of ability and motivation. Motivated high ability individuals will perform well, regardless of the teaching design, and distracted, less-motivated, lower ability individuals will perform more poorly. Although this is true for virtually all research on teaching approaches, the credibility of online assessment research will increase dramatically with more valid measures of learning and performance. Multi-institutional Research. According to Arbaugh and Duray (2001), virtually all of the studies undertaken on internet delivery have been based on a individual courses, inter-university collaboration on a single course, or multiple courses at a single institution. As a result, the findings of these studies may reflect idiosyncrasies of the instructor(s) or the institution rather than providing an accurate understanding of what influences favorable outcomes in web-based course delivery. Multi-institution studies increase the likelihood that these institution or instructor unique characteristics can be controlled for, thereby allowing for increased generalizability of findings. In addition according to Arbaugh & Benbunan-Fich (in press), since there is as many approaches to course delivery as there are institutions, multi-institutional studies could compare delivery approaches. Interdisciplinary studies. For the most part, studies that have been undertaken have explored single disciplines. Even in studies in which a variety of courses have been sampled, subject matter has not been the focus of the research. It seems that cross-discipline studies would be valuable, for four reasons. First findings across disciplines would be more generalizable than findings within a discipline. Second, it seems likely that online delivery would be more effective for some courses than for others, and it would be valuable (especially to students) to know for which courses web-based delivery is more effective and for which ones it is less so. Third, specific features of online delivery might be more effective for some subjects and less so for others, and this knowledge would help instructors better tailor their courses to each unique subject (Arbaugh, 2002). Fourth, it is also possible that the relationship between particular input and outcome variables might be different for some disciplines than for others. For example, the relationship between immediacy of instructor behavior and learning might be stronger in quantitative courses than in courses covering abstract material. Such a result would be valuable for practitioners, so they can make their courses ideally suitable to course material. Contingency studies. One reason why research should be undertaken across institutions and disciples is because these two types of variables can play the role of contingency variables. For the most part, online research has treated internet delivery as a one-size-fit-all phenomenon. It may not be. There’s already limited evidence that outcomes are higher for one software platform than for another (Arbaugh, 2000a), and it is possible that outcomes might be higher for some institution related characteristics than others and some disciplines than others. This is important. If for example, it were found that students are more satisfied with courses in institutions with class size limits, institutions that have settled intellectual property rights issues, or classes with abstract subject matter, then schools and instructors could use those results to gain increased student satisfaction. Perhaps more importantly, relations among variable might contingent on the value of other variables. Some examples: The relationship between satisfaction and experience in web courses may be different for different software platforms. The relationship between immediacy of instructor behavior and satisfaction may depend on class size and be non-existent for very small classes. DISCUSSION Contingency revisited. The stated purpose of this paper was to provide a model and variables to assess courses and programs which use internet or online based delivery approaches. Perhaps at first glance the model provided in figure 1 does not do that. After all, the relationships in figure 1 suggest influences on outcomes and not delivery effectiveness assessment. However with a scratch of the surface, the hypotheses emerging from the model do in fact help us assess internet delivery effectiveness. In this model, outcomes are hypothesized to vary with throughput and input variables, and likely, most would agree that courses on the web are better (produce greater learning and satisfaction) when the student is more comfortable with the technology, when the instructor has more experience, when the media is enriched. Also, most would likely agree that some students would be more likely to embrace this media than others, shy students perhaps and students who don’t like to compete for air time during in-class discussions. So an assessment might reveal that the internet is more suited for some students than for others. That is not a problem. Let the students who are better suited for the medium use it and those more poorly suited go to the physical classroom. However this kind of relationship, the kind that hypothesizes that online delivery is better under some conditions than others, points to potential problems with the medium. For example, it is possible that online delivery is also better suited for some kinds of courses than others, in particular courses where content is difficult and where one-on-one instruction is more important than average. Accounting and Statistics are examples. Furthermore, Peek (2000) points out that the educational platform provided by the web limits what can be accomplished, since it lacks visual cues and limits auditory stimuli. In courses where these auditory and visual cues are keys to mastery, internet delivery may be deficient. Business courses such as Sales, Training, and 284 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 Organizational Behavior are possible examples, because they require face-to-face interaction and feedback to the learner on a wide range of behaviors in order for the learner to learn optimally. So there are likely to be conditions where online delivery is inappropriate, and specified course content is one of those conditions. There may be others. If there are conditions in which online delivery is deficient, then our students and those who employ them should know about them, and those interested in undertaking online assessment research should being doing studies to attain that knowledge. Flexibility and outcome variables revisited. Course flexibility has been mentioned in this paper, and research has been undertaken with perceptions of flexibility as an outcome variable (Arbaugh & Duray, 2001), with the implication that flexibility is desirable. Yet online delivery has also been criticized (Drago & Gosen, 2002) as inflexible. These authors argue that it is very time consuming for professors change content from one term to the next because it takes a considerable amount of time and effort develop and update courses. They also claim that because course design is set in print on the internet at the beginning of a term, it is difficult and disruptive for students to change course content and procedure as the course progresses. Then, while at least one set of writers consider flexibility to be desirable, flexibility is difficult to accomplish and possibly undesirable in the eyes of students according to a different set of authors. The online approach seems to be inflexible in other ways. Online courses are limited in that closed-book tests are not practical and in that one-on-one, student-teacher assistance is at least sometimes inconvenient and, relative to face-to-face classroom situations, time consuming. But sometimes, the flexibility associated with opportunities for closed book testing and face-to-face assistance is desirable. In particular, face-to-face assistance is desirable when the subject matter is difficult, as often the subject matter of college courses (at the graduate or undergraduate level) ought to be. Such courses should challenge students to think, analyze, and solve complex problems. With such challenges, instruction needs to flexible, attentive, focused, and sensitive, and synchronous one-on-one help should be available. But with online instruction, which is not synchronous, and, because visual and auditory cues are absent, not sensitive, it is difficult to tell whether each student is “getting it,” and helping someone who is not getting it means communicating by email. The process of instructing and helping online, then, can be time consuming and frustrating for both student and teacher. Therefore, the fact that the online approach is limited with regard to one- on-one assistance means it may not be ideally suited for difficult, rigorous college courses. Furthermore, the choice of outcome measures chosen by online delivery researchers to assess internet delivery imply goals for this medium that also contradict attempts at rigor. As indicated in earlier sections of this paper, in most studies, these researcher-chosen outcome variables have been for students to be satisfied, believe that they are learning, and get good grades. But college courses should do more than help students be happy and believe they are learning. These courses should be rigorous and help students attain difficult-to-acquire skills. But as indicated above, the instruction and assistance processes on line are asynchronous, less flexible, and less sensitive, and therefore potentially frustrating for the teacher and student. This frustration means lower satisfaction. A frustrated student does not believe he is learning, and there is the danger of getting poor grades. The atmosphere becomes negative, and the goals for the delivery medium are not reached. To solve the problem, given the goals implied by online studies, the temptation might be exist to eliminate that which causes the frustration. It is possible that this means modifying courses so that they are less challenging and less rigorous. In effect, courses may be compromised to fit the medium. The contention here is that rigor might be sacrificed in online courses. To my knowledge, this has not been documented. The fact is that I am aware of instances where it seems like rigor and complex analysis has been reduced to accommodate the internet medium, but I do not know whether these are isolated instances or the practice is wide spread. I am not aware of any research that compares the rigor of course content of internet courses with that in the classroom, and I am not aware of research comparing grading schemes of internet courses with those in the classroom. Such research is needed to properly assess the effectiveness of online course delivery. This paper opened with the observation that online teaching is both established and controversial. Regardless of how effective it is, it needs to be taken seriously because it is widespread and attractive. So far research has tried to defend the medium by showing that students like the medium and can pass tests with it. Taking it the delivery medium seriously means research that helps those interested to know what online teaching can and cannot do. REFERENCES Alavi (1994) “Computer-mediated collaborative learning: An empirical evaluation”, 18, 159-174. Alavi, Wheeler, & Valacich (1995). “Using IT to re- engineer business education: An exploratory investigation of collaborative telelearnin.” MIS Quarterly, 19, 293-312. Alavi,, Yoo, & Vogel (1997) “Using information technology to add value to management education.” Academy of Management Journal, 40: 1310-1333. Althaus (1997) “Computer-mediated communication in the university classroom, An experiment with on-line discussions.” Communication Education, 46(3): 158- 174. 285 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 Coppola, Hiltz, & Rotter (2002) “Becoming a virtual professor: Pedagogical roles and Asynchronous Learning networks.” Journal of Management Information Systems, 18(4), 169-189. Anderson & Lawton (1997) “Demonstrating the learning effectiveness of simulations, where we are and where we need to go.” Developments in Business Simulation & Experiential Learning, 24, 68-73. Drago & Gosen (2002) “Web Course Instruction: Two Instructors’ Perspectives.” Presented to the Conference on Innovative Teaching in Human Resources and Industrial Relations, Columbus OH, November 8-9. Arbaugh (2000) “Virtual classroom characteristics and student satisfaction in internet-based MBA courses.” Journal of Management Education, 24: 32-54. Arbaugh (2000b) “Virtual classroom versus physical classroom, An exploratory comparison of class discussion patterns and student learning in an asynchronous internet-based MBA course.” Journal of Management Education, 24: 207-227. Dumont (1996) “Teaching and learning in cyberspace.” IEEE Transactions on Professional Communication, 39(4), 192-204. Duval (2000) “ Start-up plans in education market place.” Interactive Week, 7, 35. Arbaugh (2002) “A longitudinal study of technological and pedagogical characteristics of web-based MBA courses.” Academy of Management Best Papers Proceedings. D. Nagao (Ed.), MED, A1-A6. Dyrud (2000) “The third wave, A position paper.” Business Communication Quarterly, 63(3), 81-93. Ellram & Easton (1999) “Purchasing education on the Internet.” Journal of Supply Chain Management, 35(1), 11-19. Arbaugh & Duray (2001) “Class section size, perceived classroom characteristics, instructor experience, and student learning and satisfaction with web-based courses: A study and comparison of two on-line MBA programs”. Presented to the Academy of Management Flaherty, Pearce, & Rubin (1998) “ Internet and face-to-face communication: Not functional alternatives.” Communication Quarterly, 46(3), 250-268. Arbaugh & Benbunan-Fich (In Press) “Contextual factors that influence ALN effectiveness. In Asynchronous Learning Networks: The Research Frontier.” Starr Roxanne Hiltz and Ricki Goldman, Eds. Mahwah, NJ: Lawrence Ehrlbaum Associates. Freitas, Myers, & Avtgis (1998) “Student perceptions of instructor immediacy in conventional and distributed learning classrooms.” Communication Education, 42(4), 366-372. Arbaugh & Hiltz. (In Press) “Improving quantitative research on ALN effectiveness.In Asynchronous Learning Networks: The Research Frontier.” Starr Roxanne Hiltz and Ricki Goldman, Eds. Mahwah, NJ: Lawrence Ehrlbaum Associates. Gibson & Gibson (1995). “Lessons learned from 100+ years of distance learning.” Adults Learning, 7(1), 15. Gilbert & Moore (1998). “Building interactivity into web courses, Tools for social and instructional interaction.” Educational Technology, 38(3), 29-35. Bailey & Koltar (1994). “Teaching via the internet.” Communication Education, 43(2), 184-193. Gosen (2002) “Structural Deficiencies of Web-Based Instruction.” Presented to the Conference on Innovative Teaching in Human Resources and Industrial Relations, Columbus OH, November 8-9. Borthick & Jones (2000). “The motivation for collaborative discovery learning online and its application in an information systems assurance course.” Issues in Accounting Education, 15(2): 181-210. Greco (1999) “Going the distance for MBA candidates.” Journal of Business Strategy, 20(3), 30-34. Clarke (1999) Going the distance. Black Enterprise, 29, 113-118. Grossman (1999) “On-line U.” Scientific American, 281, 41. Cox (2000). “Why I left a university to join Internet education.” Change, 32, 12-18. Hartman, Dziuban, & Moskal (2000) “Faculty satisfaction in ALNs, A dependent or independent variable.” J. of Asynchronous Learning Networks, 4(3). Comeaux (1995) “The impact of an interactive distance learning network on classroom communication.” Communication Education, 39, 323-340. 286 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 287 Hilz (1997) “Impacts of college-level courses via Asynchronous Learning Networks, Some preliminary results.” J. of Asynchronous Learning Networks, 1 (2), 1-19. Hiltz, Coppola, Rotter, Turoff, & Benbunan-Fich (2000). “Measuring the Importance of Collaborative Learning for the Effectiveness of ALN, A Multi-Measure, Multi-Method Approach.” J. of Asynchronous Learning Networks, 4(2). Hiltz & Wellman (1997) “Asynchronous learning networks as a virtual classroom.” Communications of the ACM,, 40(9), 44-49. Kendall (2001) “Teaching online to campus based students: The experience of using WebCT for the community information module at Manchester Metropoitain University.” Education for Information, 19, 325-346 Leidner & Jarvenpaa (1995) “The use of information technology to enhance management school education, A theoretical view.” MIS Quarterly, 19, 265-291. Leasure, Davis, & Thievon (2000) “Comparison of student outcomes and preferences in a traditional vs. world wide web-based baccalaureate Nursing Research course.” Journal of Nursing Education, 39(4), 149- 154. Myers, Zhong, & Guan (1998) “Instructor immediacy in the Chinese college classroom.”. Communication Studies, 49, 240-253. Partee (1996) “Using e-mail, web sites and newsgroups to enhance traditional classroom instruction.” T.H.E. Journal, 24, 79-82. Peek (2000) “A distance learning reality check, When is it truly educational, and when is it merely web publishing?” Information Today, 17, 30. Phillips (1998) “On-line universities teach knowledge beyond the books”. HR Magazine, 43(8), 120-128. Ricketts, Wolfe, Norvelle, & Carpenter (2000) “Asynchronous distributed education, A review and case study.” Social Science Computer Review, 18(2), 132-146. Sankaran & Bui (2001) “Impact of learning strategies and motivation on performance, A study in web-based instruction.” Journal of Instructional Psychology, 28(3), 191-198. Serban (1999) “Evaluation of Fall 1999 online courses.” http,//www.sbcc.net/rt/ir/eval_cover.html Smith (2001) “Content and delivery, A comparison and contrast of electronic and traditional MBA Marketing Planning courses.” Journal of Marketing Education, 23(1), 35-44. Thoennessen, Kashy, Tsai, & Davis (1999) “Impacts of Asynchronous Learning Networks in large lecture classes.” Group Decision and Negotiation,. 8, 371-384. Swan, Shea, Fredericksen, Picket, Pelz, & Maher (2000) “Building knowledge building communities, Consistency, contact, and communication in the virtual classroom.” Journal of Educational Computing Research., 23, 389-413. Warkentin, Sayeed, & Hightower (1997). “Virtual teams versus face-to-face teams, An exploratory study of a web-based conference system.” Decision Sciences, 28, 975-996. Webster & Hackley (1997) “Teaching effectiveness in technology-mediated distance learning.” Academy of Management Journal, 40, 1282-1309. Wernet, Olliges, & Delicath (2000) “Post course evaluations of WebCT classes by Social Work students.” Research on Social Work Practice, 10, 487-514. Wolfe (1990) “Evaluation of computer-based business games, Methodology, findings and future needs.” In J. Gentry (Ed.) A Guide to Business Gaming and Experiential Learning: Nichols/GP Publishing, 279- 300. http://www.sbcc.net/rt/ir/eval_cover.html Table of Contents Volume 30, 2003 The Optimal Timing For Introducing Business Simulations Can Handicapped Students Access Your Class Web Site? The Competition Game: Decision Making In A Dynamic Environment Pan-Pacific Enterprises: Strategic Decision Making Simulation Study Of Stochastic Channel Redistribution The Impact Of Business War Games: Quantifying Training Effectiveness Experiential Learning: Introducing Faculty And Staff To A University Leadership Development Program The Feasibility Of The Balanced Scorecard For Business Games A Misuse Of Pims For The Validation Of Marketing Management Simulation Games Incorporating Technology Into The 21st Century Classroom: Are We Facilitating Academic Dishonesty? Improving The Effectiveness Of Peer Evaluations The Use Of A Simulation In An Integrated Mba Curriculum Student Portfolios In Business Education Student Portfolios In Business Education At Ashland University Using SAP ERP Technology To Integrate The Undergraduate Business Curriculum Board Games And Teaching Textile Marketing And Finance Blogging: A New Threat To Student Research? The Way We Talk! Take II Strategic Management: An Evaluation Of The Use Of Three Learning Methods In Hong Kong Adoption Of Discussion-Based Teaching And Assessment In Teaching Strategic Management In Hong Kong The Tobin Q As A Company Performance Indicator Using Representative Nominal Group Technique For Course Review And An Interactive Solicitation Of Ways To Enhance Absel's Image Making Teaching Matter: The Art And Science Of Teaching Business Communication Beyond Sex, Age, And Race: Exploring The Deeper Contents Of Diversity Teaching & Learning The Facilitation Process A Brief On Debriefing: What It Is And What It Isn't Changing Perceptions Of The Importance Of Leadership: The Contribution Of Individual Spirit Harmonics In Leadership Knowledge, Skills And Sustainable Values In Learning Organizations: Some Implications From The Multicultural Virtual Classroom Challenges Of Teaching Undergraduate Organizational Behavior In A Nontraditional Time Format Interactive Online Positioning With The Web-Based Product Positioning Map Graphics Package The Longitudinal Effects Of Entrepreneurship Training On Risk Tolerance: A Look At Similarities And Differences Between Male And Female Undergraduate Students Revisiting Strategy Learning In A Total Enterprise Simulation What Are Simulations For?: Learning Objectives As A Simulation Selection Device Gaming Agency Markets Cooperate For Profits Or Compete For Market? Study Of Oligopolistic Pricing With A Business Game The Design Of A Business Simulation Using A System-Dynamics-Based Approach Modeling The Product Development Function For An Entrepreneurial Firm Simulation Performance And Forecast Accuracy? Is That All? Business Manager Identification Of Competitors In Real World And Simulation Settings Monte Carlo Simulation Analysis On The Costs Reduction Argument Of Interest Rate Swaps Ebiz Game: A Scalable Online Business Simulation Game For Entrepreneurship Training A Model For Online Education Delivery And A Look At Online Delivery Effectiveness Incorporating "Company Reputation" into Total Enterprise Simulations The Genesis And Future Of The Absel "Classicos" Initiative