THE USE OF MULTIMEDIA LEARNING TOOLS TO FACILITATE ONLINE LEARNING OF BUSINESS STATISTICS Developments in Business Simulation and Experiential Learning, Volume 34, 2007 51 THE USE OF MULTIMEDIA LEARNING TOOLS TO FACILITATE ONLINE LEARNING OF BUSINESS STATISTICS Joyce Keller St. Edward’s University joycek@stedwards.edu Rui Li St. Edward’s University ruil@stedwards.edu ABSTRACT This study involved the development of multimedia movies covering the core concepts in an introductory business statistics course. The movies were designed to aid students in content mastery in an online course setting. Such tools are critical to online learning, particularly for courses requiring the acquisition of precision skills such as mathematics, engineering, computer programming, and the like. The movies incorporated the findings from multimedia instructional research regarding the most effective features in such tools. While the movies have been used in two online sections of the course, two of the movies were given more rigorous testing using a randomized design. Results indicated that students using the multimedia approach scored no differently from students receiving a traditional instructor led intervention. Further repetitions of the experiment will provide strength to this finding. BACKGROUND AND PURPOSE For the past decade, web-based instruction has been widely accepted and practiced in American universities with the hope to increase university enrollment, promote access to higher education, and allow greater flexibility in instruction (Waschull, 2001; Ridley et al., 1997). However, many educators are still skeptical and concerned about the quality and effectiveness of web-based instruction as compared to traditional face-to-face classrooms. Inman and Kerwin (1999) found that although most instructors who are currently teaching web-based courses are willing to teach the courses again, 50% of them reported that the quality of the web-based instruction, comparing to a traditional format, was of lower quality. A considerable body of research has been done to investigate the effectiveness of web-based instruction and has yielded mixed results. Many studies found no differences between web-based and traditional classroom courses on student performances. Russell (1999) reported that more than 300 studies investigating traditional classes and distance courses found no difference in performance of students. His reports included correspondence courses, online courses, and telecourses. However, other researchers reported contradictory results (Waschull, 2001; Wang & Newlin, 2000; Ridley, 1998). For example, Mottarella et al (2004) examined students’ course grades and achievement scores on ACAT (the Area Concentration Achievement Test in Psychology) and found that students in web-based courses had significantly lower course grades than classroom-based or web-enhanced (blended/hybrid) courses. Furthermore, students in both web-based and web-enhanced courses scored significantly lower in standardized achievement test (ACAT). While most of the studies strived to investigate performance related differences between traditional and web-based courses, the design of those web-based or web- enhanced courses are rarely addressed. Are the web-based courses designed with sound cognitive learning theories? Are the cognitive load imposed on students reasonable? Are there any multimedia components built into the course? If so, is the design of multimedia components based on multimedia learning principles proven to be effective? We believe that in order to make a valid comparison between traditional and web-based/web-enhanced courses, we need to first address the design of web-based/web-enhanced course. This approach will help us avoid comparing a well- designed traditional course to a poorly-designed web- based/web-enhanced course. The design of the web-enhanced course under investigation in this study is informed by a wide range of literature on learning and cognition theories. COGNITIVE LEARNING THEORY Cognitive learning focuses directly on human cognitive processes, “considering how people perceive, interpret, remember, and otherwise think about the environmental events they experience” (Ormrod, 1999 p. 145). Cognitive information processing theory provides a framework for understanding how human learn and think. It believes that memory plays an important role in learning. There are three stages of memory during information processing: sensory memory, short-term memory, and long- mailto:joycek@stedwards.edu mailto:ruil@stedwards.edu Developments in Business Simulation and Experiential Learning, Volume 34, 2007 52 term memory (Ormrod, 1999). During the first stage of processing, the stimuli are registered in the memory system. This sensory memory briefly holds information until the stimulus is recognized or lost. When people attend to and recognize the stimulus, the information is then sent to the short-term memory. The short-term memory can hold the information longer than sensory memory through elaboration and rehearsal. Finally, once information is encoded from short-term memory (activate and link to prior knowledge), it is stored in the long-term memory in the form of schemata, which is a complex, organized, and permanent body of knowledge. Thus, facilitating schema acquisition and construction should be a primary goal for instruction and learning. COGNITIVE LOAD THEORY Cognitive Load Theory (CLT) has evlolved ffrom information processing theory and has been used to investigate some conditions that need to be considered to construct schemas. Cognitive load refers to the total amount of mental activity imposed on working memory at an instance in time (Cooper, 1998). There are three types of cognitive load: intrinsic cognitive load, extraneous cognitive load, and germane cognitive load. Intrinsic cognitive load is associated with the nature of the instructional materials and is normally irreducible. Extraneous cognitive load refers to the demand imposed by the manner in which materials is presented and the activities required of the learner (Sweller, van Merrienboer, & Paas, 1998). It’s also called ineffective cognitive load, which usually caused by poorly design instructional materials. Germane cognitive load refers to the memory resources used to engage learners in conscious cognitive processing that is directly related to schema construction. Thus, the primary goal of CLT is to support the design of instructional procedures and activities that effectively manage cognitive load to enhance learning. In other words, reduced extraneous cognitive load and increase germane cognitive load. Some of the recommendations proposed by Sweller (1999) include: • Change problem solving methods: Avoid means- ends approaches and use goal-free problems or worked examples. • Integrating multiple resources to eliminate the need for learners to have to mentally integrate that information. • Reduce repetitive and redundant information. • Use auditory and visual information under conditions where both sources of information are essential (i.e. non-redundant) to understanding. COGNITIVE APPRENTICESHIP Cognitive apprenticeship theory takes another approach and attempts to “enculturate students into authentic practice through activity and social interaction” (Brown et al., p.37). Brown (1989) proposed the cognitive apprenticeship as an implementation of situated cognition. It is based on the traditional apprenticeship where apprentices initiated into a profession through modeling, coaching, and fading. For example, when teaching a new topic, the mentor would make his/her thinking explicit through modeling the cognitive activity. Then, modeling is followed by coaching where the mentor provides support and guidance in the learning process. Mentor support is gradually reduced as students grasp the concepts. In addition to the traditional apprenticeship components, cognitive apprenticeship also embraces collaboration, reflection, and articulation (Brown et al., 1989). Collaboration supports group discussion and problem-solving, which allows learners to understand multiple aspects and roles. Reflection enables learners to compare their own performance with other resulting in reflection and deep thinking. And articulation encourages learners to make their thinking explicit, which promotes reflection and comparison. Therefore, it is important to provide students opportunities for modeling, reflection, articulation, and collaboration during their learning process. MULTIMEDIA LEARNING THEORY Mayer (1997) reviewed 24 research studies on multimedia learning and proposed a generative cognitive theory of multimedia learning based on dual coding theory, cognitive load theory, model of working memory (Baddeley, 1992), generative theory (Wittrock, 1989), and Mayer’s (1996) SOI model of meaningful learning. Mayer’s multimedia learning theory proposed that the learner possesses a visual information processing system and a verbal information processing system (Mayer, 1997). These two systems, then, can be used to explain learner’s cognitive process in multimedia learning. Mayer and Moreno (2002) argued that learners engage in three important cognitive processes in multimedia learning. The first cognitive process is Selecting. Verbal and visual information serve as signals that help learners select relevant information. The second cognitive process is Organizing. Both visual and verbal information serves as organizers that help learners build cause-and-effect relations among pieces of visual information and among pieces of verbal information. The final cognitive process is Integrating. Learners build connections between corresponding parts in the visual representation and verbal representation. This cognitive model is illustrated in Figure 1. Derived from this cognitive model, Mayer and Moreno (2002) further recommended five principles on the design of multimedia learning. 1. The multiple presentation principle: It is better to use two modes of representation (visual and verbal) rather than one. Developments in Business Simulation and Experiential Learning, Volume 34, 2007 53 2. The contiguity principle: Students learn more deeply if they do not have to hold the entire animation in working memory until the narration is presented, or vice versa. Thus, corresponding words and pictures need to be presented contiguously. 3. The split-attention principle: Words are better presented as auditory narration rather than on- screen text. 4. The individual differences principle: Multimedia effect, contiguity effects, and split-attention effects depend on individual differences. Learners who lack prior knowledge tend to benefit more from the above principles (Mayer & Sims, 1994). 5. The coherence principle: Students learn better if they do not have to process extraneous words and sounds in verbal working memory or extra pictures in visual working memory. This principle is also referred as redundancy effect in cognitive load theory (Sweller et al., 1998). The literature review considered thus far leads us to believe that in order to promote meaningful and effective learning, the following elements should be considered and built into the design of multimedia learning modules in our web-enhanced statistics course: • Contiguous presentation of visual and verbal mode of instructional materials (multiple presentation/contiguity principle, cognitive load theory). • Audio narration of instructional materials (split- attention principle). • Elimination of repetitive and redundant information (coherence principle, cognitive load theory). • Opportunities for modeling and reflection (cognitive apprenticeship). • Presentation of worked examples (cognitive load theory). The current study focused on the development of multimedia tools that could facilitate learning without benefit of a classroom teacher while at the same time going beyond a basic textbook. The aim was the development of materials that would aid students in online statistics classes in their knowledge mastery. METHOD To aid students’ mastery of the material online beyond the mere provision of written material, the researchers created multimedia FLASH movies covering the content of the course. Thus, the movies covered: • Graphing • Measures of location and dispersion • Probability distributions • Hypothesis testing • T and z tests • ANOVA • Correlation • Univariate and multivariate regression The movies were developed based upon research findings regarding the most effective features of multimedia learning tools, using Mayer’s principles: • The movies incorporated the multi-presentation principle, using both visual graphics and verbal mechanisms to convey information. • This multi-presentation ensured that the auditory and visual offerings occurred in sync with the material addressed, that is, it abided by the contiguity principle. • The words were presented auditorily rather than as on- screen text, in accordance with the split-attention principle. • Although students were given a physical copy of the text of the movie, they were told it was only for reference; the movie was the learning intervention and contained no redundant information. • Scores on pretests indicated that students had no prior knowledge of the topics and the topics were complex (Analysis of Variance, correlation, and univariate regression), thus allowing for the individual differences principle to manifest. Figure 1. A cognitive model of multimedia learning (from Mayer, 1999) Developments in Business Simulation and Experiential Learning, Volume 34, 2007 54 • Interspersed in the movies were opportunities for students to test their knowledge using a multiple choice format. If students chose an incorrect answer, a message explaining why the answer was incorrect and redirecting the students’ thought processes was provided. This aligns with Mayer’s cognitive apprenticeship principle. • Worked examples were provided both within the movie itself and within subsequent practice opportunities. The movies incorporated other features such as • Guided learning mechanisms or cues (Huk, Steinke, and Floto, 2003). These included procedures such as moving arrows, flashing results, circled points of emphasis, and others. • Navigation tools which included the ability to stop the movie and repeat sections or to jump to specific sections (Koohan & Plessis, 2004), • Attention influencing instructional strategies such as movement, variation in presentation and color and positioning (Farley & Grant, 2001). HYPOTHESIS The hypothesis was that students watching the multimedia movies would perform no differently from students participating in an instructor-led class. Of course, a hypothesis such as this is the inverse of the usual scientific testing criterion where the effort is to discover if the performance of the treatment group is better than that of the control group. Failure to reject the null hypothesis does not mean that the null hypothesis is true. Thus, such a result does not have the same power as that in the alpha error testing approach and, as a result, is more limited in its ability to confirm the value of the treatment. However, repeated testing with similar results would provide strong evidence in favor of the null hypothesis. This study represents a first step toward the accumulation of that evidence. EXPERIMENT Students enrolled in the Introduction to Business Statistics class in the adult college at St. Edward’s University during the 2004-2005 school year participated in the study. For the most part, adults enroll at St. Edward’s because they value the classroom experience over online learning. Thus, the study was biased against a favorable performance from watching movies versus having a traditional instructor-led class. Forty-two students were randomly assigned to treatment and control groups. Both student groups first completed a pretest over the content material, analysis of variance. Then those in the treatment groups viewed the movie in a computer lab, completed homework, and took a posttest. Those in the control group attended the class in a traditional instructor led setting, completed homework questions together with the instructor, and then took the posttest. The next week, students swapped places such that those who had watched the ANOVA movie the first week received classroom instruction the second week covering regression and the converse regarding students in the control group the first week. Upon completion of the posttest, students viewing the movies were asked to rate the movies along several dimensions. RESULTS Table 1 provides descriptive statistics for each group: (See tTable 1 below) While in both experiments, students receiving in-class instruction scored better than did those watching the movies, was that performance significantly better? Followup t-tests of means indicated that students watching the movies, whether ANOVA or Regression, performed no differently from those in the traditional classroom setting (ANOVA movie: t=1.42, 40 d.f.,p=.161;Regression movie:t=2.02, df=40,p=.149). DISCUSSION AND CONCLUSIONS These results indicate that, although the control groups performed better than the treatment groups, the differences in performance were not statistically significant. This indicates that the provision of multimedia tools covering course content can assist students in mastering material related to an introductory statistics course, enabling mastery comparable to an in-class teacher-led intervention. The results are particularly significant given the population of students in the study. These students are willing to spend a Table 1: Descriptive Statistics ANOVA Median Mean Standard Deviation N Treatment 87.7% 60.4% 29.6% 21 Control 60.0% 74.1% 32.4% 21 Correlation & Regression Mean Standard Deviation N Treatment 80% 63.3% 30.7% 21 Control 80% 76.9% 29.0% 21 Developments in Business Simulation and Experiential Learning, Volume 34, 2007 55 significant amount of scheduled time in class and tuition money to receive in-class instruction, believing that this type of instruction is better for them. That they nevertheless performed at a level similar to those in the in-class environment reflects the effectiveness of the multimedia instructional design. Further, the material covered in the experiment was some of the most complex material covered in the introductory statistics course. The students’ performance again attests to the effectiveness of the instructional materials in facilitating content mastery. The development of such materials is particularly important for students enrolled in online technical courses whose only references may be textbooks or other written materials. In addition to the above test results, students watching the movies were administered a questionnaire asking for their perceptions of the effectiveness of the movies and specific features of them. The results are given in Table 2. (See Table 2 above) These results indicate that the majority of students believed most of the features to be effective aids in learning. Surprisingly, 95% of the students found the audio script to be of significant value. This runs counter to the redundancy principle. However, it is probable that for such complex material, students may find written materials helpful for reference. It should also be noted that the written script did not appear on the multimedia page, causing the split attention effect. It was only ancillary. Future repetitions of this experiment can strengthen the findings of this study. Future studies may also benefit from comparing the use of multimedia tools with the use of textbook materials only. This latter is especially valuable because of the cost of creating, revising, and continually updating multimedia tools. Finally, studies of online learning effectiveness should incorporate detailed attention to the materials provided the online students. Differentiated instructional mechanisms may be needed to optimize learning online from that occurring in the traditional classroom settings, particularly for certain types of course content. REFERENCES Baddeley, A. (1992). “Working memory”. Science, 255, 556-559. Brown, J. S., Collins, A., & Duguid, P. (1989). “Situated cognition and the culture of learning”. Educational Researcher, 18, 32-42. Cooper, G. (1998). “Research into cognitive load theory and instructional design at UNSW”. Retrieved December, 2006, from http://projects.ict.usc.edu/itw/materials/clark/UNSW.ht m Farley, F.H. & Grant, A. (2001). “Arousal and cognition: memory for color versus black and white multimedia presentation”, The Journal of Psychology, 94, 147-150. Huk, T., Steinke, M, Floto, C. (2003). „The educational value of cues in computer animations and its dependence on individual learner abilities”. Proceedings of Ed-Media 2003, 2658-2661. Inman, E. & Kerwin, M. (1999). “Instructor and student attitudes toward distance learning”. Community College Journal of Research & Practice, 23, 581-592. Koohan, A and Plessis, J (2004). “Architecting usability properties in the E-learning instructional design process”. International Journal on E-Learning, July- September 2004, 38-44. Mayer, R. E. (1996). “Learning strategies for making sense out of expository text: The SOI model for guiding three cognitive processes in knowledge construction”. Educational Psychology Review, 8, 357-371. Mayer, R. E. (1997). “Multimedia learning: are we asking the right questions?” Educational Psychologist, 32, 1- 19. Mayer, R. E., & Sims, V. K. (1994). For whom is a picture worth a thousand words? Extension of a dual-coding theory of multimedia learning. Journal of Educational Psychology, 86, 389-401. Mayer, R. E., & Moreno, R. (2002). “A cognitive theory of multimedia learning: Implications for design principles”. Retrieve October 26, 2003, from http://www.unm.edu/~moreno/PDFS/chi.pdf Table 2: Student ratings on effectiveness of instructional media. Feature Effective Neutral Ineffective Navigation controls 72% 11% 17% Audio 95% 5% 0 Graphics 77% 17% 6% Worked examples 67% 17% 11% Interactive Q&A 44% 22% 11% Speed of movement through content 50% 33% 17% Use of arrows, blinkers to call attention to content 94% 6% 0% Clarity of content presentation 77% 11% 11% Provision of audio script to accompany module 95% 5% 0% Overall quality of module 67% 28% 6% http://projects.ict.usc.edu/itw/materials/clark/UNSW.htm http://projects.ict.usc.edu/itw/materials/clark/UNSW.htm http://www.unm.edu/~moreno/PDFS/chi.pdf Developments in Business Simulation and Experiential Learning, Volume 34, 2007 56 Mottarella, K., Fritzsche, B., & Parrish, T. (2004). “Who learns more? Achievement scores following web-based versus classroom instruction in psychology courses”. Psychology Learning and Teaching, 4(1), 51-54. Ormrod, J. E. (1999). Human learning (3rd.). NJ: Prentice- Hall, Inc. Ridley, D. R. (1998, June). “The 1998 assessment report on CNU on-line. Paper presented to the State Council of Higher Education for Virginia”, Newport News, VA. Ridley, D.R., Bailey, B. L., Davies, E.S., Hash, S.G., & Varner, D.A. (1997, May). “Evaluating the impact of on-line course enrollments of FTEs at an urban university”. Paper presented at the Association for Institutional Research Annual Forum, Orlando, FL. Russell, T. L. (1999). “The no significant difference phenomenon: As reported in 355 research reports, summaries and papers”. Raleigh: North Carolina State University. Sweller (1998). “Cognitive load during problem solving: effects on learning”. Cognitive Science, 12, 257-285. Sweller, J. (1999). “Instructional design in technical areas”. Australian Educational Review No. 43, ACER Press, Camberwell, Australia. Sweller, J., van Merrienboer, J. J., & Pass, F. G. W. C. (1998). “Cognitive architecture and instructional design”. Educational Psychology Review, 10(3), 251- 296. Wang A. Y. & Newlin, M. H. (2000). “Characteristics of students who enroll and succeed in psychology web- based classes”. Journal Educational Psychology, 92, 137-143. Waschull, S. B. (2001). “The online delivery of psychology courses: attrition, performance and evaluation”. Teaching of Psychology, 28, 143-146. Wittrock, M. C. (1989). “Generative processes of comprehension”. Educational Psychologist, 24, 345- 376. Table of Contents Volume 34, 2007 Experiential Teaching May Lead To Experiential Learning The Role Of Learning Versus Performance Orientations When Reacting To Negative Outcomes In Simulation Games: Further Insights An Analysis Of The Interaction Of Firm Demand And Industry Demand In Business Simulations Consistency Of Participant Simulation Performance Across Simulation Games Of Growing Complexity Assessing And Incentivizing Learner Contribution And Performance Excellence: Creating Win-Win Evaluation Options The Technological Impact Analysis: A Research-Based Exercise To Heighten Learners' Technological Sensitivity Beginnings: How We Start the Semester and Individual Classes : A Roundtable Discussion Simulation Performance and Its Effectiveness As A PBL Problem: A Follow-Up Study The Use of Multimedia Learning Tools to Facilitate Online Learning of Business Statistics Forecasting Accuracy And Learning: A Key To Measuring Business Game Performance Panel Discussion: Alternative Ways of Using the Internet for Business Simulations to Input Decisions, Process, and Present Financial and Economic Data Output Absel Research -- One Additional Perspective On Where We Are And Where We Have Come From Assessing participant learning in a business simulation From Case Presentation To Case Facilitation: How Assessment Changed The Capstone Course The Use Of Computer-Assisted, Interactive Role-Play Simulation In Hong Kong Corporate Positioning: A Business Game Perspective Demonstration Of A Computer-Assisted Global Business Simulation Applying .NET Remoting To A Business Simulation Assessing Emotional Intelligence: The EQ Matrix Exercise Outcomes And Observations Of An Extended Accounting Board Game The Use Of Learning Styles Questionnaire In Hong Kong Forming Teams For Classroom Projects Online Budgeting And Marketing Control With The Proforma Analysis Package The Effect Of Experiential Learning Experiences On Management Skills Acquisition Students' Perceptions on the Individual Managerial Performance Orderness: A New Definition Of Alignment An Exploratory Study Of The Efficacy Of Servsafe® Online Team Behavior And Team Success Results From A Board Game Simulation Telecommuting Internships - Do They Work? The WEE GAME: A Pre-Game Effects On Learning When Students Have Information About Games And Their Outcomes When Playing Them Assessment And Simulations: Measuring The Academic Learning Compacts Within Dangerous Business: An Interactive Ethics Case Activity The Paradise Islands Assessing And Applying fiAppropriatefl Conflict Management Styles Distance Learning In Communication: Blended Or On-Line? Developing An On-Line Advanced Communication Course World Café: Simulating Seminar Dialogues in a Large Class The Programming Game: An Exploratory Collaboration Between Business Simulation And Instructional Design The Canary Principle: An Alternative Model for Providing Real-Time Coaching in an On-Line Discussion Environment Assessing Character Development Using A Direct-Approach Business Ethics Exercise Persistence Of Decisions By Simulation Game Participants The Invalidity of Profit = f(Product Quality) PIMS Validation of Marketing Games A Study of The Applicability of The Perceptions of Organizational Politics Scale (POPS) For Use in The University Classroom The ROI Of Connected Projects: A Payroll Example Initiation of Research on Gaming Simulation in Japan Inappropriate Use of Citations and Corrupting the Body of Knowledge: Accepting Urban Legends as Truth An Exploration Of The Perceived Value Of Highly Socio-Inductive Learning How "Whole" Is Whole Person Learning? An Examination Of Spirituality In Experiential Learning Modeling Outsourcing and Strategy Alignment into a Business Game Successful Integration Of Webct Into A Small Business School The Thinking Steps Model In Game Theory: A Qualitative Approach In Following The Rules The Application Of Means-End Theory To Understanding The Value Of Simulation-Based Learning Armchair Travel: An active learning approach to increasing global awareness and participant self-efficacy Goldratt's Thinking Process: Is there a place for it in the Total Enterprise Simulation Assessing And Developing Student Skills Using a Group Exercise Computer Business Simulation Design: Novelty & Complexity Issues