USING A WEB BASED TUTORIAL PROGRAM TO ENHANCE STUDENT LEARNING Developments in Business Simulation and Experiential Learning, Volume 28, 2001 USING A WEB BASED TUTORIAL PROGRAM TO ENHANCE STUDENT LEARNING Marian W. Boscia, King’s College mwboscia@kings.edu R. Bruce McAfee, Old Dominion University rmcafee@odu.edu ABSTRACT This paper describes the structure of a Web based tutorial program designed to enhance student knowledge. It is available free over the Internet and allows faculty to enter matching or multiple choice questions to meet specific and unique course objectives. The paper provides the theoretical rationale for using a tutorial program as well as discussing some of the major advantages and disadvantages in using this approach. INTRODUCTION Using computer technology to provide an active-learning experience has become a widely accepted and increasingly common technique. In addition, the number of computer- based courses continues to expand. The use of technology can be beneficial to developing a more student-centered approach to teaching in several ways. First, educators who adopt teaching methods that are computer-based will be more likely to assume the role of adviser and resource provider, while encouraging their students to work independently. Second, computers and the Internet can be catalysts that enable student-centered practices (Becker and Ravitz, 1999). Textbook publishers can be a great resource for learning activities. The many auxiliary materials supplied with the text, however, have increased the importance of being able to edit and adapt this content to achieve learning objectives that are unique to each professor=s courses. Hampton and Krentler [1993] asked management and marketing professors to classify, according to Bloom's taxonomy of educational objectives, multiple-choice questions from the textbook test files of four widely adopted introductory textbooks. The results indicated that an overwhelming 87% of management questions measure only the Bloom=s lowest level (knowledge), despite the fact that the textbook test file information claims to have test questions that measure a higher number of cognitive level questions. The numbers for marketing questions were 65% knowledge, while the publishers promised 50%of the questions were at a higher level of thinking. The authors conclude that these inaccuracies could mislead an educator who relies completely on materials provided by textbook publishers. As pervasive as online educational programs have become, the scope of the resources available to educators remains limited. William H. Graves (2000), chairman and co-founder of Eduprise.com, recently wrote: There is as yet no significant higher education marketplace in learningware. Substantive development cost, lack of learning technology standards, and return on investment issues associated with institutional adoption barriers will impede the development of highly interactive, media rich learningware. This paper describes a Web based tutorial software package that faculty can use to enhance student learning. It was developed by the senior author and is available free on the Internet. This program permits faculty (and/or students) to input their own multiple choice questions or matching exercises, along with the answers to those questions. Because the questions are entirely educator-generated and easily updated, educators have complete control over the educational objectives being met by the activity. THE WEB BASED TUTORIAL PROGRAM The Web based tutorial program, available at http://boscia.com/mcutility/index.html, has been designed to be an interactive activity for either multiple choice questions or those in a matching format. For simplicity, the underlying JavaScript is written in such a way that it can accommodate both question formats. Figure 1 presents the introductory screen which opens the exercise. This screen gives the student information about the number of questions, whom to contact in case of problems, and how to return to the linking page. The instructions are purposefully kept brief because the exercise design is very simple and is not intended to be an online testing device. Indeed, the text is kept to a minimum to enable the student to move forward quickly. Also, because the exercise appearance information is stored in a Cascading Style Sheet file, each question has a consistent appearance, and the load time for each new page is minimized. The text for the questions is also stored in a separate file. This enables the JavaScript to be 1 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 readily adapted for use with new text and also reduces load time for users with slower Internet connections. Figure 1 Linking Image Retail Inventory Practice Questions Instructions: This exercise has a total of 20 multiple-choice questions. You are asked to select the best answer on each of them. At any time, you may click to end this exercise and return to Course Title Home Page. Please email Contact Name at Contact@EmailHost.edu if you need help with this exercise. Begin the Exercise Figure 2 shows an example of the screen displayed for a multiple-choice style question. The “ linking image” usually the home page) is in the upper left-hand corner of the page, with the number of the question adjacent to it. Immediately below the graphic is the question followed by answer alternatives that are listed with a letter and “radio button” next to each. Back and Forward buttons appear at the bottom of the page. The student is not allowed to skip questions. A small alert screen will pop-up if the student clicks on the forward key without selecting a response. In the case of a multiple-choice question, the alert reads, “Please select an alternative answer before moving forward”. This left-side display, with the answer that was selected darkened, will remain visible to serve as a reference until the student moves forward to the next question. 2 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 ent exh and the Figure 2 Linking Image Question 2 Which inventory system provides the most effective means of controlling inventories? A. Perpetual B. Specific Identification C. LIFO D. Periodic E. None of the above Back Forward Upon selection of an alternative, the tutorial program ers the feedback cycle for the question. Figures 3 and 4 ibit the outcome-feedback screens displayed for incorrect correct answers. As well as telling the student whether response was correct or incorrect, the screen has a running total of items answered correctly on the first try. This total can only be viewed by moving forward to the next question. Going backward and repeating questions will not improve the percentage-correct score. Figure 3 Linking Image Question 2 Sorry, B is incorrect. Please try again. Frowning Face Image Which inventory system provides the most effective means of controlling inventories? A. Perpetual B. Specific Identification C. LIFO D. Periodic E. None of the above Back Forward On the first attempt you have correctly answered 1 item out of the 2 items shown to you, resulting in a score of 50% correct. 3 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 Regardless of whether or not the student answered a question correctly, any additional process feedback provided by the instructor is available by clicking the face image. In order to make the learningware more user-friendly, this feedback is displayed only upon the student’s request. They will not be interrupted by information they may not wish to view, thus giving them more control over the pace at which they work through the questions. In addition, making the feedback display optional allows students to more quickly page through the exercise as load time is reduced. Figure 5 shows the screen display after the feedback option has been selected. Importantly, the learningware presents feedback in a timely fashion. Each faculty member’s challenge is to develop process feedback that will be meaningful to the learning objectives being addressed. Once the student clicks on the forward button after answering the final question, an alert window opens that displays text congratulating the student for completing the activity. As soon as the student closes the alert window, the student’s browser is redirected to the same link as that connected to by the graphic at the top of the page. This assures that if the student wants to repeat the activity, the learningware will be reinitialized. THEORETICAL SUPPORT Well constructed multiple choice or matching exercises can incorporate elements of at least the first four of the six learning levels outlined in Bloom's taxonomy (knowledge, comprehension, application, analysis, synthesis, and evaluation). Questions can be asked that require students to define concepts, recall specific facts, or list steps (knowledge level). Or, they could ask students to identify, describe, or recognize terms or concepts (comprehension). However, perhaps their greatest utility is that of developing application skills. These types of questions require that students read a short scenario and apply a concept to the situation at hand. McCorkle and Diriker [1992] asserted that a student who is able to answer an application-type multiple-choice question has demonstrated an understanding of the concept at hand and should be more likely to retain what he or she has learned. Even medical schools have long considered multiple choice questions a valid and reliable way to access medical knowledge (Norcini et al., 1995; Piepmeiser, 1998). When McCorkle and Diriker asked students to evaluate an exam composed of application-type, multiple-choice questions, over 85% felt the questions were a good measure of learning, knowledge, and understanding and that the questions were interesting. As an added benefit, over 68% of the students felt that it resulted in longer-term retention of the course material. A large majority of students (about 73%) also felt that the application-type questions were difficult to answer and required more advance preparation. Their attitude points to the value of providing them with the opportunity to practice applying concepts as an enhancement to the learning process. Figure 4 Linking Image Question 2 Excellent. A is correct. Smiling Face Image Which inventory system provides the most effective means of controlling inventories? A. Perpetual B. Specific Identification C. LIFO D. Periodic E. None of the above Back Forward On the first attempt you have correctly answered 2 items out of the 2 items shown to you, resulting in a score of 100% correct. 4 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 BENEFITS OF THE WEB BASED TUTORIAL PROGRAM The tutorial program described above has been used in teaching Principles of Accounting I and II and Intermediate Accounting I and II at the undergraduate level. It has also been used in teaching a graduate course in Auditing Theory. The authors= experience with the tutorial suggests that it has numerous benefits. First, it is highly flexible in that it can be used in virtually any course. To do so, a faculty member need only develop or obtain multiple choice or matching questions along with the answers and insert them into the program. Then students need only be directed to the Web site. The tutorial program is flexible, too, in that it can be used in a variety of ways. Students in advanced courses could use it to review principles and concepts that were covered in prerequisite courses. This may be particularly beneficial to transfer students who took these courses at another college. Second, the program can be used by faculty to teach concepts/terminology that will not be discussed in class, thereby freeing up class time for experiential or other learning activities. Third, the tutorial can be used to reinforce materials covered in class, including both lecture and experiential learning concepts. Finally, the tutorial can be used in class as part of the test preparation process. From a student’s perspective, using a tutorial such as the one discussed here is both easy and enjoyable. The authors are not alone in this assessment. Oliver [1996] studied the influence of instruction and practice on the development of skills by using interactive information systems that were far more complex than the learning-ware described in this paper. The researcher reported that the majority of students developed a sense of ease in using the electronic information system despite varying levels of previous computer experience. Important also, there were no discernible differences in level of usage and/or willingness to use the software attributable to gender. Similarity, McQueen and Fleck [1999] point out that students often have problems with lost CDs and disks and damaged software. The learningware described in this present paper eliminates these problems because the instructor controls and maintains the files on the Internet. As long as the Web Host is operating, the materials are readily available. A final study that lends support to our Figure 5 Linking Image Question 2 Excellent. A is correct. Which inventory system provides the most effective means of controlling inventories A. Perpetual B. Specific Identification C. LIFO D. Periodic E. None of the above The perpetual inventory system provides the more effective means of controlling inventories, since the inventory account is updated for each purchase and sale. This also assists managers in determining when to reorder inventory items. You have correctly answered 2 of the 2 items shown to you, resulting in a score of 100% correct. Back Forward 5 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 contention was conducted by Herman, Klein, and Wakai [1997] who gathered data using the 1993 California Learning Assessment System regarding student attitudes towards various types of assessment. These researchers examine the issue of student preference for open-ended assessments that encourage students to think critically and draw their own conclusions to complex problems versus the traditional types of tests, such as multiple-choice. Findings, based on a sample of middle-school students, suggest that students do indeed understand the differences in approaches. Although students found the open-ended questions to be more interesting and challenging, they believed that the multiple-choice format made it easier to understand what to do. This can be very important in a learning setting that is intended to be completely self-moderated. Students also felt that they perform better on multiple-choice questions. This survey’s results indicate that 60% of the students like multiple-choice questions better than open-ended questions, while only 17% preferred the open-ended variety. This is an important consideration for an activity that is largely voluntary and unmonitored. One of the benefits of this tutorial over the many utilities available from book publishers is that it can be completely non-intrusive. It does not have to reside in a password- protected area nor does it collect user information, as is the case with most commercial sites. Therefore, the student's privacy is protected. Second, the graphics, text headings, buttons, and score-box background colors can easily be changed. This adds visual interest, without detracting from functionality. Also, as stated earlier, this learningware is readily adaptable to incorporate the educator=s own materials. POTENTIAL DRAWBACKS TO USING THE TUTORIAL Perhaps the major drawback of using the tutorial described here is that it may require considerable faculty time to develop both the questions and the answers. One potential solution is to use questions obtained from other faculty or from previous tests. Another potential approach, however, is asking students to develop these materials themselves. The value of having students create a test bank is supported by at least a few research studies. For example, McLeod and Snell [1996] report the results of involving second-year medical students in the process of developing multiple-choice questions. Over 90% of the questions submitted were suitable for use. For the comprehensive 70-question multiple-choice exam held at the end of the course, questions were derived from a bank of questions developed by both faculty and students. Test takers scored an average of 69% on the student generated questions, and only slightly lower (66%) on the faculty-generated questions. Thus, students were well able to write challenging questions that addressed information they themselves recognized as important to their learning. Importantly, the mean validity ratings for student-generated questions were slightly higher (2.39, range 1.80-3.00) than that of faculty-generated questions (2.26, range 1.80-2.80). This study demonstrated that students do have the capacity to be very competent question writers. There are also a few limitations to the programming aspects of the tutorial program. Internet Explorer, version 4.0 or higher must be available as the JavaScript restrictions under Netscape prevent the utility from loading and operating properly. The tutorial itself, however, does not share this limitation. It was written to be cross-browser compatible. JavaScript has another shortcoming; it reads groups of spaces as a single space. Therefore, if the user wants to have two spaces in a row, one of those spaces must be created using " " which is the character string for a space. If the user wants to insert a line break in some particular place, the HTML code,
, must be used. Further, entering text for the questions or feedback that is in table-form is a complex process and should not be undertaken without a fairly complete understanding of HTML and JavaScript, not to mention patience and a willingness to experiment. CONCLUSION In conclusion, the Web based tutorial program described here can be a powerful tool for engaging students in active learning. Its adaptable and versatile design allows faculty to write questions and provide answers that are tailor made for their own courses. Students find the program easy and enjoyable to use, and their privacy is protected. Furthermore, they can use the tutorial at their own convenience and work at their own pace. One possible use of the tutorial that the authors have not explored is to divide the class into small groups and ask them to answer the tutorial questions. Since students will be discussing each question and sharing information, this may enhance the overall learning experience. This could be done during or outside of class. As a final note, any faculty member who wishes to use the tutorial program can obtain further information, including the files needed to implement the tutorial itself, from the senior author. REFERENCES Becker, H. J. and Ravitz, J. (1999) "The Influence of Computer and Internet Use on Teachers' Pedagogical Practices and Perceptions." Journal of Research on Computing in Education, Vol. 31, Issue 4, 356-384. Graves, W. H. (2000) "The Dot.xxx Challenge." Syllabus, Vol. 13, No. 10, 30-36. Hampton, D. R. and Krentler, K. A. (1993) "The Use of Management and Marketing Textbook Multiple-choice Questions: a Case Study." Journal of Education for Business, 1993, Vol. 69, 1, 40-43. 6 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 Herman, J. L.; Klein, D. C., and. Wakai, S. T. (1997) "American Students' Perspectives on Alternative Assessment: Do They Know It's Different?" Assessment in Education: Principles, Policy & Practice, Vol. 4, 3, 339-352. McCorkle, D. E. and Diriker, M.F. (1992) " An involvement- oriented approach in a medium-sized marketing principles class." Journal of Education for Business, Vol. 67, 4, 197-205. McLeod, P.J. & Snell, L. (1996) "Student-generated MCQs." Medical Teacher, Vol. 18, 1, 23-25. McQueen, T. F. and Fleck, Jr., R. A. (1999) "An Evaluation of Alternative Technology-based Instructional Formats." T H E Journal, Vol. 26, 11, 108-115. Norcini, J., Swenson, B., Grosso, J. & Webster, G. (1995) "Reliability, Validity and Efficiency of Multiple Choice Question and Patient Management Problem Item Formats in Assessment of Clinical Competence." Medical Education, Vol.19, pp. 238-247. Oliver, R. (1996) "The Influence of Instruction And Activity on The Development of Skills in the Usage of Interactive Information Systems." Education for Information, Vol. 14. Piepmeier, E., Jr. (1998) "Use of Concept tests in a Large Lecture Course To Provide Active Student Involvement and Peer Teaching." American Journal of Pharmaceutical Education, Vol. 62, 3, 347-51. 7 Table of Contents Volume 28, 2001 Using A Web Based Tutorial Program To Enhance Student Learning Reflecting On Reflection: An Examination Of Reflective Learning And Assessing Outcomes Overcoming Obstacles To A Cross-Global Collaborative Classroom Panel Discussion On Building And Maintaining Trust In The Abselesque Classroom Cosmopolitan-Based Cross National Segmentation In Global Marketing Simulations Strategic Management And The Case Method: Survey And Evaluation In Hong Kong Designing Interactive Self-Learning Modules Using Macromedia Director Teaching Financial Markets Using The Wall Street Journal Core Course Outcomes Assessment And Program Continuous Improvement Using An Integrative Business Plan Œ An Empirical Evaluation Using A Mock Trial Activity For Interdisciplinary Learning Fidelity, Verifiability, And Validity Of Simulation: Constructs For Evaluation 2001 International Collegiate Business Strategy Competition: 37 Years Of Collegiate Competition Motivating Students: An Initial Attempt To Operationalize The Curiosity Gap Model E-Learning: The Next Wave Of Experiential Learning Helping New Game Adopters: Four Perspectives An Initial Validity Invesitigaion of A Test Assessing Total Enterprise Simulation Learning Administering A Marketing Simulation - Common And Varied Practices Among Instructors Issues In Case Method Instruction: Class Discussion Leadership Sex Composition, Cohesion, Consensus, Potency And Performance Of Simulation Teams Integrating Management Curriculum: Linking Courses With Frontpage Absel: The Way We Talk! Teaching Strategic Management: A Case Study In The Diffusion Of Innovation In Education A Demonstration Of Bussim Strategy A Total Enterprise Simulation Teaching Methods: Etic Or Emic Antecedents Of Work Team Performance In A Business Simulation: Personality And Group Interaction When Experiential Educators Reflect Upon Their Craft: A Qualitative Research Report Strategic Sensemaking In Organizations: Model Formulation And Operationalization Panel Discussion: A Model For Initiative Development: Discussion And Ideas Of Individual Spirit Harmonics Assessing The Efficacy Of Experiential Learning In A Multicultural Environment Are Business Schools Producing 21st Century Managers? Blue Horizon Cruises: An International Simulation Exercise To Illustrate Conflicting Economic And Cultural Models Different Perceptions Of The Same Thing: An Experiential Exercise Current Use Of Agribusiness Simulation Games: Survey Results Of University Agribusiness And Agricultural Economics Programs Team Building 101 For Accountants An Online Evaluation Of The Compete Online Decision Entry System (Codes) Total Enterprise Simulation Winners And Losers: A Preliminary Study Gilding The Lily, Enhancing The Pedagogical Experience, Or Fixing What Isn™t Broke: The Dilemma Of New Simulation Versions The Absel Endnote Database: The Perfect Tool For The Bernie Keys Library Decision-Making Exercises (A), (B), And (C): Examining The Role Of Conflict In Group Decision-Making Group Decision Simulation (A)-(F): Examining Procedural Justice In Group Decision-Making Processes Evaluation Of Performance In Business Games: Financial And Non Financial Approaches A Framework For Evaluating Simulations As Educational Tools Congruence II™: A Strategic Business Board Game The Tournament Concept: Extending The Idea Of Using Simulations As Instruments Of Assessment The Economic And Political Transition Of The Mexican State In The Threshold Of Twenty First Century: From The Entrepreneurial State To The State Of Entrepreneurs. Experiential Learning For Accountants: The Not For Profit Project An Investigation Of The Environmental Awareness Attained In A Simple Business Simulation Game A Daily Exercise In Theory Application: Is This Why I Had To Take Marketing, Management, Finance –? An Alternative View to Scoring Selected Simulation Games and Experiential Exercises